Acknowledgement-on-behalf detection method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the technical field of artificial intelligence, and relates to a pickup detection method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of original audio and video data, and obtaining a sound and lip joint feature vector and a sound and lip synchronization feature vector; carrying out vector splicing processing on the sound lip joint feature vector and the sound lip synchronous feature vector to obtain a fusion feature vector; performing action segmentation processing on the fusion feature vector according to a multi-scale time sequence convolutional network to obtain an action probability distribution sequence with the same length as an input time step; and performing pickup recognition processing on the action probability distribution sequence according to the classifier to obtain a pickup detection result. The method can be used for related video processing in a financial science and technology service system, action segmentation and classification recognition are performed by fusing the audio-lip joint features and the audio-lip synchronization features and using the multi-scale time sequence convolutional network, and the accuracy and reliability of pickup detection are effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, and storage medium for detecting proxy responses. Background Technology

[0002] In many scenarios involving identity verification or interaction, proxy answering is a serious problem. For example, in online exams, candidates may have someone else answer the questions for them to obtain unfair scores; in remote interviews, job applicants may have someone else take the interview on their behalf to deceive the recruiter.

[0003] Traditional methods for detecting proxy answers mainly rely on single audio or video features, such as identifying the speaker solely through speech recognition or identifying the person in the scene solely through facial recognition.

[0004] However, the applicant found that these single-feature methods have significant limitations. Specifically, audio features are easily affected by environmental noise, speech imitation, and other factors, leading to a decrease in recognition accuracy. For example, in noisy environments, speech recognition systems may fail to accurately extract the speaker's features, thus misjudging proxy behavior. While video features can provide more information to some extent, they are also easily affected by factors such as changes in lighting and occlusion. Moreover, relying solely on a single feature is insufficient to comprehensively and accurately determine whether proxy behavior exists, as the proxy may meticulously prepare to imitate the audio or video features of the person being represented.

[0005] This shows that traditional methods for detecting proxy responses have problems with low accuracy and effectiveness. Summary of the Invention

[0006] The purpose of this application is to provide a method, apparatus, computer device, and storage medium for detecting proxy responses, so as to solve the problem that traditional proxy response detection methods have low detection accuracy and effectiveness.

[0007] To address the aforementioned technical problems, this application provides a method for detecting proxy responses, employing the following technical solution: Receive proxy response detection requests carrying raw audio and video data; The original audio and video data are processed by feature extraction to obtain the joint lip-phonetic feature vector and the lip-phonetic synchronization feature vector; The joint lip-consonant feature vector and the lip-consonant synchronization feature vector are concatenated to obtain a fused feature vector. The fused feature vector is processed by a multi-scale temporal convolutional network to perform action segmentation, resulting in an action probability distribution sequence with the same time step length as the input. The proxy response detection result is obtained by performing proxy response identification processing on the probability distribution sequence of the actions according to the classifier.

[0008] Furthermore, the step of performing feature extraction processing on the original audio and video data to obtain the joint lip-phoneme feature vector and the lip-phoneme synchronization feature vector specifically includes the following steps: The audio and video data are processed by audio and video segmentation to obtain video and audio waveform data of the mouth area; The pre-trained multimodal self-supervised learning model is invoked, wherein the multimodal self-supervised learning model includes a deep residual network and a feedforward network; The deep residual network is used to extract lip features from the mouth region video to obtain temporal lip feature data. The audio waveform data is processed by the feedforward network to extract spectral features, thereby obtaining speech spectral feature data. The audio-video fusion processing of the lip temporal feature data and the speech spectral feature data yields the joint lip-speech feature vector. The lip timing feature data and the speech spectrum feature data are subjected to differential synchronization processing to obtain the lip-speech synchronization feature vector.

[0009] Furthermore, the step of performing differential synchronization processing on the lip timing feature data and the speech spectrum feature data to obtain the lip-speech synchronization feature vector specifically includes the following steps: The absolute difference between the lip timing feature data and the speech spectrum feature data at the frame level is calculated to obtain the lip-speech synchronization feature vector.

[0010] Furthermore, the step of performing differential synchronization processing on the lip timing feature data and the speech spectrum feature data to obtain the lip-speech synchronization feature vector specifically includes the following steps: Long Short-Term Memory (LSTM) networks are used to process the temporal dependencies of the lip temporal feature data and the speech spectral feature data to determine the preliminary co-matching scores between sequences. If the initial collaborative matching score is lower than the preset threshold, adjustments are made to the time alignment deviation between sequences. The lip-sync feature vector is obtained by calculating the deviation compensation value.

[0011] Furthermore, after the step of performing action segmentation processing on the fused feature vector using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input, the following step is also included: If there are abnormal peaks in the action probability distribution sequence, the action probability distribution sequence is adjusted according to time series smoothing filtering to obtain a smooth action probability distribution sequence.

[0012] Furthermore, after the step of receiving the proxy detection request carrying the original audio and video data, the following steps are also included: The original audio data is subjected to noise reduction processing; and / or The original audio data is then processed by frame segmentation.

[0013] To address the aforementioned technical problems, this application also provides a proxy detection device, which employs the following technical solution: The request acquisition module is used to receive proxy detection requests carrying raw audio and video data; The feature extraction module is used to perform feature extraction processing on the original audio and video data to obtain the joint lip-phonetic feature vector and the lip-phonetic synchronization feature vector; The vector concatenation module is used to perform vector concatenation processing on the joint lip-phoneme feature vector and the lip-phoneme synchronization feature vector to obtain a fused feature vector. The action segmentation module is used to perform action segmentation processing on the fused feature vector according to the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same input time step length. The proxy answering module is used to perform proxy answering processing on the probability distribution sequence of the actions according to the classifier to obtain proxy answering detection results.

[0014] Furthermore, the feature extraction module includes: The audio and video segmentation module is used to perform audio and video segmentation processing on the audio and video data to obtain video and audio waveform data of the mouth area; The model invocation submodule is used to invoke a pre-trained multimodal self-supervised learning model, wherein the multimodal self-supervised learning model includes a deep residual network and a feedforward network; The lip feature extraction submodule is used to perform lip feature extraction processing on the mouth region video according to the deep residual network to obtain lip temporal feature data; The spectrum feature extraction submodule is used to perform spectrum feature extraction processing on the audio waveform data based on the feedforward network to obtain speech spectrum feature data. The audio-video fusion submodule is used to perform audio-video fusion processing on the lip temporal feature data and the speech spectrum feature data to obtain the joint lip-speech feature vector. The differential synchronization submodule is used to perform differential synchronization processing on the lip timing feature data and the speech spectrum feature data to obtain the lip-speech synchronization feature vector.

[0015] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the substitution detection method as described above.

[0016] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the substitution detection method as described above.

[0017] This application provides a method for detecting proxy answers, comprising: receiving a proxy answer detection request carrying original audio and video data; performing feature extraction processing on the original audio and video data to obtain a joint lip-phoneme feature vector and a lip-phoneme synchronization feature vector; performing vector concatenation processing on the joint lip-phoneme feature vector and the lip-phoneme synchronization feature vector to obtain a fused feature vector; performing action segmentation processing on the fused feature vector according to a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input; and performing proxy answer recognition processing on the action probability distribution sequence according to a classifier to obtain a proxy answer detection result. Compared with the prior art, this application, by fusing joint lip-phoneme features and lip-phoneme synchronization features, comprehensively utilizes multi-dimensional information from audio and video, and can capture the features of proxy answer behavior more comprehensively and accurately. Compared with traditional single feature detection methods, it greatly improves the accuracy of proxy answer detection and reduces false positives and false negatives; in addition, the multi-scale temporal convolutional network can capture feature changes at different time scales and has a certain robustness to environmental noise, illumination changes, and other factors. Even in complex real-world scenarios, it can effectively perform action segmentation and classification recognition, ensuring the stability and reliability of the proxy response detection system. Attached Figure Description

[0018] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart illustrating the implementation of the proxy response detection method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the multimodal self-supervised learning model provided in the embodiments of this application; Figure 4This is a schematic diagram of the structure of the substitution detection device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0024] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0025] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0026] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0027] It should be noted that the proxy answer detection method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the proxy answer detection device is generally set in the server / terminal device.

[0028] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0029] Continue to refer to Figure 2 The diagram shows a flowchart of an embodiment of the proxy response detection method according to this application. The proxy response detection method described above includes steps S201, S202, S203, S204, and S205.

[0030] In step S201, a proxy detection request carrying the original audio and video data is received.

[0031] In the embodiments of this application, a user inputs a request through their terminal device, which is received by the system. The terminal device may be a mobile terminal such as a mobile phone, smartphone, laptop, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), navigation device, etc., or a fixed terminal such as a digital TV, desktop computer, etc. It should be understood that the examples of terminal devices given herein are for convenience of understanding only and are not intended to limit this application.

[0032] In this embodiment of the application, the aforementioned original audio and video data is the specific content that users hope the system will use to effectively identify fraudulent behavior such as proxy answering in online review scenarios. The original audio and video data contains audio and video information that may involve proxy answering and serves as the basic data for subsequent detection.

[0033] In some optional implementations of the embodiments of this application, after the step of receiving the proxy detection request carrying the original audio and video data, the following steps are further included: Noise reduction processing is performed on the raw audio data; and / or The original audio data is processed by frame segmentation.

[0034] In step S202, feature extraction processing is performed on the original audio and video data to obtain the joint lip-phonetic feature vector and the lip-phonetic synchronization feature vector.

[0035] In this embodiment, the lip-phonetic joint feature reflects the correlation information between lip movements in audio and video. By performing spectral analysis and Mel-frequency cepstral coefficient (MFCC) extraction on the audio signal, a feature representation of the audio is obtained; simultaneously, the lip region in the video is detected and tracked, and features such as lip shape and movement trajectory are extracted. Then, the audio features and lip video features are fused to obtain the lip-phonetic joint feature vector. This feature vector can capture the coordinated changes in audio and lip movements during speech, providing an important basis for proxy answer detection.

[0036] In this embodiment, the lip-sync feature primarily focuses on the temporal synchronization relationship of lip movements in audio and video. By calculating the difference between the timestamps of the audio signal and the timestamps of the lip movement keyframes, and analyzing the correlation between audio and video on the time axis, a lip-sync feature vector is extracted. If proxying occurs, the synchronization relationship between audio and video may be disrupted; therefore, this feature vector can effectively detect such anomalies.

[0037] In step S203, the lip-phonetic joint feature vector and the lip-phonetic synchronization feature vector are concatenated to obtain the fused feature vector.

[0038] In this embodiment, two different types of feature vectors are concatenated together, enabling the comprehensive utilization of their contained information to form a more comprehensive and richer feature representation. This fused feature vector can reflect the features in audio and video data from multiple perspectives, providing stronger support for subsequent action segmentation and classification recognition.

[0039] In step S204, the fused feature vector is processed by action segmentation according to the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input.

[0040] In this embodiment, the multi-scale temporal convolutional network has multiple convolutional kernels of different scales, enabling it to capture feature changes at different time scales. By performing multi-scale convolution operations on the fused feature vector, the network can learn action patterns in different time periods of audio and video data, and divide each time step into different action categories, outputting a corresponding action probability distribution sequence. This sequence reflects the probability of different actions occurring at each time step, providing a foundation for subsequent proxy recognition.

[0041] In step S205, the proxy answering process is performed on the action probability distribution sequence according to the classifier to obtain the proxy answering detection result.

[0042] In the embodiments of this application, the classifier can be various common machine learning classification models, such as support vector machines (SVM), decision trees, neural networks, etc. By analyzing and judging the action probability distribution sequence, the classifier can determine whether there is proxy answering behavior in the current audio and video data, and output the corresponding detection result, such as "it is proxy answering" or "it is not proxy answering".

[0043] In practical applications, taking remote video interviews at banks as an example, users need to submit real-time audio and video via their mobile phones / cameras. The raw data received by the system includes speech streams and video streams (lip movements, facial micro-expressions). This application requires synchronous alignment of the audio and video (e.g., through timestamp calibration), noise reduction processing (e.g., removing background noise and video blurring), and segmentation into fixed-length segments (e.g., 5 seconds per segment) to ensure the stability of subsequent feature extraction. The application extracts time-frequency domain features such as MFCC (Mel-frequency cepstral coefficients), fundamental frequency, and speech rate from the speech stream as audio features. Lip keypoint detection (e.g., a 68-point model) captures motion features such as lip opening and closing and edge changes as video features. Then, an attention mechanism is used to dynamically weight and fuse the audio and video features to obtain a joint lip-speech feature vector. Finally, a temporal alignment algorithm is used to quantize the audio. Synchronization feature vectors are formed by measuring synchronization errors (e.g., millisecond-level latency) and synchronization confidence levels with lip movements. These feature vectors are then concatenated with the synchronization feature vectors along the channel dimension to form a high-dimensional fusion feature vector (e.g., dimension 256 + 128 = 384). Next, a multi-scale temporal convolutional network is used to perform temporal modeling on the fusion feature sequence, outputting an action probability distribution sequence with the same time step length as the input. The probability value at each time step reflects the confidence level of any proxy answering behavior at that moment. Finally, a Softmax classifier or a Sigmoid binary classifier is used to map the action probability distribution sequence to proxy answering probabilities (e.g., values ​​between 0 and 1). A threshold is set according to business requirements (e.g., probability > 0.8 indicates proxy answering), and the detection results are output, thus enabling proxy answering prediction for remote video interviews at banks.

[0044] This application provides a method for detecting proxy answers, comprising: receiving a proxy answer detection request carrying original audio and video data; performing feature extraction processing on the original audio and video data to obtain a joint lip-phoneme feature vector and a lip-phoneme synchronization feature vector; performing vector concatenation processing on the joint lip-phoneme feature vector and the lip-phoneme synchronization feature vector to obtain a fused feature vector; performing action segmentation processing on the fused feature vector according to a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input; and performing proxy answer recognition processing on the action probability distribution sequence according to a classifier to obtain a proxy answer detection result. Compared with the prior art, this application, by fusing joint lip-phoneme features and lip-phoneme synchronization features, comprehensively utilizes multi-dimensional information from audio and video, and can capture the features of proxy answer behavior more comprehensively and accurately. Compared with traditional single feature detection methods, it greatly improves the accuracy of proxy answer detection and reduces false positives and false negatives; in addition, the multi-scale temporal convolutional network can capture feature changes at different time scales and has a certain robustness to environmental noise, illumination changes, and other factors. Even in complex real-world scenarios, it can effectively perform action segmentation and classification recognition, ensuring the stability and reliability of the proxy answer detection system.

[0045] In some optional implementations of the embodiments of this application, the steps of performing feature extraction processing on the original audio and video data to obtain the joint lip-phoneme feature vector and the lip-phoneme synchronization feature vector specifically include the following steps: The audio and video data are segmented to obtain video and audio waveform data of the mouth area; Call the trained multimodal self-supervised learning model, which includes a deep residual network and a feedforward network; The lip feature extraction process is performed on the video of the mouth region using a deep residual network to obtain temporal lip feature data. The audio waveform data is processed by a feedforward network to extract spectral features, resulting in speech spectral feature data. The audio-video fusion processing of lip temporal feature data and speech spectrum feature data yields a joint lip-speech feature vector. Differential synchronization processing is performed on lip temporal feature data and speech spectrum feature data to obtain lip-sound synchronization feature vectors.

[0046] In this embodiment, firstly, a face detection algorithm (such as the deep learning-based face detection model MTCNN) is used to locate the face region in the video frame. Then, the position and range of the mouth region are determined according to a face keypoint detection algorithm (such as the keypoint detection method in the Dlib library). The mouth region is then cropped out from the video frame to form a mouth region video. Simultaneously, the audio signal is separated from the original audio and video data to obtain audio waveform data, preparing for subsequent feature extraction.

[0047] In this embodiment, the multimodal self-supervised learning model is mainly used to learn speech representation by combining visual and auditory signals, thereby improving the performance of speech recognition in noisy environments. This multimodal self-supervised learning model can be the Audio-Visual HuBERT model, see [link / reference]. Figure 3 The diagram shows the structure of the Audio-VisualHuBERT (AV-HuBERT) model, where: the visual branch extracts the temporal features of the lip region through a ResNet-18 residual neural network, focusing on the details of lip movement; and the audio branch converts the audio waveform into logarithmic filter bank energy features through a feedforward network (FFN), capturing the dynamics of the speech spectrum.

[0048] In this embodiment, a video of the mouth region is input into a deep residual network. The network extracts features of the mouth region step by step through multi-layer convolution and pooling operations. The residual connections in the deep residual network can effectively solve the gradient vanishing problem during the training process of deep networks, enabling the network to learn deeper features. After processing by the deep residual network, temporal feature data of the lips that reflects changes in lip movement and time-series information is obtained.

[0049] In this embodiment, audio waveform data is input into a feedforward network. The feedforward network first preprocesses the audio signal, such as performing frame segmentation and windowing operations. Then, it converts the time-domain signal into a frequency-domain signal using Fourier transform to extract the spectral features of the audio. The feedforward network can further perform in-depth learning and transformation of the spectral features to obtain more representative speech spectral feature data.

[0050] In this embodiment, the application employs a feature concatenation method to fuse lip temporal feature data and speech spectrum feature data. Specifically, the two types of feature data are concatenated according to certain dimensions to form a new feature vector, namely, the lip-phonetic joint feature vector. This fusion method can comprehensively utilize information from lip movement and speech spectrum to reflect the collaborative relationship between lip and phonetic sounds, providing richer feature representations for subsequent proxy response detection.

[0051] In this embodiment, lip-sync features are extracted by calculating the differences and synchronization relationships between lip temporal feature data and speech spectral feature data on the time axis. For example, the Dynamic Time Warping (DTW) algorithm can be used to measure the temporal alignment of the two feature data, and feature vectors reflecting lip-sync can be extracted based on the alignment results. The lip-sync feature vector can capture temporal inconsistencies between audio and video. When proxy responses exist, these inconsistencies may be more pronounced; therefore, this feature vector is of great significance for proxy response detection.

[0052] Compared with existing technologies, this application improves the accuracy and reliability of proxy detection by accurately extracting lip temporal features and speech spectrum features, and effectively fusing and differentially synchronizing them to obtain more discriminative lip-phonetic joint feature vectors and lip-phonetic synchronization feature vectors.

[0053] In some optional implementations of the embodiments of this application, the step of performing differential synchronization processing on the lip timing feature data and speech spectrum feature data to obtain the lip-speech synchronization feature vector specifically includes the following steps: The absolute difference between the lip timing feature data and the speech spectrum feature data at the frame level is calculated to obtain the lip-speech synchronization feature vector.

[0054] In this embodiment, the frame-level absolute difference between the lip feature sequence and the audio feature sequence is calculated to generate a synchronization feature vector. The formula is .in, For the first Frame lip features, For the first Frame audio features This represents the total number of frames in the video.

[0055] In some optional implementations of the embodiments of this application, the step of performing differential synchronization processing on the lip timing feature data and speech spectrum feature data to obtain the lip-speech synchronization feature vector specifically includes the following steps: Long Short-Term Memory (LSTM) networks were used to process the temporal dependencies of lip temporal feature data and speech spectral feature data to determine the preliminary co-matching scores between sequences. If the initial collaborative matching score is lower than the preset threshold, adjustments are made to the time alignment deviation between sequences. By calculating the deviation compensation value, the lip-sync feature vector is obtained.

[0056] In this embodiment, a Long Short-Term Memory (LSTM) network is used to process the temporal dependencies of lip temporal feature data and speech spectral feature data. LSTM, as a special type of recurrent neural network (RNN), can effectively handle long-term dependencies in sequence data. The lip temporal feature sequence and the speech spectral feature sequence are respectively input into the LSTM network. The LSTM network learns and memorizes the information at each time step in the sequence, capturing the temporal correlation between the two sequences.

[0057] In this embodiment, the preliminary co-matching score between sequences is determined based on the processing results of the LSTM network. This score reflects the degree of co-matching between the lip movement sequence and the speech spectrum sequence within the current time range; a higher score indicates better synchronization between the two. The preliminary co-matching score can be calculated in various ways, such as by calculating the sum of the activation values ​​of corresponding neurons in the output layer of the LSTM network, or by using a matching index calculated based on a specific loss function.

[0058] In this embodiment, the application determines whether the preliminary coordination matching score is lower than a preset threshold. This preset threshold can be set based on actual application scenarios and experience, for example, by analyzing a large number of normal lip-synced and asynchronous samples to determine a suitable threshold range.

[0059] In this embodiment, if the initial collaborative matching score is lower than a preset threshold, it indicates a time alignment deviation between sequences. In this case, adjustments are made to address the time alignment deviation. Specifically, this adjustment involves calculating a deviation compensation value. This value can be obtained by analyzing the time difference distribution between the lip timing feature sequence and the speech spectral feature sequence, combined with the temporal dependencies learned by the LSTM network. For example, the Dynamic Time Warping (DTW) algorithm combined with the output of the LSTM can be used to calculate the optimal alignment path between the two sequences, thereby obtaining the time deviation compensation value.

[0060] In this embodiment, the original lip timing feature data and speech spectrum feature data are adjusted based on the calculated deviation compensation value to achieve better temporal alignment. Then, a lip-sound synchronization feature vector is obtained based on the adjusted feature data. The lip-sound synchronization feature vector may contain key information of the adjusted lip features and speech features, as well as indicators reflecting their synchronization state, for subsequent lip-sound synchronization evaluation or application.

[0061] Compared with existing technologies, this application can accurately process the temporal dependency between lip timing feature data and speech spectrum feature data, determine the synchronization status through preliminary collaborative matching scores, and adjust the time alignment deviation when necessary, thereby generating an accurate lip-sound synchronization feature vector and improving the accuracy of lip-sound synchronization.

[0062] In some optional implementations of the embodiments of this application, after performing action segmentation processing on the fused feature vector according to a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same length as the input time step, the following steps are further included: If there are abnormal peaks in the action probability distribution sequence, the action probability distribution sequence is adjusted according to time series smoothing filtering to obtain a smooth action probability distribution sequence.

[0063] In the embodiments of this application, due to the influence of various factors, such as noise interference during data acquisition and the possibility that the multi-scale temporal convolutional network may not fully fit all complex situations during training, the obtained action probability distribution sequence may exhibit abnormal peaks. Abnormal peaks refer to situations in the action probability distribution sequence where the probability values ​​at certain time steps are significantly higher than the probability values ​​at surrounding normal time steps, and these high probability values ​​do not conform to the actual action change patterns. Such abnormal peaks can interfere with the subsequent accurate judgment and recognition of actions; for example, they may lead to the incorrect segmentation of a continuous action into multiple segments, or the misclassification of different actions as the same action, thereby reducing the accuracy and reliability of action segmentation and recognition.

[0064] In this embodiment, to address the aforementioned issues, a judgment and adjustment step is added after obtaining the action probability distribution sequence. Specifically, the action probability distribution sequence is first detected to determine if any abnormal peaks exist. If abnormal peaks are detected, a time-series smoothing filter is used to adjust the action probability distribution sequence.

[0065] In this embodiment, time series smoothing filtering primarily reduces random fluctuations and noise in the data by applying some form of weighted averaging or local fitting to each data point in the time series data, thereby making the data smoother and more stable. This smoothing filtering method can be moving average filtering, exponential smoothing filtering, etc.

[0066] Compared with existing technologies, this application can improve the accuracy of subsequent judgment and recognition of actions.

[0067] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0068] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0070] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0071] Further reference Figure 4 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a proxy response detection device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0072] like Figure 4 As shown, the proxy response detection device 200 of this application embodiment includes: Request acquisition module 210 is used to receive a proxy detection request carrying the original audio and video data; The feature extraction module 220 is used to perform feature extraction processing on the original audio and video data to obtain the joint lip-phone feature vector and the lip-phone synchronization feature vector. The vector concatenation module 230 is used to perform vector concatenation processing on the lip-consonant joint feature vector and the lip-consonant synchronization feature vector to obtain the fused feature vector. The action segmentation module 240 is used to perform action segmentation processing on the fused feature vector based on a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same input time step length. The proxy answering module 250 is used to perform proxy answering processing on the action probability distribution sequence based on the classifier to obtain proxy answering detection results.

[0073] In this embodiment, a proxy answering detection device 200 is provided, comprising: a request acquisition module 210 for receiving a proxy answering detection request carrying original audio and video data; a feature extraction module 220 for performing feature extraction processing on the original audio and video data to obtain a joint lip-phoneme feature vector and a lip-phoneme synchronization feature vector; a vector concatenation module 230 for performing vector concatenation processing on the joint lip-phoneme feature vector and the lip-phoneme synchronization feature vector to obtain a fused feature vector; an action segmentation module 240 for performing action segmentation processing on the fused feature vector according to a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input; and a proxy answering recognition module 250 for performing proxy answering recognition processing on the action probability distribution sequence according to a classifier to obtain a proxy answering detection result. Compared with the prior art, this application, by fusing joint lip-phoneme features and lip-phoneme synchronization features, comprehensively utilizes multi-dimensional information from audio and video, and can capture the features of proxy answering behavior more comprehensively and accurately. Compared to traditional single-feature detection methods, this approach significantly improves the accuracy of proxy answer detection and reduces false positives and false negatives. Furthermore, the multi-scale temporal convolutional network can capture feature changes at different time scales and exhibits robustness to environmental noise and lighting variations. Even in complex real-world scenarios, it can effectively perform action segmentation and classification, ensuring the stability and reliability of the proxy answer detection system.

[0074] In some optional implementations of the embodiments of this application, the feature extraction module includes: The audio and video segmentation module is used to perform audio and video segmentation processing on audio and video data to obtain video and audio waveform data of the mouth area; The model invocation submodule is used to invoke the trained multimodal self-supervised learning model, which includes a deep residual network and a feedforward network. The lip feature extraction submodule is used to extract lip features from the mouth region video based on the deep residual network to obtain temporal lip feature data. The spectrum feature extraction submodule is used to perform spectrum feature extraction processing on audio waveform data based on the feedforward network to obtain speech spectrum feature data. The audio-video fusion submodule is used to perform audio-video fusion processing on lip temporal feature data and speech spectrum feature data to obtain a joint lip-speech feature vector. The differential synchronization submodule is used to perform differential synchronization processing on lip timing feature data and speech spectrum feature data to obtain lip-sound synchronization feature vectors.

[0075] In some optional implementations of the embodiments of this application, the above-mentioned difference synchronization submodule includes: The absolute difference calculation unit is used to calculate the frame-level absolute difference between lip temporal feature data and speech spectrum feature data to obtain the lip-speech synchronization feature vector.

[0076] In some optional implementations of the embodiments of this application, the above-mentioned difference synchronization submodule further includes: The matching score determination unit is used to process the temporal dependencies of lip temporal feature data and speech spectrum feature data using a long short-term memory network, and to determine the preliminary co-matching score between sequences. The time alignment deviation adjustment unit is used to adjust the time alignment deviation between sequences if the initial coordination matching score is lower than a preset threshold. By calculating the deviation compensation value, the lip-sync feature vector is obtained.

[0077] In some optional implementations of the embodiments of this application, the above-mentioned substitute answer detection device 200 further includes: The probability distribution sequence adjustment module is used to adjust the action probability distribution sequence according to time series smoothing filtering if there are abnormal peaks in the action probability distribution sequence, so as to obtain a smooth action probability distribution sequence.

[0078] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 5 , Figure 5 This is a basic structural block diagram of a computer device according to an embodiment of this application.

[0079] Computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected via a system bus. It should be noted that only computer device 300 with components 310-330 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0080] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0081] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 310 may include both internal storage units and external storage devices of the computer device 300. In the embodiments of this application, the memory 310 is typically used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for a proxy detection method. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0082] In some embodiments, processor 320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 320 is typically used to control the overall operation of computer device 300. In embodiments of this application, processor 320 is used to execute computer-readable instructions stored in memory 310 or process data, such as executing computer-readable instructions for a substitution detection method.

[0083] The network interface 330 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 300 and other electronic devices.

[0084] The computer device provided in this application, by fusing lip-sync features and lip-sounding features, comprehensively utilizes multi-dimensional information from audio and video, enabling it to capture the features of proxy answering behavior more comprehensively and accurately. Compared to traditional single-feature detection methods, it significantly improves the accuracy of proxy answering detection and reduces false positives and false negatives. Furthermore, the multi-scale temporal convolutional network can capture feature changes at different time scales and exhibits robustness to environmental noise, lighting variations, and other factors. Even in complex real-world scenarios, it can effectively perform action segmentation and classification, ensuring the stability and reliability of the proxy answering detection system.

[0085] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the substitution detection method described above.

[0086] The computer-readable storage medium provided in this application, by fusing lip-sync features and lip-sounding features, comprehensively utilizes multi-dimensional information from audio and video, enabling it to capture the features of proxy answering behavior more comprehensively and accurately. Compared with traditional single-feature detection methods, it significantly improves the accuracy of proxy answering detection and reduces false positives and false negatives. Furthermore, the multi-scale temporal convolutional network can capture feature changes at different time scales and exhibits robustness to environmental noise, lighting changes, and other factors. Even in complex real-world scenarios, it can effectively perform action segmentation and classification, ensuring the stability and reliability of the proxy answering detection system.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0088] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for detecting proxy responses, characterized in that, Includes the following steps: Receive proxy response detection requests carrying raw audio and video data; The original audio and video data are processed by feature extraction to obtain the joint lip-phonetic feature vector and the lip-phonetic synchronization feature vector; The joint lip-consonant feature vector and the lip-consonant synchronization feature vector are concatenated to obtain a fused feature vector. The fused feature vector is processed by a multi-scale temporal convolutional network to perform action segmentation, resulting in an action probability distribution sequence with the same time step length as the input. The proxy response detection result is obtained by performing proxy response identification processing on the probability distribution sequence of the actions according to the classifier.

2. The method for detecting substitution according to claim 1, characterized in that, The step of performing feature extraction processing on the original audio and video data to obtain the joint lip-phone feature vector and the lip-phone synchronization feature vector specifically includes the following steps: The audio and video data are processed by audio and video segmentation to obtain video and audio waveform data of the mouth area; The pre-trained multimodal self-supervised learning model is invoked, wherein the multimodal self-supervised learning model includes a deep residual network and a feedforward network; The deep residual network is used to extract lip features from the mouth region video to obtain temporal lip feature data. The audio waveform data is processed by the feedforward network to extract spectral features, thereby obtaining speech spectral feature data. The audio-video fusion processing of the lip temporal feature data and the speech spectral feature data yields the joint lip-speech feature vector. The lip timing feature data and the speech spectrum feature data are subjected to differential synchronization processing to obtain the lip-speech synchronization feature vector.

3. The method for detecting substitution according to claim 2, characterized in that, The step of performing differential synchronization processing on the lip timing feature data and the speech spectrum feature data to obtain the lip-speech synchronization feature vector specifically includes the following steps: The absolute difference between the lip timing feature data and the speech spectrum feature data at the frame level is calculated to obtain the lip-speech synchronization feature vector.

4. The substitution detection method according to claim 2, characterized in that, The step of performing differential synchronization processing on the lip timing feature data and the speech spectrum feature data to obtain the lip-speech synchronization feature vector specifically includes the following steps: Long Short-Term Memory (LSTM) networks are used to process the temporal dependencies of the lip temporal feature data and the speech spectral feature data to determine the preliminary co-matching scores between sequences. If the initial collaborative matching score is lower than the preset threshold, adjustments are made to the time alignment deviation between sequences. The lip-sync feature vector is obtained by calculating the deviation compensation value.

5. The method for detecting proxy answers according to claim 1, characterized in that, After the step of performing action segmentation processing on the fused feature vector using a multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same time step length as the input, the following step is also included: If there are abnormal peaks in the action probability distribution sequence, the action probability distribution sequence is adjusted according to time series smoothing filtering to obtain a smooth action probability distribution sequence.

6. The method for detecting proxy answers according to claim 1, characterized in that, Following the step of receiving the proxy detection request carrying the original audio and video data, the following steps are also included: The original audio data is subjected to noise reduction processing; and / or The original audio data is then processed by frame segmentation.

7. A substitute response detection device, characterized in that, include: The request acquisition module is used to receive proxy detection requests carrying raw audio and video data; The feature extraction module is used to perform feature extraction processing on the original audio and video data to obtain the joint lip-phonetic feature vector and the lip-phonetic synchronization feature vector; The vector concatenation module is used to perform vector concatenation processing on the joint lip-phoneme feature vector and the lip-phoneme synchronization feature vector to obtain a fused feature vector. The action segmentation module is used to perform action segmentation processing on the fused feature vector according to the multi-scale temporal convolutional network to obtain an action probability distribution sequence with the same input time step length. The proxy answering module is used to perform proxy answering processing on the probability distribution sequence of the actions according to the classifier to obtain proxy answering detection results.

8. The substitution detection device according to claim 7, characterized in that, The feature extraction module includes: The audio and video segmentation module is used to perform audio and video segmentation processing on the audio and video data to obtain video and audio waveform data of the mouth area; The model invocation submodule is used to invoke a pre-trained multimodal self-supervised learning model, wherein the multimodal self-supervised learning model includes a deep residual network and a feedforward network; The lip feature extraction submodule is used to perform lip feature extraction processing on the mouth region video according to the deep residual network to obtain lip temporal feature data; The spectrum feature extraction submodule is used to perform spectrum feature extraction processing on the audio waveform data based on the feedforward network to obtain speech spectrum feature data. The audio-video fusion submodule is used to perform audio-video fusion processing on the lip temporal feature data and the speech spectrum feature data to obtain the joint lip-speech feature vector. The differential synchronization submodule is used to perform differential synchronization processing on the lip timing feature data and the speech spectrum feature data to obtain the lip-speech synchronization feature vector.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the substitution detection method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the substitution detection method as described in any one of claims 1 to 6.

Images