A method for detecting spoken voices in remote proctoring

Abstract

This invention discloses a speech detection method for remote proctoring, comprising: collecting raw audio data containing various environmental noises according to the remote proctoring scenario, and labeling the raw audio data according to the start and end times of the speech to obtain a labeled audio dataset; using the labeled audio dataset, training a student network model with knowledge distillation using a pre-trained speech endpoint detection teacher model on a large-scale corpus to obtain a trained speech endpoint detection model; and using the trained speech endpoint detection model to process the real-time audio stream to obtain speech endpoint detection results. This invention can segment and connect contextual information in the input audio stream to achieve real-time detection and ensure the coherence of sub-audio segments.

Description

Technical Field

[0001] This invention belongs to the field of sound detection technology, and in particular relates to a method for detecting speaking voices for remote proctoring. Background Technology

[0002] Remote proctoring refers to the process of supervising and preventing cheating by remote participants through internet technology and proctoring software. It aims to ensure the fairness and integrity of examinations and is particularly suitable for remote learning and remote work scenarios. However, in practice, problems such as poor signal, insufficient hardware, and system crashes still trouble candidates and affect the smooth conduct of the exam. Noise and other complexities can cause misjudgments by proctoring software or platforms, leading to questions about the fairness of the exam and the authenticity of candidates' scores.

[0003] Early methods of speech detection relied primarily on analog circuits and devices, such as microphones, amplifiers, and filters, to focus on the physical characteristics of sound, such as volume, frequency, and waveform. With the development of digital technology, sound signals were converted into digital formats for processing, and digital signal processing (DSP) techniques were used for more complex sound analysis, such as Fourier transforms and spectral analysis. This period also saw the emergence of digital recording devices, making sound storage and analysis more convenient. In the late 20th and early 21st centuries, computers began to understand and process human speech, and speech recognition technology, including sound signal preprocessing, feature extraction, pattern matching, and language models, began to emerge. In recent years, with the development of artificial intelligence, sound detection technology has been significantly improved. Deep learning models, especially convolutional neural networks and recurrent neural networks (RNNs), have performed exceptionally well in sound recognition and classification tasks, enabling more accurate identification of whether someone is speaking and recording of the content of speech.

[0004] Despite significant advancements in speech detection technology, several challenges remain, particularly in remote proctoring. Specifically: environmental noise, such as traffic, pet barking, or music, often interferes with speech detection, impacting recognition accuracy; in noisy environments, effectively distinguishing between examinees and background voices is difficult; despite continuous improvements in hardware performance, real-time processing of audio data and rapid response remain challenging; furthermore, while trained models may perform well on specific datasets, their generalization ability is often limited when faced with new or different datasets, potentially leading to unsatisfactory performance; high-performance speech recognition systems, while effective, have significant computational resource requirements that become a considerable issue in mobile devices or resource-constrained environments; maintaining accuracy and consistency in processing long periods of audio data is also a major challenge; and current methods often neglect cross-linguistic and accent diversity, necessitating the design of personalized algorithms to ensure speech detection performance. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a speaking voice detection method for remote proctoring. This method, on the one hand, segments and connects contextual information in the input audio stream to achieve real-time detection and ensure the continuity of sub-audio. On the other hand, it builds a deep learning model for speaking voice detection to extract and learn human speech features from the original audio with noise, so as to accurately and quickly output the time domain of the target human voice in the audio.

[0006] To achieve the above objectives, the present invention provides a method for detecting speaking voices during remote proctoring, comprising: Raw audio data containing various environmental noises was collected based on the remote proctoring scenario, and the raw audio data was labeled according to the start and end times of human voices to obtain a labeled audio dataset. Based on the labeled audio dataset, the student network model is trained by knowledge distillation using a speech endpoint detection teacher model pre-trained on a large-scale corpus to obtain the trained speech endpoint detection model. The trained speech endpoint detection model is used to process real-time audio streams and obtain speech endpoint detection results.

[0007] Optionally, raw audio data containing various environmental noises is collected based on the remote proctoring scenario, and the raw audio data is labeled according to the start and end times of the human voices to obtain a labeled audio dataset, including: Audio data is collected in office, classroom, library, and residential environments using recording equipment; For the human voice segments in the audio data, the start and end times of each speech segment are labeled according to the preset speech segment boundary standards, and the labeling results are stored in array form to obtain a labeled audio dataset.

[0008] Optionally, before training on knowledge distillation from the labeled audio dataset, the following steps are also included: The tagged audio data is subjected to DC removal processing, which iteratively eliminates the DC component in the signal by combining first-order differential filtering and low-pass filtering. The audio data after DC removal is windowed, and the signal is weighted using the Hamming window function to smoothly transition the signal to zero at the cutoff point to reduce spectral leakage.

[0009] Optionally, the student network model can be trained using knowledge distillation with a pre-trained speech endpoint detection teacher model on a large-scale corpus to obtain a trained speech endpoint detection model, including: The teacher model is used to predict the labeled audio dataset, and soft labels are generated by introducing a softmax function with a temperature parameter. Construct a student network model; Using the soft labels and the hard labels in the labeled audio dataset, the student network model is jointly trained by a weighted overall loss function combining binary cross-entropy loss and KL divergence loss to obtain the trained speech endpoint detection model.

[0010] Optionally, the student network model includes: a feature extraction layer, an LSTM layer, a time-distributed fully connected layer, and a Sigmoid activation function layer; The feature extraction layer is used to extract Mel frequency cepstral coefficient features from the input audio signal frame, wherein the audio signal frame is the basic component of the original audio data; The LSTM layer is used to capture the temporal dependencies in the Mel frequency cepstral coefficient features; The time-distributed fully connected layer is used to map the high-dimensional features of each time step into a low-dimensional feature vector according to the temporal dependency relationship. The Sigmoid activation function layer is used to map the output of the temporally distributed fully connected layer to the probability value that each time frame contains speech.

[0011] Optionally, the trained speech endpoint detection model can be used to process the real-time audio stream to obtain speech endpoint detection results, including: The trained speech endpoint detection model is deployed to the proctoring terminal to segment the real-time audio stream, obtain short audio segments, and pass contextual information between adjacent short audio segments. Based on the context information, the continuity of the speech segments at the cutting edge is determined, and the speech endpoint detection result is output.

[0012] Optionally, the real-time audio stream is segmented to obtain short audio clips, and context information is passed between adjacent short audio clips, including: The audio stream is cut into segments according to a preset time interval to obtain several short audio clips; During the inference process of each short audio segment, the hidden state of the last data block of the current short audio segment is passed as context information to the first data block of the next short audio segment, so as to realize the continuation and inheritance of context information.

[0013] Optionally, determining the continuity of the speech segments at the cutting edge based on the context information includes: When the cutting position is inside a speech segment, the duration of the edge voice at the end of the previous short audio segment is passed to the next short audio segment through context information; If there is human voice at the beginning of the next short audio segment, the duration of the human voice at the edge of the two short audio segments will be combined and counted. If the combined duration exceeds the minimum speech duration threshold, the edge parts of the two short audio segments will be determined as speech. If the duration of human voice at the edge of either of the two short audio segments exceeds the minimum speech duration threshold, then the edge portions of both short audio segments will be classified as speech.

[0014] Compared with the prior art, the present invention has the following advantages and technical effects: 1. This invention designs a deep learning neural network that combines a data-driven speech endpoint detection teacher model and a lightweight student model. The teacher pre-trained model is trained on three million hours of audio data, covering multiple languages ​​and various speaking scenarios, which is sufficient to ensure the accuracy and generality of the model. Through knowledge distillation, the student model maintains a basically unchanged performance while significantly reducing the parameter size, which creates conditions for subsequent deployment on remote examination terminals or mobile devices.

[0015] 2. During training, this invention uses a high proportion of short audio segments to enhance the model's detection accuracy for phrase-level audio. Subsequently, during deployment, a carefully designed algorithm is used to segment long audio segments while ensuring continuity. This ensures that the speech detection method of this invention has no weaknesses in detecting both short and long audio segments, and also achieves the key performance goal of real-time detection.

[0016] 3. This invention is adaptable to the differences in speech among different individuals. Through personalized algorithms, some factors that determine the inference effect of the model can be adjusted to achieve dynamic improvement in speech detection performance. This is of great significance for remote proctoring to avoid various external interferences during the exam. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the student model structure according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the knowledge distillation training process of the teacher model on the student network model according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the speech classification algorithm according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the detection results according to an embodiment of the present invention; Figure 5This is a flowchart of a speech sound detection method for remote proctoring according to an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] This embodiment proposes a method for detecting spoken voices during remote proctoring, such as... Figure 5 As shown, the specific steps include: Raw audio data containing various environmental noises was collected based on the remote proctoring scenario, and the raw audio data was labeled according to the start and end times of human voices to obtain a labeled audio dataset. Based on the labeled audio dataset, the student network model is trained by knowledge distillation using a speech endpoint detection teacher model pre-trained on a large-scale corpus to obtain the trained speech endpoint detection model. The trained speech endpoint detection model is used to process real-time audio streams and obtain speech endpoint detection results.

[0021] Specifically, this embodiment aims to address the three major challenges in speech detection—noise interference, inability to detect in real time, and lack of personalization—while ensuring accuracy and efficiency, providing a speech detection method for remote proctoring. This method, on the one hand, segments and connects contextual information from the input audio stream to achieve real-time detection and ensure the coherence of sub-audio segments; on the other hand, it builds a deep learning model for speech detection to extract and learn human speech features from the noisy original audio, accurately and quickly outputting the time domain of the target human voice in the audio. Furthermore, considering the diversity of language accents, a personalized algorithm is designed to dynamically adjust the key parameters of this method, further enhancing the versatility and robustness of speech detection. This invention is applicable to remote proctoring scenarios with limited hardware resources and high efficiency requirements for anti-cheating measures. The speech detection method can be used to supervise adaptive examinations, which perfectly aligns with the need for candidates to adhere to examination rules even without supervision.

[0022] Furthermore, raw audio data containing various environmental noises is collected based on the remote proctoring scenario, and the raw audio data is labeled according to the start and end times of human voices to obtain a labeled audio dataset, including: Audio data is collected in office, classroom, library, and residential environments using recording equipment; For the human voice segments in the audio data, the start and end times of each speech segment are labeled according to the preset speech segment boundary standards, and the labeling results are stored in array form to obtain a labeled audio dataset.

[0023] Specifically, step one involves obtaining the audio dataset for the remote proctoring scenario and labeling it based on the start and end times of the voices: A. Use microphones, radios, mobile phones, or computer recording devices to collect audio data. The scenarios should include offices, classrooms, libraries, residences, etc. B. Label the human voice parts in the audio according to the labeling format. If there are multiple human voices in an audio segment, each segment needs to be labeled and finally represented in array form. C. To ensure real-time performance, increase the proportion of short audio data of about 3 seconds in the training set.

[0024] Furthermore, prior to training using knowledge distillation on the labeled audio dataset, the following steps are also included: The tagged audio data is subjected to DC removal processing, which iteratively eliminates the DC component in the signal by combining first-order differential filtering and low-pass filtering. The audio data after DC removal is windowed, and the signal is weighted using the Hamming window function to smoothly transition the signal to zero at the cutoff point to reduce spectral leakage.

[0025] Specifically, the raw audio dataset obtained in step one is preprocessed: A. Removing DC from audio data eliminates offsets caused by bias in electronic devices or non-zero long-term average values, which helps distinguish between signals and noise; B. Window the audio data to smooth the signal of each frame and reduce spectral leakage during the Fourier transform process.

[0026] Furthermore, the student network model is trained using knowledge distillation with a pre-trained speech endpoint detection teacher model on a large-scale corpus, resulting in a trained speech endpoint detection model including: The teacher model is used to predict the labeled audio dataset, and soft labels are generated by introducing a softmax function with a temperature parameter. Construct a student network model; Using the soft labels and the hard labels in the labeled audio dataset, the student network model is jointly trained by a weighted overall loss function combining binary cross-entropy loss and KL divergence loss to obtain the trained speech endpoint detection model.

[0027] Specifically, in step three, a speech endpoint detection model trained on a large-scale corpus is used as the teacher layer to perform knowledge distillation, achieving phrase-level speech sound detection: A. Design the silent-Hi speech endpoint detection model; B. The model in step A acts as a teacher to predict the training data and generate soft labels; C. Design the student network model timi-vad according to the requirements, where students learn both hard labels (real labels) and soft labels simultaneously; D. Define a loss function to achieve joint training, which combines the cross-entropy loss for hard labels and the cross-entropy loss for soft labels; E. Design fine-tuning parameters to further optimize model performance and lay the groundwork for designing personalized algorithms in subsequent deployments.

[0028] Furthermore, the student network model includes: a feature extraction layer, an LSTM layer, a time-distributed fully connected layer, and a Sigmoid activation function layer; The feature extraction layer is used to extract Mel frequency cepstral coefficient features from the input audio signal frame, wherein the audio signal frame is the basic component of the original audio data; The LSTM layer is used to capture the temporal dependencies in the Mel frequency cepstral coefficient features; The time-distributed fully connected layer is used to map the high-dimensional features of each time step into a low-dimensional feature vector according to the temporal dependency relationship. The Sigmoid activation function layer is used to map the output of the temporally distributed fully connected layer to the probability value that each time frame contains speech.

[0029] Furthermore, the trained speech endpoint detection model is used to process the real-time audio stream to obtain speech endpoint detection results, including: The trained speech endpoint detection model is deployed to the proctoring terminal to segment the real-time audio stream, obtain short audio segments, and pass contextual information between adjacent short audio segments. Based on the context information, the continuity of the speech segments at the cutting edge is determined, and the speech endpoint detection result is output.

[0030] Specifically, step four involves the actual deployment of real-time data inference for the audio stream: A. Convert the trained timi-vad model into ONNX model format; B. The audio stream (long-duration audio) is segmented every 3 seconds, meaning that inference is performed every 3 seconds to ensure real-time detection of spoken sound; C. In order to maintain the continuity between the short audio segments after segmentation, the contextual information is continued and inherited; D. Design personalized algorithms to adjust model parameters to adapt to the diversity of different users.

[0031] E. To address issues with abnormal voice detection at the cutting edge, further ensuring contextual coherence.

[0032] Furthermore, the real-time audio stream is segmented to obtain short audio clips, and contextual information is passed between adjacent short audio clips, including: The audio stream is cut into segments according to a preset time interval to obtain several short audio clips; During the inference process of each short audio segment, the hidden state of the last data block of the current short audio segment is passed as context information to the first data block of the next short audio segment, so as to realize the continuation and inheritance of context information.

[0033] Furthermore, determining the continuity of speech segments at the cutting edge based on the context information includes: When the cutting position is inside a speech segment, the duration of the edge voice at the end of the previous short audio segment is passed to the next short audio segment through context information; If there is human voice at the beginning of the next short audio segment, the duration of the human voice at the edge of the two short audio segments will be combined and counted. If the combined duration exceeds the minimum speech duration threshold, the edge parts of the two short audio segments will be determined as speech. If the duration of human voice at the edge of either of the two short audio segments exceeds the minimum speech duration threshold, then the edge portions of both short audio segments will be classified as speech.

[0034] The following describes this embodiment in detail with reference to the accompanying drawings. This embodiment mainly consists of four steps: Step 1: Obtain the audio dataset for the remote proctoring scenario and label it according to the start and end times of the voices: A. First, based on the purpose and objectives of the dataset, it is roughly divided according to criteria such as the sounds made by test takers during the process and the presence of noise in the examination environment. Next, a data collection plan is designed, including selecting appropriate hardware devices such as microphones and recording equipment, as well as software tools such as remote proctoring platforms and audio processing software. Then, different scenarios are considered, such as offices, classrooms, libraries, and residences, where the ambient noise levels vary when these scenarios are set as examination rooms. B. Label the dataset from sub-step A, ensuring it includes the start and end times of each speech segment. The specific format is {start time: XX.XX.XX, end time: YY.YY.YY}. To avoid variations in speech intervals and segment divisions, a unified standard is established: speech intervals should be within 32ms; intervals exceeding 32ms indicate the speech has entered a silent phase, and silence exceeding 100ms signifies the end of the speech segment. If an audio clip contains multiple speech segments, label each segment strictly according to this standard, and finally represent them as an array. The C.vad model works by inferring the speech endpoints (start and end times) from a complete audio data segment. However, in actual proctoring, data recording continues until the end of the entire exam and is streamed in. This long-duration data lacks real-time performance requirements. Therefore, during training, the amount of short-duration audio data should be increased to improve the model's real-time inference capabilities. Furthermore, to better handle complex situations such as noise and multiple speakers in various remote proctoring environments, the sample design should reflect these factors.

[0035] Step 2: Preprocess the raw audio dataset obtained in Step 1 (Note that this preprocessing operation only applies to the student model, as the teacher model already includes these preprocessing steps): A. Since the acquisition devices are all electronic products, it is necessary to remove DC from the audio data to eliminate the offset caused by the bias of the electronic equipment or the non-zero long-term average value. This helps to distinguish between signal and noise. Essentially, it is done by combining first-order differential filtering and low-pass filtering to continuously and iteratively eliminate the DC component in the signal. The formula for removing DC is as follows: s_new(n)=s_old(n)-s_old(n-1)+0.9997 s_new(n-1); Where s_new(n) represents the value of the current frame after removing the DC component, s_old(n) represents the original value of the current frame, s_old(n-1) represents the original value of the previous frame, s_new(n-1) represents the value of the previous frame after removing the DC component, and 0.9997 is a filter coefficient used for smoothing. This formula effectively removes the DC offset from the signal by calculating the difference between the current frame and the previous frame and using a filter coefficient to update the value after removing the DC component.

[0036] B. The Hamming window is a commonly used window function. Its principle is based on a simple cosine function, used to weight data in signal processing. Its main function is to truncate the signal in the time domain, allowing the signal to smoothly transition to zero at the cutoff point, rather than abruptly stopping. This gradual reduction decreases spectral leakage and improves spectral resolution. The formula for the Hamming window is as follows: w(n) = 0.54 - 0.46 cos(2πn / (N-1)); Where w(n) is the value of the window function, corresponding to the nth sample, N is the total length of the window function (i.e., the frame size), n is the sample index, ranging from 0 to N-1, and π is pi, approximately equal to 3.14159. This formula means that the weight of each sample is 0.54 minus 0.46 multiplied by the value of the cosine function. The input to the cosine function is the ratio of the sample index to the window function length multiplied by 2π. In this way, each sample is assigned a weight between 0.08 and 1, causing the window function to gradually decrease to zero at both ends.

[0037] Step 3: Utilize the speech endpoint detection model trained on a large-scale corpus as the teacher layer to perform knowledge distillation, achieving phrase-level speech sound detection: A. First, build the student model timi-vad. This model needs to be simple in structure while being able to capture the temporal dependencies in audio features. The structure diagram is as follows. Figure 1 As shown: The input to timi-vad is an audio signal frame; during feature extraction, Mel frequency cepstral coefficients (MFCCs) are extracted from the original data using steps such as Mel filter and discrete cosine transform. This step can highlight useful information and suppress noise and non-speech components; LSTM Layer 1 to LSTM Layer 3 represent multiple LSTM layers, and finally each LSTM memory unit is skipped to a fully connected layer to capture long temporal dependencies; TimeDistributedDense is a fully connected layer applied at each time step; finally, the Sigmoid activation function is used because this is a binary classification problem, and the probability prob(Voice / No-Voice) is the final output of the model, indicating whether each time frame contains speech.

[0038] B. The corresponding teacher model is silero-vad, a pre-trained, enterprise-grade speech activity detector that excels in speech detection tasks, boasting leading detection accuracy. It is trained on a massive corpus containing over 6000 languages ​​and performs well on audio from various domains with varying background noise and audio quality levels. It fully leverages the rich ecosystem of PyTorch and ONNX, enabling it to run in diverse environments. Processing time on a single-core CPU is less than 1ms. Performance can be further improved using batch processing or a GPU.

[0039] C. Knowledge distillation of timi-vad based on silero-vad: The silvero-vad dataset is used to train the timi-vad until it reaches a high performance level. Then, the trained teacher network is used to predict the training data, and the output is processed through a softmax function. A temperature parameter is introduced to adjust the smoothness of the output to generate soft labels. The weights of timi-vad are randomly initialized. During training, not only the true class labels of each sample are used, but also the soft labels generated by the teacher network are considered. The framework diagram is as follows. Figure 2 As shown.

[0040] D. Define a loss function to achieve joint training; The loss function needs to consider two parts: one is the loss between the student network output and the hard label, and the other is the loss between the student network output and the soft label.

[0041] True Label Loss: This part of the loss function measures the difference between the student's network output and the true label. For binary classification problems, binary cross-entropy loss (BCE) is used. ; in, It is a real label (0 or 1). It is the probability that the student's network prediction is speech.

[0042] Soft label loss: This part of the loss function measures the difference between the soft labels of the student network output and the teacher network output. It is calculated using KL divergence. ; in It's a soft label for teachers' networks. This is the output of the student network.

[0043] The overall loss function combining the two losses: Combining the two losses mentioned above forms an overall loss function, which is ultimately a weighted sum of the two losses. ; in, and These are hyperparameters used to adjust the relative importance of the two losses in the overall loss. In the early stages of training, the distillation loss can be given a higher weight so that the student network can learn the behavior of the teacher network more quickly; in the later stages of training, the weight of the true label loss can be gradually increased.

[0044] The design fine-tunes parameters to further optimize model performance and lays the groundwork for designing personalized algorithms in subsequent deployments.

[0045] The purpose of designing fine-tunable parameters is to lay the groundwork for designing personalized algorithms in the next step. Due to differences in speech patterns between individuals, even the best-trained model may exhibit poor detection performance among certain groups. Therefore, additional fine-tunable parameters are designed to enhance the model's flexibility in the inference pre / post-processing stages. Specific parameters and functions include, but are not limited to: `threshold`: Outputs the probability of speech for each audio segment; probabilities higher than this value are considered speech. This parameter can be adjusted individually for each dataset, but 0.5 works well for most datasets. min_speech_duration_ms: Speech segments shorter than min_speech_duration_ms will be discarded; `max_speech_duration_s`: The maximum duration (in seconds) of a speech segment. Speech segments exceeding `max_speech_duration_s` will be segmented at the last silence (if any) if the duration exceeds 100ms, to avoid overly aggressive segmentation. Otherwise, aggressive segmentation will occur when the duration approaches `max_speech_duration_s`. min_silence_duration_ms: At the end of each speech block, it is necessary to wait for min_silence_duration_ms before separating it; speech_pad_ms: The last speech block is padded on both sides with speech_pad_ms; A. Step Four: Implementing Real-Time Data Inference for the Audio Stream in Actual Deployment: In order to be deployed on commonly used electronic devices, the ONNX model format is generally required. Therefore, the student model trained in step three (default PT format) needs to be converted to ONNX. B. In step one, the model was trained with a focus on short audio to enhance its detection performance. However, real-time inference for audio streams (long audio) is still not possible. To overcome this difficulty, the idea of ​​cutting the audio stream (long audio) into several 3-second short audio segments (specifically, the shortest audio time that the model can accept can be evaluated first, and the segmentation length can even be accurate to the byte in the actual segmentation process) is proposed. The model will perform an inference task every 3 seconds to ensure real-time detection of speaking sounds. C. In step three, the student model was built using LSTM memory units to capture temporal information. In terms of inference details, the audio was called once for inference in units of 256 bytes (one data block) as reference to the teacher model, and context information parameters were designed. These operations were all to ensure the coherence between data blocks. However, due to the segmentation operation in step one, the originally coherent audio was divided into independent sub-audio, which led to differences in the results before and after the segmentation. In order to restore the coherence between sub-audio, it is necessary to add continuation and inheritance operations to the context information during the interval of calling the model. Continuation: Pass the context information of the last data block in this segment to the next sub-audio; Inheritance: Receives context information from the previous data block and passes in the first data block; The D.vad model's output speech endpoint is essentially a classification problem; that is, it outputs a confidence probability for each data block to distinguish whether it is a human voice or not. Typically, by setting a unique threshold and comparing the probability to the threshold, each data block can be easily classified as either human or not. However, this classification does not consider common-sense rules of speech or the differences in speech patterns between individuals. Therefore, considering the fine-tuning parameters in step three, a personalized algorithm is designed to achieve correct classification of each audio target. The algorithm flow is as follows: Figure 3 As shown: Condition 1: speech_prob >= threshold and temp_end: • The current frame has a speech probability greater than or equal to the threshold and contains a temporary end marker; • Handling: Reset temp_end (i.e., clear the temporary end marker); Condition 2: speech_prob >= threshold and not triggered: • The current frame has a speech probability greater than or equal to the threshold and has not yet triggered a speech segment; • Processing: Set triggered to True and record the start time (start of a new audio segment). Condition 3: Triggered and exceeds max_speech_samples: • The triggered speech segment and the current speech segment length exceeds the maximum limit; • Processing: Process the end time of current_speech (force to end the current speech segment); Condition 4, speech_prob < neg_threshold and triggered: • The current frame's speech probability < negative threshold and the speech segment has been triggered; • Processing: Process temp_end (detect the possible end of the speech segment, but need to meet a certain silence duration).

[0046] The key parameters are: temp_end: Used to track a temporary end point. In the context of speech detection, it may be used to mark a potential end point of the speech, which may be confirmed or reset in subsequent processing.

[0047] triggered: Used to indicate whether the start of a speech event has been detected. When a speech event is detected, this variable is set to True, indicating that the detection of a speech event has been triggered.

[0048] prev_end: Used to store the end time of the previous speech event. When processing consecutive speech events, this variable can help determine the start time of the new speech event.

[0049] next_start: Used to store the start time of the next possible speech event. When processing speech events, this variable can help determine the end time of the current speech event and the start time of the next speech event.

[0050] max_speech_samples: Defines the maximum length of a speech segment. During speech detection, if the length of a speech segment exceeds this value, then it is determined as an independent speech segment, even if there is a short silence in the middle.

[0051] min_speech_samples: Defines the minimum length of a speech segment. If the length of a speech segment is less than this value, it will not be recognized as a valid speech segment and may be regarded as background noise or other non-speech signals.

[0052] min_silence_samples: Defines the minimum length of silence. In speech detection, when a speech segment is detected, if the subsequent silence length exceeds this value, then the current speech segment is considered to end and a new speech segment can start.

[0053] The final three key parameters are determined through fine-tuning in step three. Therefore, by dynamically adjusting these parameters to suit the actual speaking conditions of different test takers, their values ​​can be influenced, thereby making the algorithm adaptable to the target and improving detection accuracy.

[0054] E. Addressing anomalies in speech detection at the cut edges to further ensure contextual coherence. Specifically, this can be described as follows: Before cutting, a certain time interval [A,B] in the audio is identified as someone speaking. The cutting operation then splits this speech segment into two sub-segments, [A,C] and [C,B]. To maintain consistency with the overall audio before cutting, these two sub-segments should ideally be identified as someone speaking. However, the results may differ because the speech determination in sub-step D requires a duration exceeding `min_speech_samples`, while the duration of the cut subset may be less than the minimum length. Anomalies (in red) include... Figure 4 As shown: In branch one, new contextual information is constructed. The duration of the edge vocals at the end of the previous sub-audio is input into the next audio. If the next audio has edge vocals from the beginning, both are considered as a whole and their durations are counted together. If the duration exceeds the minimum length, they are considered to be both vocals. In branch two, contextual information is also used. This can produce a cascading effect: if either sub-segment exceeds the minimum length, both sub-segments are considered vocals. As for branch three, it is the normal case and does not require contextual assistance. In summary, these strategies can eliminate the influence of the cutting edges, thus ensuring the consistency of the results before and after cutting.

[0055] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting a speaking voice for remote invigilation, characterized by, include: Raw audio data containing various environmental noises was collected based on the remote proctoring scenario, and the raw audio data was labeled according to the start and end times of human voices to obtain a labeled audio dataset. Based on the labeled audio dataset, the student network model is trained by knowledge distillation using a speech endpoint detection teacher model pre-trained on a large-scale corpus to obtain the trained speech endpoint detection model. The trained speech endpoint detection model is used to process real-time audio streams and obtain speech endpoint detection results.

2. The method for detecting speaking voices for remote proctoring according to claim 1, characterized in that, Raw audio data containing various environmental noises was collected based on a remote proctoring scenario. The raw audio data was then labeled according to the start and end times of the human voices, resulting in a labeled audio dataset including: Audio data is collected in office, classroom, library, and residential environments using recording equipment; For the human voice segments in the audio data, the start and end times of each speech segment are labeled according to the preset speech segment boundary standards, and the labeling results are stored in array form to obtain a labeled audio dataset.

3. The method for detecting speaking voices for remote proctoring according to claim 1, characterized in that, Before training with knowledge distillation based on a labeled audio dataset, the following steps are also included: The tagged audio data is subjected to DC removal processing, which iteratively eliminates the DC component in the signal by combining first-order differential filtering and low-pass filtering. The audio data after DC removal is windowed, and the signal is weighted using the Hamming window function to smoothly transition the signal to zero at the cutoff point to reduce spectral leakage.

4. The method for detecting speaking voices for remote proctoring according to claim 1, characterized in that, The student network model is trained using knowledge distillation on a pre-trained speech endpoint detection teacher model on a large-scale corpus, resulting in a trained speech endpoint detection model including: The teacher model is used to predict the labeled audio dataset, and soft labels are generated by introducing a softmax function with a temperature parameter. Construct a student network model; Using the soft labels and the hard labels in the labeled audio dataset, the student network model is jointly trained by a weighted overall loss function combining binary cross-entropy loss and KL divergence loss to obtain the trained speech endpoint detection model.

5. A method for detecting speaking voices for remote proctoring according to claim 4, characterized in that, The student network model includes: a feature extraction layer, an LSTM layer, a time-distributed fully connected layer, and a Sigmoid activation function layer; The feature extraction layer is used to extract Mel frequency cepstral coefficient features from the input audio signal frame, wherein the audio signal frame is the basic component of the original audio data; The LSTM layer is used to capture the temporal dependencies in the Mel frequency cepstral coefficient features; The time-distributed fully connected layer is used to map the high-dimensional features of each time step into a low-dimensional feature vector according to the temporal dependency relationship. The Sigmoid activation function layer is used to map the output of the temporally distributed fully connected layer to the probability value that each time frame contains speech.

6. The method for detecting speaking voices for remote proctoring according to claim 1, characterized in that, The trained speech endpoint detection model is used to process real-time audio streams to obtain speech endpoint detection results, including: The trained speech endpoint detection model is deployed to the proctoring terminal to segment the real-time audio stream, obtain short audio segments, and pass contextual information between adjacent short audio segments. Based on the context information, the continuity of the speech segments at the cutting edge is determined, and the speech endpoint detection result is output.

7. A method for detecting speaking voices for remote proctoring according to claim 6, characterized in that, The process involves segmenting a real-time audio stream to obtain short audio clips and passing contextual information between adjacent short audio clips, including: The audio stream is cut into segments according to a preset time interval to obtain several short audio clips; During the inference process of each short audio segment, the hidden state of the last data block of the current short audio segment is passed as context information to the first data block of the next short audio segment, so as to realize the continuation and inheritance of context information.

8. A method for detecting speaking voices for remote proctoring according to claim 6, characterized in that, Determining the continuity of speech segments at the cutting edge based on the context information includes: When the cutting position is inside a speech segment, the duration of the edge voice at the end of the previous short audio segment is passed to the next short audio segment through context information; If there is human voice at the beginning of the next short audio segment, the duration of the human voice at the edge of the two short audio segments will be combined and counted. If the combined duration exceeds the minimum speech duration threshold, the edge parts of the two short audio segments will be determined as speech. If the duration of human voice at the edge of either of the two short audio segments exceeds the minimum speech duration threshold, then the edge portions of both short audio segments will be classified as speech.

Images