A method for distinguishing overlapping speech for oral examination scenarios

Abstract

This invention discloses a crosstalk discrimination method for oral examination scenarios, belonging to the field of speech intelligence technology. The method includes four steps: data preparation, feature extraction, training a discrimination model, and examination discrimination. A dataset is constructed by collecting audio from multiple examination scenarios, 80-dimensional Fbank features are extracted, and a lightweight deep learning model is used to output crosstalk confidence under mono audio conditions. Combined with threshold and multi-dimensional information for joint discrimination, accurate differentiation between target speech and crosstalk is achieved. This invention requires no additional hardware such as microphone arrays, has fewer model parameters, lower computational cost, and lower deployment cost. It can effectively suppress crosstalk interference, avoid misscoring, and significantly improve the accuracy, fairness, and robustness of oral examination scoring, making it suitable for large-scale standardized oral examination scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech technology, specifically a crosstalk detection method for oral examination scenarios. This method addresses complex application environments such as multiple speakers taking English oral exams simultaneously, using mono microphones with poor sound isolation, and automatically identifies and suppresses crosstalk from non-target speakers. This avoids misidentification and incorrect scoring caused by nearby examinees' voices being mistakenly captured when the speaker hasn't answered. This method can be applied to standardized oral exams and offline oral exams, improving the accuracy, fairness, and robustness of speech scoring. Background Technology

[0002] With the development of artificial intelligence technology, especially speech evaluation technology, large-scale oral exam scoring has become a reality. Oral exam systems are widely used in language education, standardized testing, and online assessment. In actual testing, especially in school computer labs or centralized testing environments, multiple candidates often take oral exams simultaneously in the same room. Due to limited microphone sound insulation, close proximity between candidates, or complex acoustic environments, the speech of non-target candidates can easily be mistakenly captured by adjacent devices, resulting in crosstalk interference.

[0003] Existing oral assessment systems typically assume that the collected speech signals originate from the current test-taker, focusing primarily on optimizing speech recognition accuracy and pronunciation quality scoring algorithms, while lacking mechanisms to verify the validity of the speech source. When the target test-taker is not speaking, the microphone may mistakenly record the speech of a nearby test-taker and use it as the current test-taker's response for scoring, leading to misidentification and misscoring. This not only affects the accuracy and fairness of the test results but may also negatively impact the credibility of test management and assessment systems.

[0004] Furthermore, under single-microphone or single-channel recording conditions, it is difficult to spatially distinguish sound sources using traditional array localization or multi-channel beamforming techniques, making crosstalk problems more prominent. Existing speech activity detection or simple energy thresholding methods are insufficient to accurately distinguish near-field target speech from far-field interfering speech, lacking robustness in complex examination environments. Therefore, how to accurately determine the distance and validity of speech sources under single-microphone conditions, thereby suppressing the interference of non-target speech on scoring results, is a pressing technical problem to be solved in this field.

[0005] Therefore, it is necessary to propose a crosstalk detection method for oral examination scenarios to improve the accuracy, reliability and fairness of the speech evaluation system in a multi-person examination environment, which has important technical significance and practical application value. Summary of the Invention

[0006] 1) This invention can automatically identify non-target speech input when the examinee does not speak, avoid the speech of nearby examinees being mistakenly included in the scoring results, effectively reduce the risk of misscoring, improve the authenticity and fairness of the examination results, and enhance the credibility of the examination system.

[0007] 2) Under mono recording conditions, this invention uses a deep learning-based neural network model to distinguish between the target candidate's speech and the crosstalk of neighboring candidates, without relying on microphone arrays or multi-channel hardware devices, thus reducing system deployment costs.

[0008] 3) The model adopts a lightweight model structure, which significantly reduces the model parameter size and computational complexity while ensuring accuracy. Compared with large-scale deep neural network models, it can effectively reduce memory usage and inference latency, enabling the system to run stably on ordinary PC terminals or low-computing-power examination terminals.

[0009] This method collects thousands of hours of exam audio from multiple online and offline exams as a training dataset and uses an Encoder+sigmoid neural network structure to train a discrimination model. Then, the actual exam audio is processed through this model to obtain the corresponding confidence score. Based on the confidence score, combined with the score distribution of other question types and multiple dimensions such as audio amplitude, the method determines whether the candidate's audio has crosstalk.

[0010] The technical solution of this invention is a method for detecting crosstalk in oral examination scenarios, comprising the following steps: Step 1, Data Preparation: Audio data was collected for different age groups and different regions across the country to ensure the diversity of the dataset, and the audio was cleaned to ensure the consistency of the audio format.

[0011] Step 2, Feature Extraction: In this method, the first step is feature extraction from the speech signal, which involves extracting discriminative acoustic information from the original audio and removing components with low relevance to the task. Compared to time-domain waveforms, speech exhibits more stable structural features in the frequency domain, with different speech units showing relatively clear distribution patterns across the spectrum, making it more conducive to analysis and modeling. Based on this, this method processes the speech signal in the frequency domain to extract Fbank features, which serve as input for subsequent model training and discrimination.

[0012] Step 3, train the discrimination model: A lightweight deep neural network model based on the Encoder + Sigmoid structure is constructed. The Encoder is used to perform temporal encoding on the input acoustic features to extract high-level speech representations; the Sigmoid layer has a value range of (0,1) and can be used as a probability output layer to output the confidence value of whether the exam audio has crosstalk.

[0013] The Encoder employs a convolutional neural network, a low-parameter self-attention layer, and a fully connected network layer, effectively reducing the model's parameter size and computational complexity. During training, labeled near-field effective speech and far-field crosstalk speech data are used for supervised training. A binary classification cross-entropy loss function is used as the optimization objective, and the model parameters are updated via backpropagation.

[0014] After training, the model can output a confidence value between 0 and 1 for the input mono audio test speech, whereby the confidence value represents the probability that the speech belongs to the target candidate's valid answer speech.

[0015] Step 4, Examination and Judgment: In the actual oral examination, the monophonic audio data collected by the system is subjected to acoustic feature extraction as described in step two. The extracted features are then input into the trained discrimination model to obtain the confidence value of the candidate's audio. The system then makes a preliminary judgment on the validity of the speech based on a preset confidence threshold, and combines this with a multi-dimensional information joint discrimination strategy to achieve real-time interception of crosstalk and suppression of misscoring, thereby improving the accuracy and fairness of the examination results.

[0016] Beneficial effects This method, designed for formal examinations, implements a crosstalk discrimination approach specifically for oral exams. Compared to solutions relying on microphone arrays, this method requires no additional hardware and can effectively distinguish between near-field target speech and far-field crosstalk under mono conditions. Furthermore, by employing a lightweight deep learning model, the model size is only 6MB while maintaining high accuracy, significantly reducing the model parameter scale and computational complexity. The impact on overall inference time is less than 5%, significantly lowering system deployment costs. This improves the accuracy and fairness of oral exam scoring.

[0017] In the offline mock exam in Dalian, a total of 50,289 students participated. After evaluation, 305 audio tracks with a confidence level < 0.7 were directly scored as 0 points. After manual verification, the accuracy rate was 100%. 286 audio tracks with a confidence level between 0.7 and 0.99 were evaluated. After analyzing the audio amplitude, recognition results, and overall student scores, 257 audio tracks were deemed abnormal. After teacher verification, 252 tracks were found to be unread or in the far field, resulting in an accuracy rate > 98%.

[0018] Similarly, in the offline examination in Liaoyang City, a total of 9,075 candidates were tested. After evaluation, 53 audio recordings with a confidence level of <0.7 were scored as 0, and 86 audio recordings with a confidence level between 0.7 and 0.99 were found to be abnormal after verification. The accuracy rate was about 99%, which significantly improved the situation of incorrect scoring. Attached Figure Description

[0019] Figure 1 This is a flowchart of the discriminant model's workflow in a formal examination scenario. Detailed Implementation

[0020] To verify this invention, we conducted multiple tests in online exams. The results also proved valid in offline exams. The method will now be described in further detail with reference to the accompanying drawings. Figure 1 As shown, it mainly includes the following steps.

[0021] Step 1, Data Preparation: Based on multiple online and offline oral examination scenarios, we collected and organized examination recording data from various age groups, regions, and examination venues to construct a training dataset containing near-field effective speech and far-field crosstalk speech. The data includes: normal answering speech of examinees, speech samples from neighboring examinees that have crosstalked in, and speech samples from silent / no-voice environments.

[0022] The collected data were manually labeled and categorized into three types based on their source and validity: "target near-field valid speech," "non-target far-field crosstalk speech," and "silent, noisy, and other voiceless speech." Valid speech was labeled as 1, while crosstalk and voiceless speech were labeled as 0. Simultaneously, the audio underwent a uniform sampling rate conversion and silence removal to ensure data consistency and trainability.

[0023] Step 2, Feature Extraction: The processed audio is pre-emphasized, framed, windowed, subjected to Short-Time Fourier Transform (STFT), mel-filtered, and mean-removed to obtain fbank features, which are then used as model input. To ensure computational efficiency and model lightweightness, the feature dimension is controlled at 80, and normalization is used to reduce the impact of differences between different environments.

[0024] Step 3, train the discrimination model: This method employs a lightweight neural network structure, reducing parameter size and computational complexity while maintaining discrimination accuracy, making it suitable for deployment on ordinary examination terminals or low-computing-power devices. Specific configurations include: 1. Input layer: The input is a two-dimensional acoustic feature matrix extracted from mono audio, with a feature dimension of 80. The time frame length is adaptively determined according to the duration of the audio.

[0025] 2. Encoder Layer: The encoder is used to extract high-level representations of temporal acoustic features. It employs a network structure with multiple sub-layers stacked together, where the structure of each sub-layer is as follows: 1) One-dimensional convolutional neural network (CNN); 2) Low-level self-attention coding layer.

[0026] 3) Fully connected layer The encoder has 8 layers to reduce model complexity; the hidden unit dimension is set to 128 dimensions; and layer normalization is used to improve training stability.

[0027] 3. Sigmoid output layer Set up a fully connected layer with a Sigmoid activation function, and output a confidence value between 0 and 1, representing the probability that the input speech belongs to the target near-field valid speech.

[0028] Step 4, Examination and Judgment: During the examination, the mono audio data collected by the system is processed for acoustic feature extraction as described in step two. The extracted features are then input into the discrimination model to obtain the confidence value of the current examinee's audio. The confidence value is between (0, 1), where a value closer to 0 indicates a higher probability that the audio is crosstalk / abnormal audio with no voice, while a value closer to 1 indicates normal audio. The system judges the validity of the speech based on the preset confidence threshold as follows: 1) When the confidence level is higher than 0.99, the speech is considered a valid response and is scored normally; 2) When the confidence level is below 0.7, the speech is determined to be invalid audio, which is either crosstalk or no voice from the target examinee; 3) When the confidence level is between 0.7 and 0.99, a joint judgment is made by combining the candidate's scores on other question types, the audio recognition results, and the overall amplitude of the audio. If the judgment is abnormal, a manual review is conducted.

[0029] To verify the effectiveness of this invention, online mock exams, offline standardized exam rooms, low-computing-power terminals, and multi-environment complex exam rooms were conducted. The specific implementation methods are as follows.

[0030] Example 1: Example of identifying crosstalk in a routine oral exam Step 1: Data Preparation. Based on multiple online and offline oral exam scenarios, we collected and organized exam recordings from various age groups, regions, and exam venues to construct a training dataset containing near-field effective speech and far-field crosstalk speech. The data includes: normal candidate responses, samples of speech from neighboring candidates interfering with the recording, and silent / no-voice audio samples from the recording environment. The collected data was manually labeled, with target effective speech marked as 1 and crosstalk and no-voice audio marked as 0. Simultaneously, the audio underwent uniform sampling rate conversion and silence removal to ensure data consistency and trainability.

[0031] Step 2: Feature Extraction. The processed audio is sequentially subjected to pre-emphasis, frame segmentation, windowing, short-time Fourier transform (STFT), Mel filtering, and mean removal to obtain 80-dimensional Fbank features, which are then normalized and used as model input.

[0032] Step 3: Train the discriminative model to construct a lightweight deep neural network with an Encoder + Sigmoid structure. The Encoder consists of 8 layers, composed of a one-dimensional CNN, a low-parameter self-attention layer, and a fully connected layer stacked together, with 128 hidden units and layer normalization. The output layer is connected to a Sigmoid activation function, outputting a confidence score of 0 to 1. A binary classification cross-entropy loss function is used, and supervised training is conducted using a labeled dataset. The model parameters are updated through backpropagation.

[0033] Step 4: Examination Judgment During the examination, 80-dimensional Fbank features are extracted from the mono audio and input into the model to obtain the confidence level: 1) Confidence level > 0.99, it is judged as valid answer speech and scored normally; 2) Confidence level < 0.7, it is judged as crosstalk / no voice invalid audio; 3) Confidence level 0.7~0.99, combined with the distribution of the candidate's scores for other question types and audio amplitude, is judged jointly. If abnormal, manual review is initiated.

[0034] This embodiment can effectively identify crosstalk audio, avoid misscoring, and operate stably under mono conditions.

[0035] Example 2: Standardized single-microphone computer lab examination example Step 1: Data Preparation. Approximately 2000 hours of audio data from computer lab exams for junior and senior high school students in multiple cities across China were collected, covering four categories: normal responses, crosstalk from seatmates, noisy computer lab environments, and silent environments. A uniform sampling rate of 16kHz was used, and silence was removed using VAD. Manual labeling was applied: 1 for valid speech and 0 for crosstalk / no voice.

[0036] Step 2: Feature Extraction. 80-dimensional Fbank features are extracted from all audio data using a unified process, and normalization is performed to ensure consistent feature distribution across different examination rooms.

[0037] Step 3: The model training uses an 8-layer encoder structure with 128 hidden dimensions, layer normalization, a 1D CNN, and low-parameter self-attention. After training, the model size is <10MB, and the inference time for a single audio clip is <20ms.

[0038] Step 4: Real-time evaluation of 1000 candidates in a single-microphone, mono computer lab environment: Confidence level > 0.99: Normal rating; Confidence level < 0.7: Directly deemed invalid and not scored; The middle range is determined by combining the score of the question type with the amplitude.

[0039] Implementation results: Crosstalk recognition accuracy ≥97.2%, false alarm rate reduced by 92%, no microphone array required, stable operation on ordinary PC terminals.

[0040] Example 3: Lightweight Deployment Example of Low Computing Power Terminals Step 1: Lightweight Model Processing. The trained crosstalk discrimination model is subjected to floating-point quantization and weight pruning, compressing the model parameters by 65% ​​and reducing memory usage to <15MB.

[0041] Step 2: The test environment is configured using a low-configuration testing terminal: CPU J1900, 4GB memory, no GPU, Windows / Linux system.

[0042] Step 3: Feature and Inference Optimization. The features are fixed at 80-dimensional Fbank, and redundant calculations are turned off; the Encoder maintains an 8-layer structure, and the self-attention calculation is reduced, resulting in a 50% reduction in single-frame calculation.

[0043] Step 4: Performance verification. Continuously process 1000 exam audio recordings. The average time for inference per recording is 12ms, CPU utilization is <10%, and the crosstalk detection accuracy is ≥95.8%.

[0044] Implementation results: It meets the requirements for offline real-time operation of low-computing-power terminals, reduces deployment costs by 70%, and is suitable for unified deployment in large-scale examination rooms.

[0045] Example 4: Multi-environment robust crosstalk discrimination example Step 1: Constructing a Multi-Scenario Test Set. Construct four types of test audio: a poorly soundproof computer room, a regular classroom, an open examination room, and a reverberant environment. Include speech data of different genders, speech rates, and signal-to-noise ratios.

[0046] Step 2: Preprocessing and feature extraction uniformly adopt VAD silence detection, 16kHz sampling, 80-dimensional Fbank feature extraction and normalization to ensure cross-environment feature consistency.

[0047] Step 3: Model Reasoning and Joint Judgment 1) Confidence > 0.99: Normal scoring; 2) Confidence < 0.7: Crosstalk is judged, and the score is set to 0; 3) Middle interval: Combine audio amplitude, zero crossing rate, and question type score dispersion for joint judgment.

[0048] Step 4: Results Statistics The crosstalk detection rate in the poorly soundproofed computer room was 96.5%, and in the reverberant classroom it was 94.7%, with an overall misjudgment rate of <2.1%.

[0049] Implementation results: It maintains high robustness in complex examination environments, prevents misjudgment of neighboring voices to the current examinee, and significantly improves the fairness of the examination.

[0050] Example 5: Mono Non-Array Pure Software Crosstalk Interception Example Step 1: Deployment conditions: All existing single-channel microphones in the examination room will be used. No new arrays or hardware upgrades will be added. Only the software module of this invention will be deployed.

[0051] Step 2: Data and Model A lightweight Encoder model is trained using labeled data from the exam scenario, with input being a mono 80-dimensional Fbank feature.

[0052] Step 3: Real-time discrimination process: Audio acquisition → Preprocessing → 80-dimensional F-bank extraction → Model inference → Confidence judgment → Scoring control. Confidence scores < 0.7 are directly blocked and not sent to the scoring engine; multi-dimensional verification is initiated for intermediate intervals.

[0053] Step 4: Implementation Results Under pure software, mono, and no hardware upgrade conditions, the crosstalk interception rate is ≥96%, the system deployment cost is reduced by more than 80%, and it is suitable for large-scale standardized examinations nationwide.

[0054] From the above process, we draw the following conclusions: 1) This method can effectively identify abnormal audio from crosstalk in test takers and improve the accuracy of test results; 2) This method does not require the use of microphone arrays or multi-channel devices and achieves the discrimination of crosstalk audio under mono audio conditions; 3) The model adopts a lightweight model structure, which ensures accuracy while enabling the system to run stably on low-computing-power servers.

Claims

1. A method for detecting crosstalk in oral examination scenarios, characterized in that, To distinguish between near-field target speech and far-field crosstalk speech under mono conditions, the following steps are included: 1) Data preparation: Collect oral test audio from different age groups, regions, and test environments, and construct a dataset that includes target near-field effective speech, non-target far-field crosstalk speech, silence, and environmental noise; The audio was manually annotated, with the target valid speech marked as 1 and crosstalk speech and no-voice audio marked as 0; the sampling rate was unified and silent segments were removed to ensure consistent data format. 2) Feature extraction: The preprocessed audio is subjected to pre-emphasis, frame segmentation, windowing, short-time Fourier transform, Mel filtering, and mean removal processing in sequence to extract 80-dimensional Fbank features and normalize them as model input; 3) Training the discriminant model: Construct a lightweight deep neural network with an "Encoder+Sigmoid" structure; The Encoder consists of a one-dimensional convolutional neural network, a low-parameter self-attention layer, and a fully connected layer, totaling 8 layers. The hidden unit dimension is 128, and layer normalization is used. The output layer uses a fully connected layer connected to a Sigmoid activation function, and outputs a confidence level between 0 and 1. The binary classification cross-entropy loss function is used, and supervised training is performed using a labeled dataset. The model parameters are updated through backpropagation. 4) Examination discrimination: Extract features from the real-time collected mono audio of the examination and input them into the trained model to obtain the confidence score.

2. The method according to claim 1, characterized in that, The confidence level represents the probability that the input speech is the target candidate's valid near-field response speech. The closer the value is to 0, the higher the probability of crosstalk / no voice.

3. The method according to claim 1, characterized in that, A response is considered valid if the confidence level is greater than 0.

99. Audio with a confidence level < 0.7 is considered crosstalk / invalid audio without voice. When the confidence level is between 0.7 and 0.99, a joint judgment is made by combining the score distribution of other question types, audio amplitude, and audio recognition results. If an anomaly is found, a manual review is initiated.

4. The method according to claim 1, characterized in that, The Fbank feature dimension is fixed at 80 dimensions to balance discrimination accuracy and model lightweighting.

5. The method according to claim 1, characterized in that, The method achieves crosstalk detection under single-microphone conditions without the need for microphone arrays or multi-channel hardware support.

6. The method according to claim 1, characterized in that, The encoder has an 8-layer structure, with each layer containing a one-dimensional CNN, a low-level self-attention encoding layer, and a fully connected layer, and layer normalization is used to improve training stability.

Images