A language data preprocessing method for multilingual and complex scenes

Abstract

The application relates to the technical field of language data processing, and discloses a language data preprocessing method for multiple languages and complex scenes, a speech data preprocessing system for multiple languages and complex scenes based on an AutoPrep framework, and five modules of speech enhancement, speech segmentation, speaker clustering, target speech extraction and quality filtering are integrated to realize automatic and structured processing of speech data. The scheme realizes differentiated suppression of stable-state and transient-state noises in multiple language speech signals, especially in small language (such as Kazakh and Tagalog) scenes, effectively improves the speech signal-to-noise ratio and the language independence of speech characteristics, overcomes the problem that the existing technology lacks a special phonetic system processing module for small languages, resulting in a high phoneme mapping error rate, and enhances the availability and processing effect of low-resource language data.

Description

Technical Field

[0001] This invention relates to the field of language data processing, and in particular to a method for language data preprocessing in multilingual and complex scenarios. Background Technology

[0002] The development of speech data preprocessing has iterated alongside the evolution of speech technology. Early focus was primarily on basic processing such as noise reduction and endpoint detection in monolingual, clean environments. However, with the increasing demand for multilingual interaction and more complex scenarios (such as noisy environments and multi-source interference), preprocessing systems have gradually developed that integrate language adaptation, enhanced environmental robustness, and cross-lingual feature normalization. Simultaneously, deep learning is combined to achieve end-to-end noise suppression and feature optimization. Its importance lies in the fact that, in multilingual and complex scenarios, high-quality preprocessing is the cornerstone of downstream tasks such as speech recognition and synthesis. It effectively eliminates interference from differences in pronunciation between different languages, environmental noise, and device distortion, improving the purity and consistency of the speech signal. This avoids model training bias caused by problems with the quality of the original data, laying a crucial foundation for the accurate understanding and expression of multilingual speech systems in complex scenarios. It is a core prerequisite for solving practical application challenges such as cross-lingual challenges and strong interference.

[0003] Existing speech data preprocessing technologies have many limitations in practical applications.

[0004] In terms of language processing, current solutions have established mature processing procedures for high-resource languages ​​(such as Chinese and English), covering pronunciation standardization based on phoneme dictionaries, dialect accent adaptive models, and acoustic feature extraction driven by large-scale corpora. However, for less common languages ​​such as Kazakh and Tagalog, due to the lack of customized processing modules for their unique phonological rules (such as vowel harmony in Kazakh and stress patterns in Tagalog), relying solely on transfer learning or cross-language models results in a high error rate in phoneme set mapping.

[0005] In terms of scene adaptability, endpoint detection and simple noise reduction technology have been engineered and applied in near-field clean scenes. However, in complex scenes, traditional multi-person overlapping speech separation algorithms often fail under single-channel conditions. Far-field recordings suffer from sound source localization deviations due to microphone array calibration errors. Real-time suppression of sudden noise also lacks robust algorithms, making it difficult to cope with real-world environments such as conference room reverberation and street noise.

[0006] In terms of data annotation and metadata processing, mainstream solutions mostly use manual or semi-automatic annotation of audio start and end times, lacking key metadata such as speaker ID, signal-to-noise ratio quality score, and language type label. The lack of speaker attribute annotation makes it impossible to construct speaker embedding, and the absence of fine-grained labels such as environmental noise type also makes it difficult to directly connect the data to the training requirements of ASR / TTS models, requiring additional manpower for secondary annotation, which seriously restricts the efficiency of engineering implementation of multilingual speech systems. Therefore, this invention proposes a language data preprocessing method for multilingual and complex scenarios to solve the above problems. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a language data preprocessing method for multilingual and complex scenarios.

[0008] The present invention provides a language data preprocessing method for multilingual and complex scenarios, which adopts the following technical solution:

[0009] A language data preprocessing method for multilingual and complex scenarios is proposed. Based on the AutoPrep framework, the audio is processed sequentially through five modules: speech enhancement module, speech segmentation module, speaker clustering module, target speech extraction module, and quality filtering module. By integrating cross-language pre-trained models with deep learning algorithms, end-to-end processing from raw speech to structured data is achieved.

[0010] The speech enhancement module consists of dynamic segmentation, XLSR-53, BSRNN and pBSRNN core components, which realize intelligent noise reduction processing of input audio;

[0011] The framed audio from the speech segmentation module will be input into the pre-trained VAD model, which is based on the TDNN-Transformer architecture and can accurately identify whether each frame of audio belongs to "speech" or "non-speech".

[0012] The speaker clustering module divides the processed speech data into blocks and inputs them into the WeSpeaker-XL model to extract speaker embedding vectors. A similarity matrix is ​​constructed by calculating the cosine similarity between each pair of embedding vectors.

[0013] Preferably, the speech enhancement module mainly consists of dynamic segmentation, a pre-trained model with cross-language feature extraction capability, and core components of the speech enhancement model, thereby achieving intelligent noise reduction processing of the input audio.

[0014] Preferably, in the voice enhancement module:

[0015] S11. Input audio is first divided into long audio and short audio according to its duration;

[0016] S12. After segmentation, the audio is used to extract language-independent robust features by a pre-trained model with cross-language feature extraction capabilities, suppressing environmental noise and capturing long-distance dependencies in the audio sequence.

[0017] S13, long audio blocks and short audio blocks are all processed by the speech enhancement model. Long audio blocks capture long temporal dependencies; short audio blocks quickly suppress burst noise.

[0018] S14. The denoised audio segments are integrated through time alignment and overlapping splicing technology. Long audio segments use a weighted overlapping and addition algorithm to eliminate boundary effects, while short audio segments are post-processed to optimize spectral continuity, thereby outputting high-fidelity denoised audio.

[0019] Preferably, in the speech segmentation module:

[0020] S21. Perform amplitude normalization processing on the audio signal;

[0021] S22. Process the audio by dividing it into frames;

[0022] S23. The framed audio will be input into the trained speech activity detection model;

[0023] S24. Based on the speech probability output by the model, set an adaptive silence threshold, identify high-probability consecutive frames as speech segments, and classify the rest as silence.

[0024] S25. Mark the start and end times of the effective speech and extract the corresponding waveform data from the original audio.

[0025] Preferably, in the speaker clustering module:

[0026] S31. The processed speech data is divided into blocks and input into the speaker embedding extraction network model to extract speaker embedding vectors. A similarity matrix S is constructed by calculating the cosine similarity between each pair of embedding vectors.

[0027] S32. Perform spectral clustering on the matrix S, directly converting S into an adjacency matrix W, calculating the degree matrix D, and constructing a symmetric normalized Laplacian matrix.

[0028]

[0029] S33, L sym Eigenvalue decomposition is performed, and the number of speakers k is automatically estimated using the "eigenvalue gap". The first k eigenvectors are selected to form a matrix U. After normalizing each row vector, K-means is applied to divide them into k clusters.

[0030] S34. By calculating the cosine similarity between the centers or representative embeddings of each cluster, if it exceeds the dynamic threshold, similar clusters are merged to further refine the speaker segmentation.

[0031] S35. Assign a unique speaker ID to each cluster and map the position of each segment of the original audio embedded in the sliding window to the corresponding ID.

[0032] Preferably, for continuous streams of recordings longer than 2 hours, the spectral clustering state can be reset every 2 hours: retaining historical cluster centers, matching and updating cluster assignments with newly generated embedding vectors to maintain speaker ID consistency, while cleaning up old data.

[0033] Preferably, in the target speech extraction module:

[0034] S41. Merge adjacent speech segments from the same speaker in the speaker clustering results, and input the merged speech segments into STFT to extract spectral features and provide frequency domain representation for subsequent processing.

[0035] S42. Combine the spectral features obtained from STFT with the target speaker embedding, and use this as input to the advanced speech separation network model in target speech extraction to obtain M. tgt (t,f) and M noiset (t,f);

[0036] S43. For single-channel cases, use M. tgt (t,f) through Calculate the frequency domain representation of the separated target speaker, and input it into the RNN filter: The residual noise, reverberation, and frequency smoothing information are further modeled in the frequency domain, and then iSTFT is performed to transform the frequency-enhanced signal back to the time domain.

[0037] S44, For multi-channel cases, use M noiset The covariance and target direction estimation are calculated, and the MVDR beamforming formula is calculated as follows:

[0038]

[0039] in Let d(f) be the noise covariance matrix, d(f) be the target direction vector, and w(f) be the optimal weighting vector at frequency f. The multi-channel spectrum is weighted, and iSTFT is performed to transform the frequency-enhanced signal back to the time domain.

[0040] Preferably, the quality filtering module includes language adaptive scoring and anomaly detection technology:

[0041] In language adaptive scoring, the scoring model uses a differentiated threshold.

[0042] Anomaly detection detects anomalies by analyzing changes in audio energy, spectral characteristics, and time-domain waveforms.

[0043] Preferably, the quality filtration process includes:

[0044] S51, Receive a speech segment from the target speech extraction module;

[0045] S52. Apply the appropriate threshold to score the quality based on the language type;

[0046] S53. Perform anomaly detection and remove or repair unqualified segments;

[0047] S54 outputs high-quality speech segments, along with quality scoring indicators, providing a basis for further screening and processing in downstream tasks.

[0048] In summary, the present invention has at least one of the following beneficial technical effects:

[0049] This invention introduces the XLSR-53 cross-lingual pre-trained model and the BSRNN / pBSRNN dynamic block denoising algorithm to achieve differentiated suppression of steady-state and transient noise in multilingual speech signals. Especially in the case of minority languages ​​(such as Kazakh and Tagalog), it effectively improves the speech signal-to-noise ratio and the language independence of speech features. It overcomes the problem of high phoneme mapping error rate caused by the lack of dedicated phonology processing modules for minority languages ​​in the existing technology, and enhances the availability and processing effect of low-resource language data.

[0050] By constructing a multilingual speech activity detection (VAD) model based on the TDNN-Transformer hybrid architecture, adaptively adjusting the silence detection threshold according to the signal-to-noise ratio, and combining it with a speech segment duration standardization strategy, accurate segmentation of continuous audio in complex scenarios (such as street noise and far-field conference reverberation environments) is achieved. This improves the recognition accuracy of speech segment boundaries and significantly enhances the speech separation effect in single-channel, multi-speaker overlapping speech environments, solving the problem of poor adaptability of traditional segmentation techniques under complex conditions.

[0051] This solution employs the WeSpeaker-XL model to extract universal speaker embeddings for multiple languages, and combines incremental spectral clustering with language-adaptive thresholding to achieve high-precision clustering and automated speaker annotation for multi-speaker speech. This solves the problem of insufficient speaker metadata and the need for extensive manual secondary annotation in traditional preprocessing workflows. This method effectively improves speaker tag generation efficiency, reducing manual annotation costs by 40%-60% while maintaining clustering accuracy, significantly shortening the engineering deployment cycle of multilingual speech systems.

[0052] By using speaker embedding as prior input, combined with the pBSRNN model and simulated beamforming algorithm, accurate speech extraction and reverberation suppression of the target speaker are achieved in single-channel and multi-channel scenarios.

[0053] Finally, the invention constructs a language-adaptive speech quality filtering system, which sets DNSMOS and PDNSMOS scoring thresholds for high-resource and low-resource languages ​​respectively. Combined with an anomaly detection mechanism and short-term noise repair strategy, it effectively removes speech segments with low sound quality, incomplete content, or non-speech interference, avoiding the problem of over-filtering of minority language data due to low scores. This ensures the scalability and stability of the system when processing multilingual and large-scale speech data, and is suitable for diverse application scenarios such as cross-border customer service and multilingual conferencing. Attached Figure Description

[0054] Figure 1 This is a system flowchart of an embodiment of the invention.

[0055] Figure 2 This is a flowchart illustrating the audio enhancement process according to an embodiment of the invention.

[0056] Figure 3 This is a schematic diagram of the VAD model structure based on the TDNN-Transformer architecture in an embodiment of the invention.

[0057] Figure 4 This is a flowchart of speaker clustering in an embodiment of the invention.

[0058] Figure 5 This is a structural diagram of the SAM according to an embodiment of the invention.

[0059] Figure 6 This is a flowchart of the target speech extraction process according to an embodiment of the invention. Detailed Implementation

[0060] The following is in conjunction with the appendix Figure 1 - Appendix Figure 6 The present invention will be described in further detail below.

[0061] Example 1: A Language Data Preprocessing Method for Multilingual and Complex Scenarios

[0062] I. Overall Framework Design

[0063] A multilingual, complex-scene speech data preprocessing system based on the AutoPrep framework is built, comprising five core modules: speech enhancement, speech segmentation, speaker clustering, target speech extraction, and quality filtering. Through the fusion of cross-language pre-trained models and deep learning algorithms, it achieves end-to-end processing from raw speech to structured, high-quality data. The system supports cross-language processing of over 50 languages ​​and outputs JSON structured data containing speaker tags and quality metrics.

[0064] Before preprocessing, the input audio needs to be formatted as WAV and the sampling rate adjusted to 16kHz. The audio then sequentially flows through the speech enhancement, speech segmentation, speaker clustering, target speech extraction, and quality filtering modules. In the speaker clustering stage, the "merge" operation integrates adjacent speech segments belonging to the same speaker, ultimately outputting the fully processed audio.

[0065] II. Core Module Design

[0066] 1. Speech enhancement

[0067] The speech enhancement module mainly consists of core components such as dynamic segmentation, XLSR-53, BSRNN, and pBSRNN, enabling intelligent noise reduction processing of input audio. Input audio is divided into two categories based on duration: long audio (duration greater than 3 minutes) and short audio (duration less than 3 minutes). During dynamic segmentation, long audio employs a sliding segmentation strategy with a 12-second segmentation window and a 4-second offset, ensuring the continuity of temporal information through overlapping windows. Short audio uses a fixed 8-second segmentation method to balance computational efficiency and feature integrity.

[0068] After segmentation, the audio will be used to extract language-independent robust features using XLSR-53. This model is trained on large-scale multilingual data, which can overcome language limitations, effectively suppress environmental noise, and capture long-distance dependencies in audio sequences.

[0069] BSRNN and pBSRNN are neural networks trained on mixed audio data in multilingual and noisy scenarios. Long audio blocks are processed by BSRNN, whose bidirectional recursive structure is suitable for capturing long temporal dependencies, while short audio blocks are processed by pBSRNN, whose lightweight architecture can quickly suppress burst noise for short audio.

[0070] Finally, the denoised audio segments are integrated through time alignment and overlapping splicing techniques. Long audio segments are eliminated by weighted overlapping and adding algorithms, while short audio segments are post-processed to optimize spectral continuity, thereby outputting high-fidelity denoised audio.

[0071] 2. Speech segmentation

[0072] To ensure effective speech segmentation, after speech enhancement, the audio signal is first normalized. Specifically, the amplitude of each sample point in the entire audio segment is divided by the maximum absolute amplitude of that segment, thereby compressing the maximum amplitude of the waveform to within ±1. The mathematical expression is as follows:

[0073]

[0074] in, This represents the original amplitude at the nth sampling point. It is the largest absolute amplitude in the entire audio segment. This is the normalized amplitude value.

[0075] After normalization, the audio will be processed in frames (FrameBlocking). Each frame is set to a duration of 25 milliseconds (corresponding to 400 sampling points), with a frame shift of 10 milliseconds (160 sampling points). If the audio ends with less than one frame, zero-padding is used to fill the gap. Then, each frame is multiplied by a Hamming window function to reduce spectral leakage at frame boundaries and improve the accuracy of subsequent modeling.

[0076] After the above preprocessing, the framed audio will be input into the pre-trained VAD (Voice Activity Detection) model. This model is based on the TDNN-Transformer architecture and has been trained on multilingual silent and non-silent corpora, and can accurately identify the probability of each audio frame belonging to "speech" or "non-speech".

[0077] Next, based on the speech probabilities output by the model, an adaptive silence threshold is set (e.g., 0.8 in a high SNR environment and 0.6 in a low SNR environment) to identify high-probability consecutive frames as speech segments, and the rest as silence. Post-processing is then performed on the initially detected speech segments: first, speech segments with adjacent silence intervals of less than 1 second are merged to preserve the integrity of the speech flow; second, for long speech segments exceeding 30 seconds, they are segmented at the first silence point within each segment to prevent excessively long information blocks; finally, short speech segments of less than 1.5 seconds are filtered out to improve the effectiveness and usability of the speech fragments.

[0078] After the above processing is completed, each valid speech segment will be labeled with its start and end times, and the corresponding waveform data will be extracted from the original audio. These well-structured and accurately defined speech segments will serve as key inputs for subsequent speaker clustering, ensuring the robustness of the overall speech processing workflow and high-quality output.

[0079] 3. Speaker clustering

[0080] The segmented speech data is divided into blocks (each block is 1.5 seconds long, sliding every 0.75 seconds) and then input into the WeSpeaker-XL model to extract speaker embedding vectors. A similarity matrix is ​​constructed by calculating the cosine similarity between each pair of embedding vectors. The formula for calculating the cosine similarity is as follows:

[0081]

[0082] in It is a dot product. It is an L2 norm.

[0083] Next, spectral clustering is performed, directly converting S into an adjacency matrix W, calculating the degree matrix D, and constructing a symmetric normalized Laplacian matrix.

[0084]

[0085] For L sym Eigenvalue decomposition is performed, and the number of speakers k is automatically estimated using the "eigenvalue gap". The top k eigenvectors are selected to form a matrix U, and after normalizing each row vector, K-means is applied to divide them into k clusters. Then, the cosine similarity between the cluster centers or representative embeddings is calculated. If it exceeds a dynamic threshold (usually set to 0.75, which can be adjusted to 0.7 or 0.6 in some language systems), similar clusters are merged to further refine the speaker segmentation.

[0086] Finally, a unique speaker ID (such as spk_001, spk_002, etc.) is assigned to each cluster, and the position of each segment of the original audio embedded in the sliding window is mapped to the corresponding ID. The output format can be a JSON or CSV file with timestamps, or the tags can be directly embedded in the audio metadata.

[0087] For continuous streaming or very long recordings (>2 hours), the process can reset the spectral clustering state every 2 hours: retain historical cluster centers, match and update cluster assignments with newly generated embedding vectors to maintain speaker ID consistency, and clean up old data to ensure computation and memory control.

[0088] 4. Target speech extraction

[0089] Speech segments from adjacent speakers in the speaker clustering results are merged. The merged speech segments are then input into the STFT to extract spectral features, providing a frequency domain representation for subsequent processing. The spectral features obtained from the STFT are combined with the target speaker embedding and used as input to the pBSRNN model in target speech extraction, yielding M. tgt (t,f) and M noiset (t,f), where the pBSRNN differs from the pBSRNN in the speech processing section. Unlike the pBSRNN in the speech enhancement section, the pBSRNN in target speech extraction adds a Sequence Attention (SAM) module after the Band & Sequence Modeling Module. This module enhances the model's ability to model the temporal dependencies of speech sequences through a multi-head attention mechanism, significantly improving separation performance in complex scenarios. This pBSRNN model is trained using multilingual, mixed speech, and single-person speaking audio datasets, optimizing its generalization ability to multiple languages ​​and noisy environments.

[0090] For single-channel applications, use M. tgt (t,f) through The frequency domain representation of the separated target speaker is calculated and input into the RNN filter. The residual noise, reverberation, and frequency smoothing information are further modeled in the frequency domain, and then iSTFT is performed to transform the frequency-enhanced signal back to the time domain.

[0091] For multi-channel situations, use M noiset The covariance and target direction estimation are calculated, and the MVDR beamforming formula is calculated as follows:

[0092]

[0093] in Let f be the noise covariance matrix, d(f) be the target direction vector, and w(f) be the optimal weighted vector at frequency f.

[0094] Then, the multi-channel spectrum is weighted. Then, iSTFT is executed to transform the frequency-domain enhanced signal back to the time domain.

[0095] Through the above single-channel and multi-channel processes, the target speech extraction module can achieve high-quality separation and dereverberation of the target speaker under different hardware and environmental conditions.

[0096] 5. Quality Filtration

[0097] Quality filtering is a crucial step in the speech processing workflow, designed to eliminate low-quality speech segments and ensure that downstream tasks (such as speech recognition and speaker recognition) receive high-quality, reliable input data, thereby improving the overall system performance and robustness. This module achieves effective control over speech quality through language adaptive scoring and anomaly detection techniques.

[0098] In terms of language adaptive scoring, the system employs differentiated quality thresholds based on the language resources available. For high-resource languages ​​(such as Chinese and English), due to abundant training data and mature model support, the system sets higher quality standards, requiring a DNSMOS (a deep learning-based speech denoising quality assessment metric) greater than 3.2 and a PDNSMOS (a variant or related metric of DNSMOS) greater than 3.8 for the speech segment, to ensure high-quality input speech. For low-resource languages ​​(such as Tajik), due to limited training data and limited model evaluation accuracy, the system appropriately relaxes the thresholds, requiring a DNSMOS greater than 2.8 and a PDNSMOS greater than 3.2, while also incorporating speaker similarity compensation (similarity must be greater than 0.5) to achieve a balance between quality and data availability.

[0099] Anomaly detection is another core component of quality filtering. Its purpose is to identify and remove speech segments containing non-speech burst noise (such as horns or impact sounds) or equipment malfunctions (such as microphone disconnections or audio distortion). The system detects anomalies by analyzing audio energy changes, spectral characteristics, and temporal waveforms. For short-duration noise lasting less than 0.5 seconds, the system uses interpolation techniques based on adjacent frames to repair it, preserving as much valuable speech content as possible; while for anomaly segments that cannot be repaired, they are directly removed.

[0100] The overall process of quality filtering can be summarized as follows: First, it receives speech segments from the target speech extraction module; then, it applies appropriate DNSMOS and PDNSMOS thresholds to score the quality based on the language type; next, it performs anomaly detection, removing or repairing substandard segments; finally, it outputs high-quality speech segments along with quality scoring metrics (such as DNSMOS, PDNSMOS, and STOI), providing a basis for further screening and processing in downstream tasks. Through this series of technical means, the quality filtering module effectively ensures the quality of the speech segments, providing a reliable data foundation for subsequent processing. Example

[0101] 1. Replacement of speech enhancement modules: In the denoising process, in addition to BSRNN or pBSRNN, deep learning models such as Demucs, SEGAN, and DCCRN can also be used for speech enhancement to improve the signal-to-noise ratio and reduce reverberation. In addition, the XLSR-53 model can be replaced by pre-trained models with cross-language feature extraction capabilities such as wav2vec2.0, HuBERT, and Whisper to adapt to different application requirements and computing resources.

[0102] 2. Alternatives to speech segmentation methods: In multilingual VAD (voice activity detection), deep learning models based on Conformer, Bi-GRU, Bi-LSTM, etc., can be used to replace the TDNN-Transformer architecture, or multimodal information (such as video or voiceprint features) can be introduced to improve the accuracy of silence / speech boundary recognition.

[0103] 3. Speaker clustering alternatives: In addition to incremental spectral clustering methods, end-to-end speaker separation methods (such as EEND), clustering based on PLDA (Probabilistic Linear Discriminant Analysis), or self-supervised clustering strategies can be used to separate and annotate multi-speaker speech. At the same time, WeSpeaker-XL can also be replaced by other speaker embedding extraction networks such as ECAPA-TDNN and ResNet-based embedding models.

[0104] 4. Alternative Target Speech Extraction Methods: For the separation of the target speaker, it is not limited to using the pBSRNN model, but can also be achieved through advanced speech separation networks such as Conv-TasNet, DPRNN, and Sepformer; in terms of simulating neural beamforming, the directional extraction of multi-channel signals can also be completed through traditional MVDR or GSS (Geometric Source Separation) methods.

[0105] 5. Alternative solutions for quality filtering mechanisms: In addition to DNSMOS and PDNSMOS scoring models, a combination of subjective and objective evaluation indicators such as PESQ, STOI, and SI-SDR can be used to evaluate speech quality from multiple dimensions; for anomaly detection, convolutional neural networks (CNN) or Transformer-type models can be used to detect spectral anomalies instead of energy / waveform statistical methods to enhance the intelligence and accuracy of detection.

[0106] Alternative system architecture: The overall processing flow can be modularized or rearranged based on computing resources and real-time requirements. For example, in a high-performance computing environment, speech enhancement and target speech extraction can be merged into an integrated separate model; in edge computing scenarios, only speech segmentation and quality filtering modules can be retained to quickly extract usable segments for real-time recognition.

[0107] It should be noted that, in this document, terms such as “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0108] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A language data preprocessing method for multilingual and complex scenarios, characterized in that: Based on the AutoPrep framework, audio is processed sequentially through five modules: speech enhancement, speech segmentation, speaker clustering, target speech extraction, and quality filtering. By integrating cross-language pre-trained models with deep learning algorithms, end-to-end processing from raw speech to structured data is achieved. The speech enhancement module consists of dynamic segmentation, XLSR-53, BSRNN and pBSRNN core components, which realize intelligent noise reduction processing of input audio; The framed audio from the speech segmentation module will be input into the pre-trained VAD model, which is based on the TDNN-Transformer architecture and can accurately identify whether each frame of audio belongs to "speech" or "non-speech". The speaker clustering module divides the processed speech data into blocks and inputs them into the WeSpeaker-XL model to extract speaker embedding vectors. A similarity matrix is ​​constructed by calculating the cosine similarity between each pair of embedding vectors.

2. The language data preprocessing method for multilingual and complex scenarios according to claim 1, characterized in that: The speech enhancement module mainly consists of dynamic segmentation, a pre-trained model with cross-language feature extraction capabilities, and core components of the speech enhancement model, enabling intelligent noise reduction processing of input audio.

3. The language data preprocessing method for multilingual and complex scenarios according to claim 2, characterized in that: In the speech enhancement module: S11. Input audio is first divided into long audio and short audio according to its duration; S12. After segmentation, the audio is used to extract language-independent robust features by a pre-trained model with cross-language feature extraction capabilities, suppressing environmental noise and capturing long-distance dependencies in the audio sequence. S13, long audio blocks and short audio blocks are both processed by the speech enhancement model, and long audio blocks capture long temporal dependencies; Short audio blocks are used to quickly suppress sudden noise. S14. The denoised audio segments are integrated through time alignment and overlapping splicing technology. Long audio segments use a weighted overlapping and addition algorithm to eliminate boundary effects, while short audio segments are post-processed to optimize spectral continuity, thereby outputting high-fidelity denoised audio.

4. The language data preprocessing method for multilingual and complex scenarios according to claim 1, characterized in that: In the speech segmentation module: S21. Perform amplitude normalization processing on the audio signal; S22. Process the audio in frames; S23. The framed audio will be input into the trained speech activity detection model; S24. Based on the speech probability output by the model, set an adaptive silence threshold, identify high-probability consecutive frames as speech segments, and classify the rest as silence. S25. Mark the start and end times of the effective speech and extract the corresponding waveform data from the original audio.

5. The language data preprocessing method for multilingual and complex scenarios according to claim 1, characterized in that: In the speaker clustering module: S31. The processed speech data is divided into blocks and input into the speaker embedding extraction network model to extract speaker embedding vectors. A similarity matrix S is constructed by calculating the cosine similarity between each pair of embedding vectors. S32. Perform spectral clustering on the matrix S, directly converting S into an adjacency matrix W, calculating the degree matrix D, and constructing a symmetric normalized Laplacian matrix. ； S33, L sym Eigenvalue decomposition is performed, and the number of speakers k is automatically estimated using the "eigenvalue gap". The first k eigenvectors are selected to form a matrix U, and after normalizing each row vector, K-means is applied to divide them into k clusters. S34. By calculating the cosine similarity between the centers or representative embeddings of each cluster, if it exceeds the dynamic threshold, similar clusters are merged to further refine the speaker segmentation. S35. Assign a unique speaker ID to each cluster and map the position of each segment of the original audio embedded in the sliding window to the corresponding ID.

6. The language data preprocessing method for multilingual and complex scenarios according to claim 5, characterized in that: For continuous streams of recordings longer than 2 hours, the spectral clustering state can be reset every 2 hours: retain historical cluster centers, match and update cluster assignments with newly generated embedding vectors to maintain speaker ID consistency, and clean up old data.

7. The language data preprocessing method for multilingual and complex scenarios according to claim 1, characterized in that: In the target speech extraction module: S41. Merge adjacent speech segments from the same speaker in the speaker clustering results, and input the merged speech segments into STFT to extract spectral features and provide frequency domain representation for subsequent processing. S42. Combine the spectral features obtained from STFT with the target speaker embedding, and use this as input to the advanced speech separation network model in target speech extraction to obtain M. tgt (t,f) and M noiset (t,f); S43. For single-channel cases, use M. tgt (t,f) through Calculate the frequency domain representation of the separated target speaker, and input it into the RNN filter: The residual noise, reverberation, and frequency smoothing information are further modeled in the frequency domain, and then iSTFT is performed to transform the frequency-enhanced signal back to the time domain. S44, For multi-channel cases, use M noiset The covariance and target direction estimation are calculated, and the MVDR beamforming formula is calculated as follows: ； in Let d(f) be the noise covariance matrix, d(f) be the target direction vector, and w(f) be the optimal weighting vector at frequency f. The multi-channel spectrum is weighted, and iSTFT is performed to transform the frequency-enhanced signal back to the time domain.

8. The language data preprocessing method for multilingual and complex scenarios according to claim 1, characterized in that: The quality filtering module includes language-adaptive scoring and anomaly detection technology: In language adaptive scoring, the scoring model uses a differentiated threshold. Anomaly detection detects anomalies by analyzing changes in audio energy, spectral characteristics, and time-domain waveforms.

9. A language data preprocessing method for multilingual and complex scenarios according to claim 8, characterized in that: The quality filtration process includes: S51, Receive a speech segment from the target speech extraction module; S52. Apply the appropriate threshold to score the quality based on the language type; S53. Perform anomaly detection and remove or repair unqualified segments; S54 outputs high-quality speech segments, along with quality scoring indicators, providing a basis for further screening and processing in downstream tasks.

Images