Massive audio data preprocessing method, device, equipment and medium

Abstract

This invention relates to the field of artificial intelligence technology, providing a method, apparatus, device, and medium for large-scale audio data preprocessing. Applied to a processing system built on a multi-stage pipeline scheduling mechanism, it improves throughput and resource utilization. Based on a speech quality evaluation mechanism, it performs adaptive source separation for each candidate audio segment, solving the problems of unnecessary processing of high-quality audio, ineffective computation of low-quality audio, and lack of quantitative tracking of processing effects caused by a unified source separation strategy. It performs speaker separation and conditional speech activity detection and segmentation, reducing problems such as multiple speakers, sentence truncation, excessively long segments, or incomplete segment semantics. Combining audio language detection and text language filtering, it can ensure that the output speech-text pairs meet the target language requirements from both the audio and text sides, while reducing processing time. The complete processing chain for generating speech-text pairs according to the audio processing data structure meets the requirements for traceable management.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for large-scale audio data preprocessing. Background Technology

[0002] With the development of applications such as voice interaction, intelligent customer service, content moderation, speech synthesis, automatic captioning, multimedia retrieval, and large-scale model training, the scale of audio data continues to grow. Large amounts of raw audio data often come from complex sources and may contain issues such as different sampling rates, varying volumes, background noise interference, multiple speakers, long periods of silence, mixed languages, missing transcriptions, or unstable annotation quality. To use this audio data for speech recognition, speech synthesis, speaker modeling, or multimodal large-scale model training, it is typically necessary to first perform audio preprocessing, segmentation, transcription, quality screening, and speech-text pair construction.

[0003] Existing audio preprocessing techniques still have several shortcomings when dealing with large-scale unlabeled or weakly labeled audio data.

[0004] First, traditional processing workflows often use a fixed order or a single strategy for segmentation, which makes it difficult to handle problems such as long audio, multiple speakers, and inaccurate semantic boundaries at the same time. This can easily result in samples with multiple overlapping speakers, truncated sentences, excessively long segments, or incomplete semantics.

[0005] Secondly, existing audio source separation methods typically perform uniform processing on all audio, lacking a dynamic decision-making mechanism based on audio quality. This may result in unnecessary processing of high-quality audio, or ineffective computational resources being invested in extremely low-quality audio.

[0006] Furthermore, in multilingual audio processing scenarios, non-target languages, mixed languages, or low-confidence texts are easily introduced, affecting the linguistic consistency of subsequent training data. For automatically transcribed speech-text pairs, without an independent verification mechanism, ASR errors are difficult to detect in a timely manner, and data quality is difficult to quantify. On the other hand, common deduplication methods are mostly based on complete text matching or audio similarity judgment, which can easily lead to the accidental deletion of valid samples of different speakers saying the same text, or the retention of duplicate content from the same speaker, affecting the diversity and effectiveness of the dataset.

[0007] Finally, in terms of engineering implementation, large-scale audio preprocessing usually involves multiple computationally intensive and network-request-based modules. Existing serial or semi-automatic script-based processing methods cannot fully utilize computing resources, and the cost of repeated processing after task interruption is high. It is also not convenient to record the filtering reasons, performance indicators and quality statistics at each stage, making it difficult to meet the needs of large-scale continuous library building, incremental processing and traceable management.

[0008] Therefore, there is an urgent need for an automated preprocessing technology solution for large-scale audio data. Summary of the Invention

[0009] In view of the above, it is necessary to provide a method, apparatus, device and medium for large-scale audio data preprocessing, which aims to solve the problems of low quality, high cost and long time consumption in the preprocessing results of large-scale audio data.

[0010] A large-scale audio data preprocessing method is applied to a processing system built on a multi-stage pipeline scheduling mechanism. Each task unit in the processing system employs an independent input queue, output queue, and thread, with the processing result of the previous task unit serving as the input for the next task unit. The large-scale audio data preprocessing method includes: In response to a preprocessing instruction triggered by a large-scale audio list, a breakpoint-based continuation processing mechanism is used to identify the audio data to be processed in the audio list, and the audio data to be processed is standardized to obtain each candidate audio segment. Based on the speech quality evaluation mechanism, adaptive sound source separation is performed on each candidate audio segment to obtain the speaking voice and background sound of each candidate audio segment; Speaker segmentation is performed on each candidate audio segment based on the speaking voice, to obtain the speaking range of each speaker; Conditional speech activity detection and segmentation are performed on the speaking interval of each speaker to obtain multiple candidate segments for parallel processing; Audio language detection is performed on the multiple candidate segments to obtain detection results, and the multiple candidate segments are filtered according to the detection results to obtain multiple retained segments; Based on the multiple retained segments, speech-text pairs are generated, and a complete processing chain for the speech-text pairs is generated according to the audio processing data structure.

[0011] A large-scale audio data preprocessing device operates within a processing system built on a multi-stage pipeline scheduling mechanism. Each task unit in the processing system employs an independent input queue, output queue, and thread, with the processing result of the previous task unit serving as the input for the next task unit. The large-scale audio data preprocessing device includes: The standardization unit is used to respond to a preprocessing instruction triggered by a large-scale audio list, identify the audio data to be processed in the audio list using a breakpoint continuation processing mechanism, and perform standardization processing on the audio data to be processed to obtain each candidate audio segment. The separation unit is used to perform adaptive sound source separation for each candidate audio segment based on the speech quality evaluation mechanism, so as to obtain the speech and background sound of each candidate audio segment; The separation unit is also used to perform speaker separation on each candidate audio segment based on the speaking voice of each candidate audio segment, so as to obtain the speaking range of each speaker; The segmentation unit is used to perform conditional speech activity detection segmentation on the speaking interval of each speaker to obtain multiple candidate segments; A filtering unit is used to perform audio language detection on the plurality of candidate segments to obtain detection results, and to filter the plurality of candidate segments according to the detection results to obtain a plurality of retained segments; The generation unit is used to generate speech-text pairs based on the plurality of retained segments, and to generate a complete processing link for the speech-text pairs according to the audio processing data structure.

[0012] A computer device, the computer device comprising: A memory that stores at least one instruction; and a processor that executes the instructions stored in the memory to implement the large-scale audio data preprocessing method.

[0013] A computer-readable storage medium storing at least one instruction that is executed by a processor in a computer device to implement the large-scale audio data preprocessing method.

[0014] As can be seen from the above technical solutions, this invention, applied to a processing system built on a multi-stage pipeline scheduling mechanism, improves throughput and resource utilization. Based on a speech quality evaluation mechanism, it performs adaptive source separation for each candidate audio segment, solving the problems of unnecessary processing of high-quality audio, ineffective computation of low-quality audio, and lack of quantitative tracking of processing effects caused by a unified source separation strategy. By performing speaker separation for each candidate audio segment based on the speaker's voice and conditional speech activity detection segmentation for each speaker's speaking interval, it reduces problems such as multiple speakers mixing, sentence truncation, excessively long segments, or incomplete segment semantics, improving the usability of the final speech-text segment. Combining audio language detection and text language filtering ensures that the output speech-text pair meets the target language requirements from both the audio and text sides, while reducing processing time. The complete processing chain for generating speech-text pairs according to the audio processing data structure meets the requirements for traceable management. Attached Figure Description

[0015] Figure 1 This is a flowchart of a preferred embodiment of the large-scale audio data preprocessing method of the present invention.

[0016] Figure 2 This is a functional block diagram of a preferred embodiment of the large-scale audio data preprocessing device of the present invention.

[0017] Figure 3This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the large-scale audio data preprocessing method of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the large-scale audio data preprocessing method of the present invention. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0020] The large-scale audio data preprocessing method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0021] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), smart wearable device, etc.

[0022] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0023] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0024] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0025] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0026] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0027] This embodiment is applied to a processing system built on a multi-stage pipeline scheduling mechanism. Each task unit in the processing system adopts an independent input queue, output queue and thread, and the processing result of the previous task unit is used as the input of the next task unit.

[0028] Specifically, this embodiment organizes multiple processing stages in the audio preprocessing and speech-text pair construction process into a continuous, collaborative pipeline. These processing stages may include, but are not limited to: audio input, source separation, audio normalization, speaker separation, conditional VAD (Voice Activity Detection), language detection, ASR (Automatic Speech Recognition) transcription, forced alignment, quality filtering, ASR verification, deduplication, and result writing.

[0029] Among them, processing plans for different task units can be generated based on input modes and configuration parameters.

[0030] Different task units can be configured with different concurrency levels based on their computational characteristics. For example, audio normalization, language detection, ASR request, alignment post-processing, and ASR verification can each be configured with independent concurrency parameters to improve overall throughput efficiency.

[0031] For multiple candidate segments derived from the same original audio, segment-level parallel processing can be performed, and the number of retained and filtered samples in the corresponding source file can be counted after processing. Task backlog can also be limited by queue capacity, audio duration, or audio data size to reduce the risk of memory and GPU memory consumption during large-scale processing.

[0032] One option is to use asynchronous writing to write the retained samples, filtered samples, and running lists to the corresponding files or database.

[0033] The run list can record information such as audio ID, original file path, input mode, processing status, number of records to be retained, and number of records to be filtered, so that if the task is interrupted, completed audio can be skipped and only incomplete data can be processed.

[0034] It can also record the number of inputs, outputs, filters, filtering reasons, error types, processing time, queue depth, audio throughput, and resource usage information such as CPU (Central Processing Unit), memory, and GPU (Graphics Processing Unit) at each stage.

[0035] The pipeline scheduling mechanism described above can improve the processing efficiency, stability, resource utilization, and engineering traceability of large-scale audio preprocessing tasks, enabling continuous processing, concurrent execution, and breakpoint resume of large-scale audio data.

[0036] The processing system configures independent queues, threads, and concurrency parameters for different task units, enabling continuous scheduling and parallel processing of tasks at each stage. It also supports asynchronous writing, run lists, breakpoint continuation, stage statistics, error logging, and resource sampling. It can continue processing incomplete data after a task is interrupted, recording the number of data processed, filtering reasons, error types, and resource usage. It is adaptable to various backends, including local models, remote services, GPU inference, and API (Application Programming Interface) calls, improving the efficiency, stability, and deployment compatibility of large-scale audio preprocessing.

[0037] Specifically, the large-scale audio data preprocessing method includes: S10, in response to a preprocessing instruction triggered by a large-scale audio list, a breakpoint continuation processing mechanism is used to identify the audio data to be processed in the audio list, and the audio data to be processed is standardized to obtain each candidate audio segment.

[0038] In this embodiment, the audio list is used to store audio data, which may be unlabeled or weakly labeled.

[0039] The audio list may include audio IDs and audio file paths. In weakly labeled scenarios, the audio list may also include existing text annotations.

[0040] In this embodiment, a breakpoint-resume processing mechanism is employed. Processing tasks can be created based on the audio ID, file path, and input mode, and the running list can be read to determine whether the audio has been processed completely. For audio that has been processed, processing can be skipped; for audio that has not been processed, subsequent processing steps can be initiated, thus supporting breakpoint-resume processing.

[0041] In this embodiment, the standardization process for the audio data to be processed includes: The audio data to be processed is subjected to format unification, sampling rate conversion, channel conversion, loudness normalization, and peak value normalization.

[0042] It can maintain two types of audio simultaneously: one is the final output sampling rate audio, and the other is 16kHz audio, which is used as input for models such as speaker separation, VAD, ASR, language detection, forced alignment, and quality assessment.

[0043] Through the above embodiments, audio from different sources, in different formats, and at different volume levels can be converted into a unified input for subsequent processing.

[0044] S11, based on the speech quality evaluation mechanism, performs adaptive sound source separation on each candidate audio segment to obtain the speaking voice and background sound of each candidate audio segment.

[0045] In this embodiment, the adaptive sound source separation performed on each candidate audio segment based on the speech quality evaluation mechanism to obtain the speaking voice and background sound of each candidate audio segment includes: A speech quality score for each candidate audio segment is generated based on a pre-configured objective speech quality assessment model. Obtain the preset low-quality threshold and high-quality threshold; For each candidate audio segment, if the speech quality score is less than the low quality threshold, the candidate audio segment is determined to be a low quality segment and filtered; or When the speech quality score is greater than the high-quality threshold, the candidate audio segment is determined to be a high-quality segment, and adaptive source separation is skipped; or When the speech quality score is greater than or equal to the low quality threshold and less than or equal to the high quality threshold, adaptive sound source separation is performed on the candidate audio segment to obtain the speaking voice and background sound of the candidate audio segment.

[0046] The objective speech quality evaluation model can be an evaluation model based on DNSMOS (Deep Noise Suppression Mean Opinion Score).

[0047] For segments that have undergone audio source separation, the DNSMOS score can be recalculated after separation, and the score before audio source separation, the score after audio source separation, and the score change can be recorded for subsequent quality tracking.

[0048] Compared to the method of uniformly separating all audio sources, this embodiment can avoid over-processing of high-quality speech, reduce invalid calculations for low-quality audio, and improve preprocessing efficiency and the usability of output speech data.

[0049] S12, perform speaker separation on each candidate audio segment based on the speaking voice of each candidate audio segment to obtain the speaking range of each speaker.

[0050] In this embodiment, the step of performing speaker separation on each candidate audio segment based on the speaking voice of each candidate audio segment to obtain the speaking range of each speaker includes: For adjacent audio segments belonging to the same speaker in the candidate audio segments, the segments are merged according to the interval threshold to obtain the speaking range of each speaker; Specifically, segments with overlapping speakers, durations shorter than the duration threshold, or that do not meet preset requirements will be deleted or marked as non-compliant.

[0051] Among them, low-quality segments can be filtered based on conditions such as overlapping speech and minimum duration.

[0052] Through the above embodiments, candidate intervals for single speakers or near-single speakers can be obtained, thereby reducing the problem of multiple speakers mixing in subsequent samples.

[0053] S13, perform conditional speech activity detection and segmentation on the speaking interval of each speaker to obtain multiple candidate segments for parallel processing.

[0054] In this embodiment, the conditional speech activity detection segmentation performed on the speaking interval of each speaker to obtain multiple candidate segments includes: Obtain the segmentation duration threshold and the duration of each speaker's speaking interval; For each speaker's speaking interval, if the duration of the speaking interval is less than or equal to the segmentation duration threshold, the speaking interval is retained as the candidate segment; or When the duration of the speech interval is greater than the segmentation duration threshold, the speech interval is segmented according to the preset segmentation constraints to obtain the multiple candidate segments.

[0055] The step of segmenting the speech interval according to the preset segmentation constraints includes: VAD segmentation is performed on the speech interval, and then shorter candidate segments are generated based on segmentation constraints such as speech detection threshold, minimum speech duration, minimum silence duration, maximum duration, minimum duration and padding parameters.

[0056] The above embodiments can avoid long segments from affecting subsequent ASR and alignment effects.

[0057] This embodiment integrates a multi-level audio segmentation mechanism that combines speaker perception and semantic alignment. It uses speaker separation, conditional VAD, and forced text alignment in a coordinated manner to generate high-quality speech segments. Compared with segmentation methods that rely solely on silence or fixed length, this mechanism can effectively reduce problems such as multiple speakers mixing, long audio recognition drift, and inaccurate boundary segmentation. It improves the speaker consistency, semantic integrity, and temporal boundary accuracy of audio segments, and reduces situations such as multiple speakers mixing, sentence truncation, excessively long segments, or incomplete segment semantics, thereby improving the usability of the final speech-text segment.

[0058] S14, perform audio language detection on the multiple candidate segments to obtain detection results, and filter the multiple candidate segments according to the detection results to obtain multiple retained segments.

[0059] In this embodiment, the detection results may include the detected language and confidence level.

[0060] In this embodiment, filtering the plurality of candidate fragments based on the detection results to obtain a plurality of retained fragments includes: The candidate segments that fail to be detected, have a confidence level lower than the preset confidence level, or whose detected language does not match the target language are filtered from the plurality of candidate segments to obtain the plurality of retained segments.

[0061] S15, generate speech-text pairs based on the multiple retained segments, and generate a complete processing link for the speech-text pairs according to the audio processing data structure.

[0062] In this embodiment, generating speech-text pairs based on the plurality of retained segments includes: The main automatic speech recognition model is invoked to transcribe each retained segment, resulting in each candidate text; Each candidate text is normalized to obtain multiple normalized texts; The multiple standardized texts are aligned with their corresponding preserved segments to obtain the alignment results. The audio in the alignment result is re-segmented according to the preset semantic re-segmentation rules to obtain the speech-text pair.

[0063] When performing ASR transcription, for weakly labeled audio segments with existing text annotations, the existing text can be used directly, or the existing text can be validated or corrected in conjunction with the ASR results. If the initial ASR output is empty or abnormal, the segment is filtered out.

[0064] ASR transcription provides the necessary text for subsequent forced alignment.

[0065] The normalization process may include number normalization, character set checking, word segmentation, whitespace removal, text length checking, and text language detection. If the text does not meet the target language requirements, or quality requirements such as character set or length, the segment will be filtered.

[0066] Specifically, during forced alignment and semantic-level resegmentation, the multiple standardized texts and corresponding retained segments can be forcibly aligned to obtain word-level or sentence-level timestamps and thus the alignment results. Further, based on word spacing thresholds, sentence merging thresholds, boundary padding, and maximum / minimum duration constraints, the audio in the alignment results is resegmented into semantically more complete sentence-level or phrase-level segments, generating the speech-text pairs with corresponding text, start and end times, and alignment scores.

[0067] This embodiment employs a dual language consistency verification mechanism based on audio language detection and text language filtering. Language recognition and filtering are performed on both the audio and text sides to ensure that the output speech-text matches the target language requirements. On the audio side, language detection is performed on the segmented audio segments to obtain the detected language and its confidence level. The detection results are then matched with the target language and its language variants. When the detection confidence level is lower than a preset threshold, or the detected language does not match the target language, the audio segment is filtered. On the text side, language-related normalization processing is performed on the ASR transcribed text or existing annotated text, and secondary filtering is conducted using strategies such as character set constraints, text language detection, confidence thresholds, and text length constraints. Different text normalization and word segmentation strategies can be adopted for different languages, such as differentiated processing for Arabic numerals, Thai word segmentation, Chinese word segmentation, uppercase and lowercase letters, and whitespace characters. Through the joint constraints of audio language detection and text language filtering, problems such as cross-language false recall, ASR language drift, and the mixing of non-target language text can be effectively reduced, thereby improving the language purity and consistency of the final training data or question-and-answer corpus.

[0068] In this embodiment, the complete processing chain for generating the speech-text pair according to the audio processing data structure includes: The speech-text pair is filtered according to a preset multidimensional filtering rule to obtain a retained sample set of filtered samples. The retained samples are written into the retained sample record, and the filtered samples are written into the filtered sample record. The retained samples are rewritten using the automatic speech recognition model to obtain the verification text. Calculate the difference between the text in the retained sample and the verification text, and write the verification text and the difference into a metadata file; Calculate the text hash of the retained samples, and group the retained samples according to the text hash; For each sample in a group, speaker embedding vectors are extracted, and the similarity between the speaker embedding vectors is calculated. Speaker embedding vectors with the same text and a similarity greater than the similarity threshold are identified as duplicate content of the same speaker, and the sample corresponding to one of the speaker embedding vectors is retained as a valid sample. Speaker embedding vectors with the same text and a similarity less than or equal to the similarity threshold are identified as valid samples of different speakers, and the valid samples are retained. Write the audio data to be processed into the original audio record, and write the candidate segments into the candidate segment record; The complete processing chain is obtained by integrating the original audio recording, the candidate segment recording, the retained sample recording, and the filtered sample recording.

[0069] The multidimensional filtering rules may include duration filtering (for filtering segments that are shorter than the minimum duration or longer than the maximum duration), speech quality filtering (for performing speech quality evaluation on the output segments and filtering samples that are lower than the quality threshold, such as using DNSMOS scores or other speech quality evaluation indicators to calculate speech quality scores), and alignment quality checks.

[0070] In addition, the corresponding filtering stage and the reason for filtering can also be recorded for the filtered sample.

[0071] The verification automatic speech recognition model can be an ASR model with a completely different architecture, vendor, or configuration.

[0072] Specifically, the WER (Word Error Rate) and CER (Character Error Rate) calculation methods can be selected according to the target language. The difference between the text in the retained samples and the corresponding verification text is then calculated, and the verification text, WER, and CER are written into the sample metadata to form an automatic annotation quality closed loop. This embodiment does not rely on manual line-by-line verification. Compared to directly inputting a single ASR output into the database, it provides quantifiable evidence for subsequent data filtering, training sample weighting, manual review priority ranking, and quality report generation, thereby forming a traceable speech-text quality closed loop.

[0073] The retained samples can be normalized, and a stable text hash can be generated based on the normalized text to group samples with the same text or semantic equivalence into a group.

[0074] Specifically, speaker embedding vectors are extracted from samples within the same text hash group, and speaker similarity between samples is calculated. When two sample texts are identical and the speaker similarity exceeds a preset threshold, they are determined to be repeated speech-text pairs from the same speaker, and only one sample is retained. When the texts are identical but the speaker similarity is below the threshold, they are determined to be different speakers expressing the same content, and are retained. Thus, this approach can remove repeated content from the same speaker while retaining valuable training samples of the same text but different speakers in multi-speaker scenarios.

[0075] This refined deduplication mechanism, which integrates text fingerprints and speaker representations into a speech-text pair system, combines text consistency assessment with speaker identity similarity assessment to avoid mistakenly deleting valid samples from different speakers. Compared to deduplication methods based solely on text hashes or audio fingerprints, this mechanism can remove duplicate content from the same speaker while retaining valuable training samples of the same text from different speakers in multi-speaker scenarios, thereby improving the diversity and effectiveness of the dataset.

[0076] It can output the final speech-text pairs, metadata file records, filtered sample records, and running lists, which have stronger interpretability, filterability, and maintainability, and facilitate subsequent model training, quality tracking, manual sampling, and incremental processing.

[0077] The metadata file records and retains the audio path, text, duration, sampling rate, source file, start and end time, speaker, audio language detection results, alignment score, DNSMOS score, verification text, WER, CER, etc. of the sample.

[0078] The original audio record may include, but is not limited to, information such as: audio ID, audio file path, optional text annotation, data source, and task batch.

[0079] The candidate segment record may include, but is not limited to, information such as: segment ID, original audio ID, segment start and end time, speaker identifier, number of speakers, audio language, language confidence, and segment audio data.

[0080] The retained sample records may include, but are not limited to: sample ID, output audio path, text content, segment duration, sampling rate, original file path, start and end time, speaker information, language detection information, forced alignment score, speech quality score (such as DNSMOS score), verification text, WER, CER, and other information.

[0081] The filtered sample record is used to store the filtered samples and the reasons for filtering. The filtered sample record may include, but is not limited to, information such as: sample ID, filtering stage, filtering reason, original file path, segment start and end time, language confidence, and speech quality score (such as DNSMOS score).

[0082] The running list stores the source audio processing status, which is used for incremental processing and resuming from breakpoints.

[0083] The complete processing chain is used to record the processing chain of each audio segment from the original audio to the final sample, thereby realizing the traceability, statistical accuracy and verifiability of the data construction process.

[0084] Through the above embodiments, large-scale raw audio can be automatically converted into high-quality, verifiable, traceable, and deduplicated speech-text pairs, improving the efficiency and quality of audio data cleaning, corpus construction, and training data preparation.

[0085] In this embodiment, when the input audio already contains reliable text annotations, the initial ASR can be skipped and the existing text can be used directly for forced alignment, or the initial ASR can be retained and used as the basis for verifying the existing text.

[0086] In some embodiments, when the processing scale is small, the above steps can also be performed serially without enabling multi-queue concurrent scheduling.

[0087] This embodiment features continuous pipeline scheduling, breakpoint continuation processing, and quality tracking capabilities, thereby improving the accuracy, stability, processing efficiency, and engineering scalability of voice-text pairs.

[0088] This embodiment is applicable to various tasks such as standardizing raw audio data, speaker-level segmentation, target language selection, automatic annotation, quality verification, and training dataset construction.

[0089] As can be seen from the above technical solutions, this invention, applied to a processing system built on a multi-stage pipeline scheduling mechanism, improves throughput and resource utilization. Based on a speech quality evaluation mechanism, it performs adaptive source separation for each candidate audio segment, solving the problems of unnecessary processing of high-quality audio, ineffective computation of low-quality audio, and lack of quantitative tracking of processing effects caused by a unified source separation strategy. By performing speaker separation for each candidate audio segment based on the speaker's voice and conditional speech activity detection segmentation for each speaker's speaking interval, it reduces problems such as multiple speakers mixing, sentence truncation, excessively long segments, or incomplete segment semantics, improving the usability of the final speech-text segment. Combining audio language detection and text language filtering ensures that the output speech-text pair meets the target language requirements from both the audio and text sides, while reducing processing time. The complete processing chain for generating speech-text pairs according to the audio processing data structure meets the requirements for traceable management.

[0090] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the large-scale audio data preprocessing device of the present invention. The large-scale audio data preprocessing device 11 operates on a processing system built upon a multi-stage pipeline scheduling mechanism. Each task unit in the processing system employs an independent input queue, output queue, and thread. The processing result of the previous task unit serves as the input to the next task unit. The system includes a standardization unit 110, a separation unit 111, a segmentation unit 112, a filtering unit 113, and a generation unit 114. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module / unit will be detailed in subsequent embodiments.

[0091] The standardization unit 110 is used to respond to a preprocessing instruction triggered by a large-scale audio list, identify the audio data to be processed in the audio list using a breakpoint continuation processing mechanism, and perform standardization processing on the audio data to be processed to obtain each candidate audio segment. The separation unit 111 is used to perform adaptive sound source separation on each candidate audio segment based on the speech quality evaluation mechanism to obtain the speaking voice and background sound of each candidate audio segment; The separation unit 111 is also used to perform speaker separation on each candidate audio segment according to the speaking voice of each candidate audio segment, so as to obtain the speaking range of each speaker; The segmentation unit 112 is used to perform conditional speech activity detection segmentation on the speaking interval of each speaker to obtain multiple candidate segments; The filtering unit 113 is used to perform audio language detection on the plurality of candidate segments to obtain detection results, and to filter the plurality of candidate segments according to the detection results to obtain a plurality of retained segments; The generation unit 114 is used to generate speech-text pairs based on the plurality of retained segments, and to generate a complete processing link for the speech-text pairs according to the audio processing data structure.

[0092] As can be seen from the above technical solutions, this invention, applied to a processing system built on a multi-stage pipeline scheduling mechanism, improves throughput and resource utilization. Based on a speech quality evaluation mechanism, it performs adaptive source separation for each candidate audio segment, solving the problems of unnecessary processing of high-quality audio, ineffective computation of low-quality audio, and lack of quantitative tracking of processing effects caused by a unified source separation strategy. By performing speaker separation for each candidate audio segment based on the speaker's voice and conditional speech activity detection segmentation for each speaker's speaking interval, it reduces problems such as multiple speakers mixing, sentence truncation, excessively long segments, or incomplete segment semantics, improving the usability of the final speech-text segment. Combining audio language detection and text language filtering ensures that the output speech-text pair meets the target language requirements from both the audio and text sides, while reducing processing time. The complete processing chain for generating speech-text pairs according to the audio processing data structure meets the requirements for traceable management.

[0093] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a preferred embodiment of the large-scale audio data preprocessing method of the present invention.

[0094] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a large-scale audio data preprocessing program.

[0095] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0096] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0097] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a large-scale audio data preprocessing program, but also to temporarily store data that has been output or will be output.

[0098] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing large-scale audio data preprocessing programs) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0099] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps described in the various embodiments of the large-scale audio data preprocessing methods above, for example... Figure 1 The steps are shown.

[0100] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a standardization unit 110, a separation unit 111, a segmentation unit 112, a filtering unit 113, and a generation unit 114.

[0101] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the large-scale audio data preprocessing method described in the various embodiments of the present invention.

[0102] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0103] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0104] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0105] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0106] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0107] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0108] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the computer device 1 and other computer devices.

[0109] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0110] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0111] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0112] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a large-scale audio data preprocessing method, and the processor 13 can execute the multiple instructions to achieve the following: In response to a preprocessing instruction triggered by a large-scale audio list, a breakpoint-based continuation processing mechanism is used to identify the audio data to be processed in the audio list, and the audio data to be processed is standardized to obtain each candidate audio segment. Based on the speech quality evaluation mechanism, adaptive sound source separation is performed on each candidate audio segment to obtain the speaking voice and background sound of each candidate audio segment; Speaker segmentation is performed on each candidate audio segment based on the speaking voice, to obtain the speaking range of each speaker; Conditional speech activity detection and segmentation are performed on the speaking interval of each speaker to obtain multiple candidate segments for parallel processing; Audio language detection is performed on the multiple candidate segments to obtain detection results, and the multiple candidate segments are filtered according to the detection results to obtain multiple retained segments; Based on the multiple retained segments, speech-text pairs are generated, and a complete processing chain for the speech-text pairs is generated according to the audio processing data structure.

[0113] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0114] It should be noted that all the data involved in this case was legally obtained.

[0115] If any AI models, software tools, or components not belonging to this company appear in the embodiments of this invention, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this invention has been obtained by an entity authorized (with the knowledge and consent) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0116] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0117] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0118] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0120] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0121] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0122] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for large-scale audio data preprocessing, characterized in that, This is applied to a processing system built on a multi-stage pipeline scheduling mechanism, wherein each task unit in the processing system adopts an independent input queue, output queue and thread, and the processing result of the previous task unit is used as the input of the next task unit. The large-scale audio data preprocessing method includes: In response to a preprocessing instruction triggered by a large-scale audio list, a breakpoint-based continuation processing mechanism is used to identify the audio data to be processed in the audio list, and the audio data to be processed is standardized to obtain each candidate audio segment. Based on the speech quality evaluation mechanism, adaptive sound source separation is performed on each candidate audio segment to obtain the speaking voice and background sound of each candidate audio segment; Speaker segmentation is performed on each candidate audio segment based on the speaking voice, to obtain the speaking range of each speaker; Conditional speech activity detection and segmentation are performed on the speaking interval of each speaker to obtain multiple candidate segments for parallel processing; Audio language detection is performed on the multiple candidate segments to obtain detection results, and the multiple candidate segments are filtered according to the detection results to obtain multiple retained segments; Based on the multiple retained segments, speech-text pairs are generated, and a complete processing chain for the speech-text pairs is generated according to the audio processing data structure.

2. The large-scale audio data preprocessing method as described in claim 1, characterized in that, The adaptive sound source separation performed on each candidate audio segment based on the speech quality evaluation mechanism yields the speech and background sounds of each candidate audio segment, including: A speech quality score for each candidate audio segment is generated based on a pre-configured objective speech quality assessment model. Obtain the preset low-quality threshold and high-quality threshold; For each candidate audio segment, if the speech quality score is less than the low quality threshold, the candidate audio segment is determined to be a low quality segment and filtered; or When the speech quality score is greater than the high-quality threshold, the candidate audio segment is determined to be a high-quality segment, and adaptive source separation is skipped; or When the speech quality score is greater than or equal to the low quality threshold and less than or equal to the high quality threshold, adaptive sound source separation is performed on the candidate audio segment to obtain the speaking voice and background sound of the candidate audio segment.

3. The large-scale audio data preprocessing method as described in claim 1, characterized in that, The step of performing speaker segmentation on each candidate audio segment based on the speaking voice of each candidate audio segment to obtain the speaking range of each speaker includes: For adjacent audio segments belonging to the same speaker in the candidate audio segments, the segments are merged according to the interval threshold to obtain the speaking range of each speaker; Specifically, segments with overlapping speakers, durations shorter than the duration threshold, or that do not meet preset requirements will be deleted or marked as non-compliant.

4. The large-scale audio data preprocessing method as described in claim 1, characterized in that, The conditional speech activity detection segmentation performed on the speaking interval of each speaker yields multiple candidate segments, including: Obtain the segmentation duration threshold and the duration of each speaker's speaking interval; For each speaker's speaking interval, if the duration of the speaking interval is less than or equal to the segmentation duration threshold, the speaking interval is retained as the candidate segment; or When the duration of the speech interval is greater than the segmentation duration threshold, the speech interval is segmented according to the preset segmentation constraints to obtain the multiple candidate segments.

5. The large-scale audio data preprocessing method as described in claim 1, characterized in that, The step of filtering the multiple candidate segments based on the detection results to obtain multiple retained segments includes: The candidate segments that fail to be detected, have a confidence level lower than the preset confidence level, or whose detected language does not match the target language are filtered from the plurality of candidate segments to obtain the plurality of retained segments.

6. The large-scale audio data preprocessing method as described in claim 1, characterized in that, The step of generating speech-text pairs based on the plurality of retained segments includes: The main automatic speech recognition model is invoked to transcribe each retained segment, resulting in each candidate text; Each candidate text is normalized to obtain multiple normalized texts; The multiple standardized texts are aligned with their corresponding preserved segments to obtain the alignment results. The audio in the alignment result is re-segmented according to the preset semantic re-segmentation rules to obtain the speech-text pair.

7. The large-scale audio data preprocessing method as described in claim 1, characterized in that, The complete processing chain for generating the speech-text pair according to the audio processing data structure includes: The speech-text pair is filtered according to a preset multidimensional filtering rule to obtain a retained sample set of filtered samples. The retained samples are written into the retained sample record, and the filtered samples are written into the filtered sample record. The retained samples are rewritten using the automatic speech recognition model to obtain the verification text. Calculate the difference between the text in the retained sample and the verification text, and write the verification text and the difference into a metadata file; Calculate the text hash of the retained samples, and group the retained samples according to the text hash; For each sample in a group, speaker embedding vectors are extracted, and the similarity between the speaker embedding vectors is calculated. Speaker embedding vectors with the same text and a similarity greater than the similarity threshold are identified as duplicate content of the same speaker, and the sample corresponding to one of the speaker embedding vectors is retained as a valid sample. Speaker embedding vectors with the same text and a similarity less than or equal to the similarity threshold are identified as valid samples of different speakers, and the valid samples are retained. Write the audio data to be processed into the original audio record, and write the candidate segments into the candidate segment record; The complete processing chain is obtained by integrating the original audio recording, the candidate segment recording, the retained sample recording, and the filtered sample recording.

8. A large-scale audio data preprocessing apparatus, characterized in that, The system operates on a multi-stage pipeline scheduling mechanism. Each task unit in the system uses an independent input queue, output queue, and thread. The processing result of the previous task unit serves as the input of the next task unit. The large-scale audio data preprocessing device includes: The standardization unit is used to respond to a preprocessing instruction triggered by a large-scale audio list, identify the audio data to be processed in the audio list using a breakpoint continuation processing mechanism, and perform standardization processing on the audio data to be processed to obtain each candidate audio segment. The separation unit is used to perform adaptive sound source separation for each candidate audio segment based on the speech quality evaluation mechanism, so as to obtain the speech and background sound of each candidate audio segment; The separation unit is also used to perform speaker separation on each candidate audio segment based on the speaking voice of each candidate audio segment, so as to obtain the speaking range of each speaker; The segmentation unit is used to perform conditional speech activity detection segmentation on the speaking interval of each speaker to obtain multiple candidate segments; A filtering unit is used to perform audio language detection on the plurality of candidate segments to obtain detection results, and to filter the plurality of candidate segments according to the detection results to obtain a plurality of retained segments; The generation unit is used to generate speech-text pairs based on the plurality of retained segments, and to generate a complete processing link for the speech-text pairs according to the audio processing data structure.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the large-scale audio data preprocessing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the large-scale audio data preprocessing method as described in any one of claims 1 to 7.

Images