Voice model training data set generation method and system, and electronic equipment

Abstract

The invention discloses a voice model training data set generation method and system and electronic equipment. The method comprises the following steps: obtaining an initial data set containing audio data, wherein the audio data is obtained from network crawling as a public data source and / or an existing database; and performing preliminary screening on the initial data set to obtain a first target data set, performing data size adjustment to obtain a second target data set, and performing data type conversion processing to obtain a third target data set, thereby realizing conversion between audio and text, namely converting audio data into corresponding text. And combining sample data included in the second target data set and the third target data set according to a corresponding relation to form audio text pairing data as a first initial training data set. And finally, carrying out sampling inspection processing to obtain a high-quality first target training data set for automatic speech recognition model training. Therefore, the generation efficiency and reliability of the training data set are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence speech technology, and in particular to a method, system and electronic device for generating speech model training datasets. Background Technology

[0002] Artificial intelligence speech technology is a key human-computer interaction technology centered on Automatic Speech Recognition (ASR) and speech synthesis (e.g., Text-to-Speech, TTS), primarily relying on the training and application of machine learning models. In the development of AI speech technology, the efficiency and reliability of the training dataset for speech model training are crucial to determining the training efficiency and performance of the speech model. Traditionally, the generation of speech model training datasets relies on manual operation, involving manual filtering, segmentation, and classification of massive amounts of data to generate training datasets for speech model training. This process not only consumes enormous human and time costs but also has low productivity, and it is difficult to maintain stable and consistent data quality. Therefore, achieving automated, efficient, and reliable generation of speech model training datasets is extremely important. Summary of the Invention

[0003] The purpose of this application is to solve the problem of how to generate speech model training datasets in an automated, efficient, and reliable manner.

[0004] To address the aforementioned technical problems, in a first aspect, embodiments of this application disclose a method for generating a speech model training dataset. The method includes: obtaining an initial dataset, which includes multiple sample data points, each sample data point being multiple audio data points from publicly available data sources, and the multiple sample data points being crawled from the web and / or obtained from an existing database that serves as a publicly available data source; performing preliminary screening on the sample data points included in the initial dataset to obtain a first target dataset; performing data size adjustment on the sample data points included in the first target dataset to obtain a second target dataset; performing data type conversion on the sample data points included in the second target dataset to obtain a third target dataset, where the sample data points included in the third target dataset are text data; combining the sample data points included in the second target dataset and the sample data points included in the third target dataset according to data correspondence to obtain multiple audio-text sample data pairs as sample data to obtain a first initial training dataset; and performing sampling inspection on the sample data points included in the first initial training dataset to obtain a first target training dataset for training an automatic speech recognition model.

[0005] By employing the above method, an initial dataset is obtained by comprehensively utilizing web crawling and / or existing databases to acquire audio data from public data sources. This diversifies data sources and automates acquisition, laying the foundation for efficient dataset generation. The automated processing steps—data acquisition, preliminary screening, data size adjustment, data type conversion, and data sampling—replace traditional, tedious manual operations, thereby improving data processing efficiency, which in turn improves the efficiency of training dataset generation. Furthermore, compared to manual processing methods, it also enhances data reliability.

[0006] Furthermore, preliminary screening ensured data quality, data size adjustment optimized data applicability, and data type conversion enabled the conversion between audio and text, thus creating aligned data pairs. Sampling was used for data quality control to ensure the reliability of the generated dataset. The entire process of generating the speech model training dataset is automated and efficient, reducing manual intervention and improving the efficiency and reliability of the generated dataset.

[0007] This enables the generation of automated, efficient, and highly reliable speech model training datasets.

[0008] In one possible implementation of the first aspect above, the initial data, including the sample data, undergoes preliminary screening, which includes at least one of the following screening processes: sampling rate screening; audio format screening; audio integrity screening; and audio duplication screening.

[0009] By employing the above methods, the clarity and applicability of the audio are ensured through sampling rate screening, compatibility issues are avoided through audio format screening, incomplete data is eliminated through audio integrity screening, and redundant data is removed through audio duplication screening. These preliminary screening processes on the initial data, including the sample data, improve data quality, thereby reducing the impact of noisy data on model training, making subsequent processing more efficient, and the generated speech model training dataset more reliable.

[0010] In one possible implementation of the first aspect above, the data size adjustment processing of the sample data included in the first target dataset includes: segmenting the sample data included in the first target dataset using a multi-scale speech activity detection method or an endpoint detection method to segment at least a portion of the sample data included in the first target dataset into multiple data segments.

[0011] The above method involves segmenting the sample data using multi-scale speech activity detection or endpoint detection methods to divide at least a portion of the sample data into multiple data segments. This segmentation process avoids the computational burden caused by excessively long audio clips, thereby optimizing the data size and structure and improving the efficiency and reliability of the generated dataset.

[0012] In one possible implementation of the first aspect above, the data type conversion processing of the sample data included in the second target dataset to obtain the third target dataset includes: inputting the sample data included in the second target dataset into multiple automatic speech recognition models for data type conversion processing to obtain corresponding multiple model processing results, wherein the data type conversion processing is audio-to-text processing and the model processing result is text data corresponding to the audio data; comparing the multiple model processing results, determining the target model processing result based on a preset model processing result evaluation threshold, and obtaining the third target dataset based on the target model processing result.

[0013] Using the above method, sample data is input into multiple automatic speech recognition models for conversion, resulting in multiple model processing results. These results are then compared, and the target model's processing result is determined based on a preset evaluation threshold. This approach reduces the bias of a single model by using multiple models, and the comparative processing selects the best result, improving the quality of data conversion and thus enhancing the efficiency and reliability of dataset generation.

[0014] In one possible implementation of the first aspect described above, the sampling process of the sample data included in the first initial training dataset to obtain a first target training dataset for training an automatic speech recognition model includes: presenting the sample data included in the first initial training dataset to a target user, and having the target user perform sampling processing on the audio-text sample data pairs included in the first initial training dataset to remove or adjust unqualified sample data pairs to obtain a first target training dataset for training an automatic speech recognition model; or performing sampling processing on the sample data included in the first initial training dataset based on a target sampling algorithm to remove or adjust unqualified sample data pairs to obtain a first target training dataset for training an automatic speech recognition model.

[0015] By employing the above method, random sampling can be conducted by target users or based on a target sampling algorithm to eliminate or adjust unqualified data pairs, thereby further improving the efficiency and reliability of the generated dataset.

[0016] In one possible implementation of the first aspect above, the method further includes: performing audio enhancement processing on the sample data included in the second target dataset to obtain a fourth target dataset; combining the sample data included in the fourth target dataset and the sample data included in the third target dataset according to the data correspondence to obtain multiple text-audio data pairs as sample data to obtain a second initial training dataset; and performing sampling processing on the sample data included in the second initial training dataset to obtain a second target training dataset for training the text-to-speech model.

[0017] Using the above method, sample data for training a text-to-speech model can be directly obtained from the audio and text data in the sample data used for training the automatic speech recognition model, thus obtaining the corresponding training dataset. This improves data utilization and increases the efficiency and cost of generating the training dataset. Furthermore, by adding audio enhancement processing, the purity of the speech data can be effectively improved, significantly reducing the interference of environmental noise on model training. This allows the trained text-to-speech model to produce cleaner audio output with less background noise when synthesizing speech, thereby improving the reliability of the generated training dataset.

[0018] In one possible implementation of the first aspect above, the audio enhancement processing includes at least one of the following processes: audio background noise removal processing; audio restoration and reconstruction processing.

[0019] In one possible implementation of the first aspect above, the method further includes: storing and / or archiving a first target training dataset for use in training an automatic speech recognition model; and storing and / or archiving a second target training dataset for use in training a text-to-speech model.

[0020] By employing the above methods, data is securely stored and easily retrieved through storage and archiving, thereby supporting long-term model training needs, reducing data management costs, and improving the efficiency and reliability of the entire dataset generation process.

[0021] Secondly, this application discloses a system for generating a speech model training dataset. The system includes: a data acquisition module, a data filtering module, a data size adjustment module, a data type conversion module, a first data combination module, and a first data sampling module. The data acquisition module acquires an initial dataset, which includes multiple sample data points. These sample data points are multiple audio data points from publicly available data sources, and the sample data points are obtained by crawling from the web and / or from existing databases that are also publicly available data sources. The data filtering module performs preliminary filtering on the sample data included in the initial dataset to obtain a first target dataset. The data size adjustment module adjusts the data size of the first target dataset. The dataset includes sample data that has undergone data size adjustment to obtain a second target dataset; a data type conversion module is used to convert the data types of the sample data included in the second target dataset to obtain a third target dataset, which includes text data; a first data combination module is used to combine the sample data included in the second and third target datasets according to the data correspondence to obtain multiple audio-text sample data pairs as sample data to obtain a first initial training dataset; a first data sampling module is used to sample the sample data included in the first initial training dataset to obtain a first target training dataset for training an automatic speech recognition model.

[0022] In one possible implementation of the second aspect described above, the system further includes an audio enhancement module, a second data combination module, and a second data sampling module. The audio enhancement module is used to perform audio enhancement processing on the sample data included in the second target dataset to obtain a fourth target dataset. The second data combination module is used to combine the sample data included in the fourth target dataset and the sample data included in the third target dataset according to the data correspondence relationship to obtain multiple text-audio data pairs as sample data to obtain a second initial training dataset. The second data sampling module is used to perform sampling processing on the sample data included in the second initial training dataset to obtain a second target training dataset for training the text-to-speech model.

[0023] Thirdly, this application discloses an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to cause the electronic device to execute the above-described method for generating the speech model training dataset.

[0024] The relevant beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0026] Figure 1 A flowchart illustrating a method for generating a speech model training dataset according to an embodiment of this application;

[0027] Figure 2 A schematic diagram illustrating a process for generating a training dataset for a corresponding speech model based on audio data, as provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram illustrating a process for generating a training dataset for a corresponding automatic speech recognition model based on audio data, as provided in an embodiment of this application;

[0029] Figure 4 A schematic diagram of a process for generating a training dataset for a text-to-speech model based on sample data corresponding to an automatic speech recognition model, as provided in an embodiment of this application;

[0030] Figure 5 A schematic diagram of the structure of a speech model training dataset generation system provided in an embodiment of this application;

[0031] Figure 6 A schematic diagram of the structure of a speech model training dataset generation system provided in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] As mentioned earlier, artificial intelligence speech technology, with Automatic Speech Recognition (ASR) and speech synthesis (e.g., Text-to-Speech, TTS) as its core key technologies for human-computer interaction, relies heavily on large-scale, high-quality machine learning model training for its development. In the development of artificial intelligence speech technology, the efficiency and quality of the datasets generated for speech model training determine the convergence speed, final performance, and practical application effects of the model training.

[0034] Traditional methods for generating training datasets for speech models heavily rely on manual operations, requiring professionals to manually filter, clean, segment, label, and align massive amounts of raw audio and text data. This process not only consumes enormous human and time resources, resulting in lengthy data preparation cycles, but also suffers from inherent subjectivity and inconsistency, making it difficult to maintain uniformity in format, quality, and length standards. This leads to significant fluctuations in data quality, which in turn affects the accuracy of the trained model. Furthermore, facing the ever-increasing demand for multilingual, multi-accent, and multi-scenario speech interaction, traditional methods are no longer sufficient to meet the training requirements of modern large-scale deep learning models in terms of the scale, diversity, and efficiency of data generation.

[0035] Therefore, it is extremely important to achieve automated, efficient, and reliable generation of speech model training datasets.

[0036] Based on the above issues, such as Figure 1 As shown, an embodiment of this application discloses a method for generating a speech model training dataset, the method comprising:

[0037] Step S100: Obtain an initial dataset, which includes multiple sample data, which are multiple audio data from public data sources, and the multiple sample data are obtained from the web as a public data source and / or from an existing database as a public data source.

[0038] Step S200: Perform preliminary screening on the sample data included in the initial dataset to obtain the first target dataset.

[0039] Step S300: Perform data size adjustment on the sample data included in the first target dataset to obtain the second target dataset.

[0040] Step S400: Perform data type conversion processing on the sample data included in the second target dataset to obtain the third target dataset, wherein the sample data included in the third target dataset is text data.

[0041] Step S500: Combine the sample data included in the second target dataset and the sample data included in the third target dataset according to the data correspondence to obtain multiple audio text sample data pairs as sample data to obtain the first initial training dataset.

[0042] Step S600: The sample data included in the first initial training dataset is sampled and processed to obtain the first target training dataset for training the automatic speech recognition model.

[0043] The audio data includes raw audio files of varying lengths, such as video audio, podcast recordings, and meeting transcripts, crawled from the internet. Of course, the audio data can also be other data obtained through web crawling and / or existing databases. Existing databases could be, for example, private databases that are not publicly available online. In other words, the audio data originates from publicly available data sources. Furthermore, the audio data can be unlabeled audio data from publicly available data sources, i.e., unannotated audio data from publicly available data sources.

[0044] Furthermore, the first target dataset refers to the data set obtained after the initial dataset has undergone preliminary screening; the second target dataset refers to the data set obtained after the first target dataset has undergone data size adjustment; and the third target dataset refers to the data set obtained after the second target dataset has undergone data type conversion. The data type of the data included in the third target dataset is the opposite of that of the initial dataset. That is, the third target dataset is another data type that is paired with the second target dataset after modal conversion. For example, if the initial dataset is audio data, the third target dataset obtained after data type conversion is text data identified and converted from the audio data.

[0045] By employing the above method, an initial dataset is obtained by comprehensively utilizing web crawling and / or existing databases to acquire unlabeled audio data from public data sources. This diversifies data sources and automates acquisition, laying the foundation for efficient dataset generation. The automated processing steps—data acquisition, preliminary screening, data size adjustment, data type conversion, and data sampling—replace traditional, tedious manual operations, thereby improving data processing efficiency, which in turn improves the efficiency of training dataset generation. Furthermore, compared to manual processing methods, it also enhances data reliability.

[0046] Furthermore, preliminary screening ensured data quality, data size adjustment optimized data applicability, and data type conversion enabled the conversion between audio and text, thus creating aligned data pairs. Sampling was used for data quality control to ensure the reliability of the generated dataset. The entire process of generating the speech model training dataset is automated and efficient, reducing manual intervention and improving the efficiency and reliability of the generated dataset.

[0047] This enables the generation of automated, efficient, and highly reliable speech model training datasets.

[0048] In one possible implementation of the above method, the initial data, including sample data, undergoes preliminary screening, including at least one of the following screening processes: sampling rate screening; audio format screening; audio integrity screening; and audio duplication screening.

[0049] Sampling rate filtering refers to the process of selecting audio data with a specific sampling rate from the acquired audio data. For example, selecting only 16kHz audio data from the acquired audio data. Alternatively, sampling rate filtering can also refer to the process of uniformly converting the audio sampling rate to a standard sampling rate. For example, reading the original sampling rate of the audio data (e.g., 48kHz) and using a resampling algorithm, such as linear interpolation or a sine function, to convert the audio data to a preset standard sampling rate (e.g., 16kHz).

[0050] Audio format filtering refers to the process of selecting audio data of a preset format from acquired audio data. For example, selecting only .mp3 audio data from acquired audio data. Further, audio format filtering also includes converting audio data to a single channel or selecting single-channel audio data from audio data. That is, the audio data after audio format filtering is single-channel audio data of a uniform audio format. Alternatively, audio format filtering refers to the process of batch converting multiple audio formats to a uniform format. For example, identifying the container format of the audio data (e.g., .mp3), and then using an audio processing library to batch decode and re-encode it into a uniform standard format (e.g., .wav).

[0051] Audio integrity screening refers to the process of verifying and removing damaged or incomplete audio data files. For example, it checks whether the information at the beginning of the audio data file is correct and complete. A damaged audio data file header generally cannot be opened normally, thus it is incomplete audio data. It also checks whether the size of the audio data file is abnormal, such as 0 KB. If it is abnormal, it is incomplete audio data. It attempts to fully decode the audio file. If an error occurs during the decoding process, it terminates prematurely, or the complete audio stream cannot be read, it is determined to be incomplete audio data, and damaged or other unacceptable audio data is removed from the audio data.

[0052] Audio duplication filtering refers to the process of removing duplicate audio content through acoustic fingerprint comparison. For example, after preprocessing audio data files, key acoustic features are extracted, such as significant peaks in the spectrogram. A fingerprint is generated using a hash algorithm. When processing new audio, its acoustic fingerprint is calculated and queried against a database of fingerprints of processed audio. If the same or highly similar fingerprints are found, it is determined to be duplicate content and is removed or ignored.

[0053] By employing the above methods, the clarity and applicability of the audio are ensured through sampling rate screening, compatibility issues are avoided through audio format screening, incomplete data is eliminated through audio integrity screening, and redundant data is removed through audio duplication screening. These preliminary screening processes on the initial data, including the sample data, improve data quality, thereby reducing the impact of noisy data on model training, making subsequent processing more efficient, and the generated speech model training dataset more reliable.

[0054] Of course, the initial screening of audio data can also be done using other methods besides those mentioned above, which can be set as needed.

[0055] In one possible implementation of the above method, the data size adjustment processing of the sample data included in the first target dataset includes: segmenting the sample data included in the first target dataset using a multi-scale speech activity detection method or an endpoint detection method to segment at least a portion of the sample data included in the first target dataset into multiple data segments.

[0056] Among them, multi-scale speech activity detection and endpoint detection in audio segmentation processing refers to extracting the effective speech portion from long audio data, such as lectures, interviews, and podcasts, and segmenting it into short audio data suitable for training input of speech models.

[0057] Furthermore, the endpoint detection method refers to finding the start and end of the speech data signal and segmenting it into audio segments. For example, a continuous audio data signal is divided into small frames, and the volume and frequency of each frame crossing zero are calculated. When the volume of multiple consecutive frames suddenly jumps from a low level and continues to exceed a preset threshold, it is determined as the start of the speech data signal; otherwise, it is determined as the end of the speech data signal. The audio data is then segmented based on the found start and end points of the speech data signal.

[0058] Multi-scale speech activity detection (VAD) refers to a more advanced endpoint detection method that can analyze audio data signals at different temporal granularities to adapt to changes in speech rate, pauses, and complex background noise. It can more accurately determine whether each frame is speech or non-speech, enabling more accurate audio segmentation. For example, it can extract more complex acoustic features than endpoint detection and use machine learning classifiers to analyze the features of each frame at multiple scales for audio data segmentation.

[0059] Furthermore, in the process of scaling audio data, adjustments can be made using a multi-level scaling approach, starting with adjustments at a large scale and then at a small scale.

[0060] The above method involves segmenting the sample data using multi-scale speech activity detection or endpoint detection methods to divide at least a portion of the sample data into multiple data segments. This segmentation process avoids the computational burden caused by excessively long audio clips, thereby optimizing the data size and structure and improving the efficiency and reliability of the generated dataset.

[0061] Of course, the data size adjustment processing for audio data can also be done in other ways besides those mentioned above, and can be set as needed.

[0062] In one possible implementation of the above method, the sample data included in the second target dataset is subjected to data type conversion processing to obtain the third target dataset. This includes: inputting the sample data included in the second target dataset into multiple automatic speech recognition models for data type conversion processing to obtain corresponding multiple model processing results. The data type conversion processing is audio-to-text processing, and the model processing result is text data corresponding to the audio data. The multiple model processing results are compared and processed. Based on a preset model processing result evaluation threshold, the target model processing result is determined, and the third target dataset is obtained based on the target model processing result.

[0063] The automatic speech recognition model can be called Automatic Speech Recognition (ASR). The data type conversion process is implemented as follows: sample data from the second target dataset is input in parallel into multiple ASR models for processing. When processing audio data, audio-to-text conversion is performed, resulting in multiple model processing results for the same input. These results are compared and the best is selected. For example, for audio-to-text data, the confidence scores of the output text of each model and the consistency between the results are considered, i.e., cross-validation is performed by calculating word error rate and text similarity. For text-to-audio data, the synthesized audio is input back into the ASR model for verification, and the word error rate is used as an indicator to evaluate the fit between the synthesized content and the original text. Based on preset multi-dimensional evaluation thresholds, the optimal processing result is automatically selected. This processing result is then integrated according to the original correspondence to form a corresponding initial audio dataset composed of text data or a corresponding initial text dataset composed of audio data.

[0064] Furthermore, the evaluation threshold can be set as needed, and based on the evaluation threshold, the target model processing result is determined. This can be achieved, for example, by calculating the evaluation score or confidence score corresponding to each model processing result, and using the model processing result whose evaluation score or confidence score is greater than the evaluation threshold as the target evaluation result, that is, as the model processing result with higher accuracy. The target evaluation result and its corresponding data are then selected from the model processing results as sample data in the third target dataset.

[0065] Using the above method, sample data is input into multiple automatic speech recognition models for conversion, resulting in multiple model processing results. These results are then compared, and the target model's processing result is determined based on a preset evaluation threshold. Using multiple models reduces the bias of a single model, and the comparative processing selects the best result, improving the quality of data conversion. This enhances the efficiency and reliability of generating the dataset.

[0066] In one possible implementation of the above method, the sample data included in the first initial training dataset is subjected to sampling inspection to obtain a first target training dataset for training an automatic speech recognition model. This includes: presenting the sample data included in the first initial training dataset to a target user, who then performs sampling inspection on the audio-text sample data pairs included in the first initial training dataset to remove or adjust unqualified sample data pairs, thereby obtaining the first target training dataset for training an automatic speech recognition model; or performing sampling inspection on the sample data included in the first initial training dataset based on a target sampling inspection algorithm to remove or adjust unqualified sample data pairs, thereby obtaining the first target training dataset for training an automatic speech recognition model.

[0067] Ineligible data refers to data where the audio and text content do not match, such as transcription errors, omissions, or disordered word order. Data with significant defects, such as completely irrelevant content, completely corrupted audio, or data containing sensitive information, is removed. Data that is partially repairable, such as individual word errors, improper punctuation, or minor audio flaws, is corrected by revising the text or processing the audio.

[0068] Furthermore, the audio-text data pair can be, for example, a recording of "hello" and the text "hello" data pair, or other audio-text data pairs.

[0069] By employing the above method, random sampling is conducted either manually by target users or based on a target sampling algorithm to eliminate or adjust unqualified data pairs. This further improves the efficiency and reliability of the generated dataset.

[0070] In one possible implementation of the above method, the method further includes: storing and / or archiving the first target training dataset for use in training an automatic speech recognition model.

[0071] By employing the above methods, data is securely stored and easily retrieved through storage and archiving, thereby supporting long-term model training needs, reducing data management costs, and improving the efficiency and reliability of the entire dataset generation process.

[0072] The following section uses the method for generating training datasets based on Automatic Speech Recognition (ASR) models as an example to further illustrate the method for generating training datasets for speech models provided in this embodiment.

[0073] The first target dataset refers to the data set obtained after the initial dataset has undergone preliminary screening. The second target dataset refers to the data set obtained after the first target dataset has undergone data size adjustment. The third target dataset refers to the data set obtained after the second target dataset has undergone data type conversion. Its data type is the opposite of the initial dataset. That is, the third target dataset is another data type that is paired with the second target dataset after modal conversion. For example, if the initial dataset is audio data, the third target dataset obtained after data type conversion is the text data identified and converted from the audio data.

[0074] Furthermore, audio data includes raw audio files of varying lengths, such as video audio, podcast recordings, and meeting transcripts, crawled from the internet. Of course, audio data can also be other data obtained through web crawling and / or existing databases. Existing databases could be, for example, private databases that are not publicly available online.

[0075] like Figure 2 As shown, the method for obtaining a training dataset for ASR model training based on audio data includes the following steps.

[0076] Step S201: Use web crawlers to extract audio data from internet resources and integrate it with existing audio data in the database to form an initial dataset.

[0077] The web crawling audio data process is an automated data processing pipeline that primarily handles raw audio data that is crawled or collected but lacks text annotations. This allows for the convenient and rapid acquisition of richer audio data based on publicly available data sources.

[0078] Step S202: Perform at least one of the aforementioned sampling rate filtering, audio format filtering, audio integrity filtering, and audio duplication filtering on the audio data.

[0079] Step S203: Apply the aforementioned multi-scale speech activity detection (VAD) or endpoint detection techniques to the filtered audio data to cut long audio segments into short audio segments of appropriate duration.

[0080] Step S204: Input short audio into multiple ASR models in parallel for recognition, achieve text annotation, compare the generated text results, and select the optimal text based on a preset confidence threshold.

[0081] Step S205: Automatically combine the resized data and the type-converted data according to their correspondence to form audio-text pairing data, thus forming the first initial training dataset.

[0082] For audio-to-text data, the confidence scores of the output texts from each model and the consistency between the results are considered. This is achieved through cross-validation using word error rate and text similarity calculations, and the optimal processing result is automatically selected based on preset multi-dimensional evaluation thresholds. These processing results are then integrated according to the original correspondences to form the corresponding initial audio dataset composed of text data.

[0083] Step S206: Perform final quality verification on the first initial training dataset by manual sampling or algorithm-based automatic sampling, remove or correct unqualified data pairs, and obtain the first target training dataset for training the automatic speech recognition model.

[0084] Ineligible data refers to data where the audio and text content do not match, such as transcription errors, omissions, or disordered word order. Data with significant defects, such as completely irrelevant content, completely corrupted audio, or data containing sensitive information, is removed. Data that is partially repairable, such as individual word errors, improper punctuation, or minor audio flaws, is corrected by revising the text or processing the audio.

[0085] Furthermore, the final qualified first target training dataset is standardized, stored, and versioned, and relevant metadata is recorded for use in model training.

[0086] Furthermore, such as Figure 3As shown, in this embodiment, the audio data processing flow (i.e., the ASR path) is an automated production process from raw audio to text data. Audio in languages ​​such as Chinese and English is obtained through web crawling. The obtained audio undergoes initial screening, for example, setting the sampling rate to 16kHz (i.e., if the sampling rate is not suitable, it will be resampled to 16kHz), and single-channel processing (i.e., if it is multi-channel, it will be merged into a single channel, and audio data from different sources will be formatted uniformly). Lossy audio files are deleted or removed, and global deduplication is performed using a remote dictionary server (Redis). Large VADs are used for coarse segmentation, and small VADs for fine segmentation to extract effective speech segments. These segments are then fed into two large speech recognition models, ASR1 and ASR2, for conversion. The screening results are processed using corresponding filtering strategies, such as using a voting mechanism to compare the text results labeled by multiple speech recognition models to select the best quality text data. The data can be filtered using cosine similarity, with the amount of data to be filtered set to, for example, 2 million. The filtering threshold is to be determined and can be set as needed. The data format is, for example, lossless audio format (FLAC). Then, the filtered text data is combined with its corresponding audio data input to the ASR model to obtain corresponding audio-text data pairs, which serve as the final filtering results. During the data platform sampling phase, the filtering results undergo self-checking and algorithm sampling. After that, the data is archived through cloud document archiving and backup machine archiving to obtain an audio-text dataset that can be used for ASR model training.

[0087] Based on the above, the data acquisition process for training the ASR model provided in this application embodiment can be understood as follows: first, acquire the aforementioned publicly available speech data (e.g., data without text annotations obtained through crawlers or other channels), then perform audio quality detection and other data filtering, then perform data segmentation based on a multi-scale VAD model, then perform data labeling based on a multi-ASR model, and finally archive the data to obtain the data for training the ASR model.

[0088] By employing the above method and constructing a fully automated data processing pipeline, efficient and high-quality generation of speech training data is achieved. Specifically, intelligent integration of web crawlers and existing databases automates data acquisition and classification; targeted initial screening rules ensure the purity of the data source; multi-scale speech detection standardizes data size; an innovative multi-ASR model comparison and selection mechanism significantly improves the alignment accuracy of audio-text pairs; and a sampling strategy combining manual and algorithmic checks ensures the reliability of the final dataset. This method completely changes the traditional data preparation model that relies on manual intervention, significantly reducing labor costs while generating larger-scale, higher-quality speech training data with broader coverage, effectively supporting the development and iteration of high-performance speech models.

[0089] Furthermore, in one embodiment of this application, based on the aforementioned audio and text data, a training dataset for training a text-to-speech model can also be generated. Thus, as... Figure 4 As shown, the method for generating the speech model training dataset disclosed in this application further includes the following steps:

[0090] Step S301: Perform audio enhancement processing on the sample data included in the second target dataset to obtain the fourth target dataset.

[0091] Step S302: Combine the sample data included in the fourth target dataset and the sample data included in the third target dataset according to the data correspondence to obtain multiple text-audio data pairs as sample data to obtain the second initial training dataset.

[0092] Step S303: Perform sampling processing on the sample data included in the second initial training dataset to obtain the second target training dataset for training the text-to-speech model.

[0093] In one possible implementation of the first aspect above, the audio enhancement processing includes at least one of the following processes: audio background noise removal processing; audio restoration and reconstruction processing.

[0094] Audio background noise removal refers to separating and eliminating environmental noise that is not part of human voice in audio, such as background music (BGM) and other ambient noise. It involves using algorithms, such as deep learning networks, to analyze the audio signal, recovering clean speech from the noisy background, and then subtracting the noise component from the original audio's spectrum to reconstruct a cleaner human voice signal. For example, audio background noise removal can effectively remove continuous air conditioner hum, keyboard typing, or background conversations from a meeting recording.

[0095] Audio restoration and reconstruction processing is used to repair damaged or severely distorted audio. It utilizes advanced signal processing algorithms or deep learning models to fill in missing, broken, or severely contaminated parts of the audio. For example, for old record recordings that produce crackling sounds due to aging storage media, audio restoration and reconstruction processing can be used to locate and interpolate the damaged points.

[0096] Of course, audio enhancement processing can also be performed using methods other than those mentioned above, which can be configured as needed.

[0097] In other words, the fourth target dataset is a dataset obtained by audio enhancement processing of the second target dataset. Its input is short audio segments that have been resized and cut, and its output is the same batch of audio with improved quality.

[0098] Using the above method, sample data for training a text-to-speech model can be directly obtained from the audio and text data in the sample data used for training the automatic speech recognition model, thus obtaining the corresponding training dataset. This improves data utilization and increases the efficiency and cost of generating the training dataset. Furthermore, by adding audio enhancement processing, the purity of the speech data can be effectively improved, significantly reducing the interference of environmental noise on model training. This allows the trained text-to-speech model to produce cleaner audio output with less background noise when synthesizing speech, thereby improving the reliability of the generated training dataset.

[0099] In other words, based on the above, in the embodiments provided in this application, the training ASR model data obtained above can be further processed to obtain the training TTS model data. The process of obtaining the training TTS model data can be understood as follows: first, based on the above process of obtaining the training ASR model data, the corresponding training ASR model data is obtained; then, the aforementioned speech enhancement and other processing are performed, as well as quality screening processing; finally, the data is archived to obtain the training TTS model data.

[0100] Further, the sample data included in the second initial training dataset is subjected to sampling inspection to obtain a second target training dataset for text-to-speech model training. This includes: presenting the sample data included in the second initial training dataset to the target user, who then performs sampling inspection on the audio-text sample data pairs included in the second initial training dataset to remove or adjust unqualified sample data pairs, thereby obtaining a second target training dataset for text-to-speech model training; or performing sampling inspection on the sample data included in the second initial training dataset based on a target sampling inspection algorithm to remove or adjust unqualified sample data pairs, thereby obtaining a second target training dataset for text-to-speech model training.

[0101] As mentioned above, unqualified data refers to defective data such as data where the audio and text content do not match. It can be set according to the requirements of the training dataset corresponding to the text-to-speech model.

[0102] Furthermore, the second target training dataset can be stored and / or archived for use in text-to-speech model training.

[0103] This application also discloses a system for generating a speech model training dataset, such as... Figure 5 As shown, the system includes: a data acquisition module, a data filtering module, a data size adjustment module, a data type conversion module, a first data combination module, and a first data sampling module. The data acquisition module acquires an initial dataset, which includes multiple sample data points. These sample data points are audio data from multiple public data sources, obtained through web crawling and / or from existing databases that serve as public data sources. The data filtering module performs preliminary filtering on the sample data in the initial dataset to obtain a first target dataset. The data size adjustment module adjusts the data size of the sample data in the first target dataset to obtain a second target dataset. The data type conversion module converts the data type of the sample data in the second target dataset to obtain a third target dataset, which includes text data. The first data combination module combines the sample data from the second and third target datasets according to data correspondences to obtain multiple audio-text sample data pairs, which serve as the first initial training dataset. The first data sampling module performs sampling on the sample data in the first initial training dataset to obtain the first target training dataset used for training the automatic speech recognition model.

[0104] Furthermore, such as Figure 6 As shown, in one possible implementation of the above system, the system further includes an audio enhancement module, a second data combination module, and a second data sampling module. The audio enhancement module is used to perform audio enhancement processing on the sample data included in the second target dataset to obtain a fourth target dataset. The second data combination module is used to combine the sample data included in the fourth target dataset and the sample data included in the third target dataset according to the data correspondence relationship to obtain multiple text-audio data pairs as sample data to obtain a second initial training dataset. The second data sampling module is used to perform sampling processing on the sample data included in the second initial training dataset to obtain a second target training dataset for training the text-to-speech model.

[0105] Furthermore, corresponding to the aforementioned Figure 3Each of the squares can be understood as a functional module used to implement the corresponding content, or as a sub-module of the above module.

[0106] Therefore, the embodiments of this application provide an automated production system for high-quality speech model training data based on multi-module collaborative processing. Through the collaborative processing of multiple modules, it realizes efficient production and quality improvement of speech model training data, meets the training needs of multiple scenarios and languages, significantly reduces the cost of manual intervention, and realizes automated, efficient and reliable generation of speech model training datasets.

[0107] This application discloses an electronic device for implementing a method for generating the aforementioned speech model training dataset. For example... Figure 7 As shown, the electronic device may include: transceiver 121, processor 122, and memory 123.

[0108] Processor 122 executes computer execution instructions stored in memory, causing processor 122 to perform the technical solution of the speech model training dataset generation method in the above embodiments. Processor 122 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital data processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0109] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0110] For example, and not as a limitation, memory 123 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 123 may include removable or non-removable (or fixed) media. Where appropriate, memory 123 may be internal or external to the integrated gateway device. In a particular embodiment, memory 123 is non-volatile solid-state memory. In a particular embodiment, memory 123 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only ROM (PROM), an erasable programmable read-only ROM (EPROM), an electrically erasable programmable read-only ROM (EEPROM), an electrically alterable read-only ROM (EAROM), or flash memory, or a combination of two or more of these. Transceiver 121 can be used to obtain the task to be run and its configuration information.

[0111] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between the database access system and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0112] Furthermore, the electronic device can be, for example, a computer, a mobile phone, a server, a vehicle, or other electronic devices.

[0113] This application also provides a chip for executing instructions, which is used to execute the aforementioned method for generating a speech model training dataset.

[0114] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by an electronic device, cause the electronic device to perform the aforementioned method for generating a speech model training dataset.

[0115] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the aforementioned method for generating a speech model training dataset.

[0116] It should be noted that, in addition to the specific embodiments described above, those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this application are limited to these embodiments. On the contrary, the purpose of describing the application in conjunction with the embodiments is to cover other options or modifications that may be derived based on the claims of this application. To provide a thorough understanding of this application, many specific details are included in the above description, and this application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0117] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0118] The terms “first”, “second”, etc., are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0119] Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art should understand that the above description is a further detailed explanation of the application in conjunction with specific embodiments, and should not be construed as limiting the specific implementation of the application to these descriptions. Those skilled in the art can make various changes in form and detail, including some simple deductions or substitutions, without departing from the spirit and scope of this application.

Claims

1. A method for generating a speech model training data set, characterized in that, The method comprises: obtaining an initial data set comprising a plurality of sample data, the plurality of sample data being a plurality of audio data from a public data source, and the plurality of sample data being obtained from web crawling as the public data source and / or from an existing database as the public data source; performing preliminary screening processing on the sample data included in the initial data set to obtain a first target data set; performing data size adjustment processing on the sample data included in the first target data set to obtain a second target data set; performing data type conversion processing on the sample data included in the second target data set to obtain a third target data set, the sample data included in the third target data set being text data; combining the sample data included in the second target data set and the sample data included in the third target data set according to a data correspondence relationship to obtain a plurality of audio text data pairs as sample data to obtain a first initial training data set; performing sampling inspection processing on the sample data included in the first initial training data set to obtain a first target training data set for automatic speech recognition model training.

2. The method of claim 1, wherein, The preliminary screening processing on the sample data included in the initial data set comprises at least one of the following screening processing: sampling rate screening processing; audio format screening processing; audio completeness screening processing; audio repetition screening processing.

3. The method of claim 2, wherein, The data size adjustment processing on the sample data included in the first target data set comprises: performing cutting processing on the sample data included in the first target data set by a multi-scale voice activity detection method or an endpoint detection method to cut at least part of the sample data included in the first target data into a plurality of data segments.

4. The method of claim 3, wherein, The data type conversion processing on the sample data included in the second target data set to obtain a third target data set comprises: inputting the sample data included in the second target data set into a plurality of automatic speech recognition models respectively to perform data type conversion processing to obtain a plurality of corresponding model processing results, the data type conversion processing being audio-to-text processing, and the model processing result being text data corresponding to the audio data; performing comparison processing on the plurality of model processing results, determining a target model processing result based on a preset model processing result evaluation threshold, and obtaining the third target data set according to the target model processing result.

5. The method of claim 4, wherein, The sampling inspection processing on the sample data included in the first initial training data set to obtain a first target training data set for automatic speech recognition model training comprises: presenting the sample data included in the first initial training data set to a target user, performing sampling inspection processing on the sample data included in the first initial training data set by the target user to eliminate or adjust unqualified sample data to obtain a first target training data set for automatic speech recognition model training; or The target sampling algorithm is used to sample the sample data included in the first initial training data set, so as to eliminate or adjust the sample data that fails to pass the inspection, and obtain a first target training data set for automatic speech recognition model training.

6. The method of claim 5, wherein, The method further comprises: performing audio enhancement processing on the sample data included in the second target data set to obtain a fourth target data set; combining the sample data included in the fourth target data set and the sample data included in the third target data set according to a data correspondence relationship to obtain a plurality of text-audio data pairs as sample data to obtain a second initial training data set; performing sampling processing on the sample data included in the second initial training data set to obtain a second target training data set for text-to-speech model training.

7. The method of claim 6, wherein, The audio enhancement processing comprises at least one of the following processing: audio background noise removal processing; audio repair and reconstruction processing.

8. The method of claim 7, wherein, The method further comprises: storing and / or archiving the first target training data set for automatic speech recognition model training; and storing and / or archiving the second target training data set for text-to-speech model training.

9. A system for generating a speech model training dataset, the system comprising: The system comprises a data acquisition module, a data screening module, a data size adjustment module, a data type conversion module, a first data combination module, and a first data sampling module, wherein the data acquisition module is configured to acquire an initial data set, the initial data set comprising a plurality of sample data, the plurality of sample data being a plurality of audio data from a public data source, and the plurality of sample data being obtained from network crawling as the public data source and / or from an existing database as the public data source; the data screening module is configured to perform preliminary screening processing on the sample data included in the initial data set to obtain a first target data set; the data size adjustment module is configured to perform data size adjustment processing on the sample data included in the first target data set to obtain a second target data set; the data type conversion module is configured to perform data type conversion processing on the sample data included in the second target data set to obtain a third target data set, the sample data included in the third target data set being text data; the first data combination module is configured to combine the sample data included in the second target data set and the sample data included in the third target data set according to a data correspondence relationship to obtain a plurality of audio-text sample data pairs as sample data to obtain a first initial training data set; the first data sampling module is configured to perform sampling processing on the sample data included in the first initial training data set to obtain a first target training data set for automatic speech recognition model training.

10. An electronic device, comprising: comprises: a processor, and a memory connected in communication with the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to enable the electronic device to perform the method for generating a speech model training data set according to any one of claims 1-8.

Images