Corpus expansion method, system and equipment based on speech synthesis and medium

Abstract

The invention relates to the technical field of speech synthesis, in particular to a speech synthesis-based corpus expansion method, system and device and a medium, and the method comprises the steps: carrying out the preprocessing including data annotation based on the collected audio and corresponding text of a target speaker; extracting acoustic features from the preprocessed audio; on the basis of a pre-trained acoustic model, performing personalized fine tuning by using the annotation data and the acoustic features, and training personalized acoustic models of a plurality of speakers at the same time through multi-thread parallel computing; calling the trained personalized acoustic model, and synthesizing a voice corpus of the target text in combination with a vocoder; and based on the trained personalized acoustic model, continuously expanding the corpus by changing the text. The personalized voice corpus is quickly generated through a small number of voice samples, the data acquisition cost is remarkably reduced, and the corpus construction efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech synthesis, in particular to a corpus expansion method and system based on speech synthesis, a device and a medium. BACKGROUND

[0002] With the development of artificial intelligence technology, especially in the field of speech recognition and speech synthesis, whether it is to train a widely applicable base model, or to train an accurate and efficient speech recognition model, or a virtual announcer that fits a real person, high-quality speech corpus is essential for training an accurate and efficient model. In the traditional way, obtaining these corpora mainly relies on directly purchasing datasets or recording. However, this method has significant limitations. First, purchasing ready-made datasets is costly and often difficult to meet the needs of diversity for specific applications; second, self-recording not only requires professional recording equipment and environment, but also requires a large amount of time and human resources for data review and annotation, which greatly limits the expansion speed and flexibility of the corpus. In addition, in the face of different speakers' pronunciation characteristics and rhythm changes, existing methods often cannot efficiently provide sufficient diversity, thereby affecting the performance of the final model.

[0003] Therefore, how to generate large-scale and diversified speech corpora at low cost and high efficiency to support fast corpus expansion for personalized speech synthesis is a problem to be solved. SUMMARY

[0004] The main purpose of the present application is to provide a corpus expansion method and system based on speech synthesis, a device and a medium, aiming to solve the technical problem of how to generate large-scale and diversified speech corpora at low cost and high efficiency to support fast corpus expansion for personalized speech synthesis.

[0005] In order to achieve the above-mentioned purpose of the application, the present application provides a corpus expansion method based on speech synthesis, comprising: Based on the collected audio and corresponding text of the target speaker, pre-processing including data annotation is performed; Acoustic features are extracted from the pre-processed audio; Based on the pre-trained acoustic model, the annotated data and acoustic features are used for personalized fine-tuning, and multiple speaker personalized acoustic models are trained simultaneously through multi-thread parallel computing; The trained personalized acoustic model is called to synthesize the speech corpus of the target text in combination with the vocoder; Based on the trained personalized acoustic model, the corpus is continuously expanded by changing the text.

[0006] Further, the step of pre-processing including data labeling based on the collected audio and corresponding text of the target speaker includes: collecting no less than a first preset number of voice command word audios based on a recording device, wherein each audio has a corresponding text transcription; performing ambient noise elimination processing on the collected audio, and resampling the audio to a unified sampling rate and bit depth; aligning the audio with the text to generate timestamp labeling containing phoneme boundaries, and labeling the fundamental frequency track and energy change characteristics, and outputting a labeled file containing phoneme boundary, timestamp, fundamental frequency, and energy characteristic information.

[0007] Further, the step of extracting acoustic features from the pre-processed audio includes: extracting first acoustic features including mel-frequency cepstral coefficient, fundamental frequency, and energy feature information from each audio; based on the phoneme boundary labeling information generated in the pre-processing stage, extracting corresponding local acoustic features for each phoneme segment; based on the first acoustic features and the local acoustic features, obtaining the acoustic features.

[0008] Further, the step of using labeled data and acoustic features to fine-tune the pre-trained acoustic model includes: loading the pre-trained acoustic model and adding a speaker feature adaptation layer to the pre-trained acoustic model; fixing the main structure parameters of the pre-trained model, and using the labeled data and acoustic features to perform supervised training on the speaker feature adaptation layer; stop training when the validation set accuracy reaches a preset threshold, and save the personalized acoustic model containing the speaker feature adaptation layer parameters.

[0009] Further, the step of training multiple personalized acoustic models of multiple speakers simultaneously through multi-thread parallel computing includes: configure multiple computing servers within a local area network and initialize the computing environment of each server, wherein a network service is built through Flask to connect the computing servers, and multi-thread task allocation and communication is realized through JavaScript; dynamically allocate personalized acoustic model training tasks of different speakers to each computing server, and execute the training tasks in parallel through multi-threading; manage the training tasks of each computing server through independent sub-processes to avoid request conflicts; After training is completed, collect the personalized acoustic model results on all computing servers.

[0010] Furthermore, the step of calling the trained personalized acoustic model and combining it with a vocoder to synthesize the speech corpus of the target text includes: The target text is converted into a phoneme sequence, and corresponding prosodic feature tags are generated. Input the featured text into the personalized acoustic model and output acoustic parameters containing spectral and fundamental frequency features; The generated acoustic parameters are input into a neural vocoder to generate the original waveform audio. The original waveform audio is smoothed to obtain the speech corpus of the target text.

[0011] Furthermore, the trained personalized acoustic model is invoked and combined with a vocoder to synthesize speech corpora corresponding to new texts in batches; Add speaker identifiers to the synthesized speech corpus, generate and associate the corresponding text content, and complete the current expansion operation of the corpus.

[0012] The second aspect of this application proposes a corpus augmentation system based on speech synthesis, comprising: The preprocessing module is used to perform preprocessing, including data annotation, based on the collected audio of the target speaker and the corresponding text. The feature extraction module is used to extract acoustic features from the preprocessed audio. The personalized training module is used to fine-tune the pre-trained acoustic model using labeled data and acoustic features, and train personalized acoustic models for multiple speakers simultaneously through multi-threaded parallel computing. The corpus synthesis module is used to call the trained personalized acoustic model and combine it with a vocoder to synthesize the speech corpus of the target text. The text augmentation module is used to continuously expand the corpus by changing the text, based on the trained personalized acoustic model.

[0013] A third aspect of this application also includes a computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the methods described above.

[0014] The fourth aspect of this application also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0015] This application significantly improves the efficiency and quality of corpus production compared to traditional methods that rely on manual recording and annotation. Fine-tuning techniques based on pre-trained models ensure the quality of personalized speech synthesis, while multi-threaded parallel computing greatly enhances processing power, enabling the system to process speech data from multiple speakers simultaneously. This end-to-end solution not only lowers the barrier and cost of corpus acquisition but also ensures the diversity and usability of the generated corpus. More importantly, the personalized acoustic model established by this method can be continuously reused, providing stable and reliable data support for the research and application of speech technology. It enables the rapid generation of personalized speech corpora from a small number of speech samples, significantly reducing data collection costs and improving corpus construction efficiency. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a speech synthesis-based corpus augmentation method according to an embodiment of this application. Figure 2 This is a schematic block diagram of a speech synthesis-based corpus augmentation system according to an embodiment of this application; Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when an element is referred to as “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein may include wireless connections or wireless coupling. The term “and or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0020] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0021] Reference Figure 1 This invention provides a corpus expansion method based on speech synthesis, including steps S1-S5, specifically: S1. Based on the collected audio and corresponding text of the target speaker, perform preprocessing including data annotation; S2. Extract acoustic features from the preprocessed audio; S3. Based on the pre-trained acoustic model, personalized fine-tuning is performed using labeled data and acoustic features, and personalized acoustic models of multiple speakers are trained simultaneously through multi-threaded parallel computing. S4. Call the trained personalized acoustic model and combine it with a vocoder to synthesize the speech corpus of the target text. S5. Based on the trained personalized acoustic model, the corpus is continuously expanded by changing the text.

[0022] As described in step S1, its main purpose is to provide a high-quality data foundation for subsequent acoustic feature extraction and model training. Specifically, this step includes three core components: speech acquisition, noise reduction and standardization, and audio-text alignment and annotation.

[0023] First, in the voice acquisition phase, it is necessary to acquire at least a first preset number (e.g., more than 20) of audio commands from the target speaker using recording equipment, with each audio file having a corresponding text transcription. This step ensures sufficient samples for subsequent fine-tuning of the personalized acoustic model. For example, if the goal is to build a speech recognition system that includes multiple dialects or speaking styles across different age groups, then sufficient voice samples need to be collected from multiple representative speakers to cover various pronunciation characteristics and prosodic patterns.

[0024] Next, the acquired audio needs to undergo environmental noise cancellation processing and be resampled to a uniform sampling rate and bit depth. This is done to ensure consistent audio quality across all audio files and reduce the impact of background noise and other interference on model training. For example, audio recorded in noisy environments may contain significant background noise; noise reduction algorithms can effectively remove these unnecessary components, thereby improving the performance of subsequent model training. Furthermore, a uniform sampling rate and bit depth help simplify the data processing workflow and avoid problems caused by inconsistent formats.

[0025] Finally, the audio and text are precisely aligned to generate a detailed annotation file. This process involves generating timestamps containing phoneme boundaries and annotating the fundamental frequency trajectory and energy variation characteristics. This annotation information is crucial for accurately capturing the pronunciation details of each phoneme. Taking the phrase "Hello, world" as an example, the alignment operation clearly reveals the start and end times of each word, as well as the changes in the fundamental frequency and intensity of the sound during this period. Such annotation not only improves the usability of the data but also provides a solid foundation for subsequent acoustic feature extraction.

[0026] Specifically, the first step is to acquire the original audio of the target speaker using a mobile device or professional recording equipment. Each audio clip should be accompanied by corresponding text content (such as everyday phrases like "Please turn on the air conditioner"). It is recommended to acquire at least 20 clips to ensure feature coverage. Next, the original audio undergoes standardized preprocessing: 1. Environmental noise is eliminated using spectral subtraction or deep learning methods; 2. The audio is standardized to a 16kHz sampling rate and 16-bit depth using a resampling algorithm; 3. Forced alignment between audio and text is achieved using the Dynamic Time Warping (DTW) algorithm, automatically generating a labeled file containing phoneme boundaries (e.g., splitting "air conditioner" into / k / , / ong / , / t / , / iao / ), timestamps (e.g., 0.25-0.38 seconds), fundamental frequency trajectories (reflecting tone changes), and energy characteristics (volume fluctuations). This step supports two implementation methods: the traditional method uses the Kaldi toolchain for GMM-HMM alignment, while the modern method uses end-to-end neural network alignment tools such as MontrealForcedAligner. For example, when processing accented "open the window" speech, the system can accurately identify phoneme boundaries of unconventional pronunciations and annotate special fundamental frequency variation patterns. The core advantages of this step are: 1. It automates the annotation process, reducing the time required for professional linguists to several hours to minutes; 2. Standardized processing ensures consistency in subsequent feature extraction, such as using a uniform sampling rate to avoid bias in spectral analysis; 3. Fine-grained phoneme-level annotation provides crucial supervisory signals for subsequent personalized acoustic model training, which is particularly important for capturing the unique tonal features of Chinese. In practical applications, this step has successfully processed speech data from special populations such as children and the elderly, demonstrating its robustness.

[0027] As described in step S2, first, for each audio, the first acoustic features including Mel Frequency Cepstral Coefficients (MFCCs), fundamental frequency, and energy feature information are extracted. These features can effectively capture key information in the speech signal, such as timbre, intonation, etc. Based on the phoneme boundary annotation information generated in the preprocessing stage, corresponding local acoustic features are extracted for each phoneme segment. For example, when analyzing the sentence "Hello, world", specific acoustic features can be extracted for the four phonemes "你", "好", "世", "界" respectively. The advantage of doing this is that it can more precisely describe the pronunciation characteristics of different speakers. Finally, combining the first acoustic features and local acoustic features forms a complete acoustic feature set, providing rich input data for subsequent model training.

[0028] Specifically, first, global acoustic features are extracted, including Mel Frequency Cepstral Coefficients (MFCCs, usually taking 13 - 40 dimensions), fundamental frequency (F0) trajectory, and short-time energy features, which are obtained through frame-by-frame windowing (such as a frame length of 25 ms and a frame shift of 10 ms) and Fourier transform. Secondly, based on the phoneme boundary annotation generated in S1, local features are extracted for each phoneme segment (such as Chinese initials / finals): 1. Statistics (mean / variance) of phoneme-level MFCCs; 2. Dynamic change patterns of the fundamental frequency within the phoneme (such as rising / falling slopes); 3) Spectral change features of the phoneme transition segment. For example, when processing "你好" (nihao), the system will extract the nasal resonance feature of the initial "n" and the high-frequency energy concentration characteristic of the final "i" respectively. This step supports two implementation methods: The traditional method uses tools such as Librosa for manual feature engineering, while the end-to-end method directly extracts deep features through self-supervised models such as Wav2Vec. Its advantages are: 1. Multi-scale feature fusion retains both the global characteristics of the speech and local pronunciation details; 2. Phoneme-level feature alignment significantly improves the subsequent acoustic model's ability to model pronunciation personalization, especially for the tonal language characteristics of Chinese (such as the fundamental frequency differences in "妈麻马骂").

[0029] By carefully extracting local acoustic features, the model's learning ability for the subtle pronunciation differences of different speakers can be significantly improved, thereby enhancing the authenticity and naturalness of the synthesized speech.

[0030] In step S3, the aim is to fine-tune the pre-trained acoustic model using existing labeled data and extracted acoustic features to obtain personalized acoustic models for multiple speakers. First, the pre-trained acoustic model is loaded, and a speaker feature adaptation layer is added. Then, the main structural parameters of the pre-trained model are fixed, and supervised training is performed only on the speaker feature adaptation layer. This strategy not only saves computational resources but also effectively prevents overfitting. For example, when dealing with multiple speakers, adjusting the adaptation layer parameters allows the model to better adapt to the unique pronunciation habits of each speaker. Training stops when the validation set accuracy reaches a preset threshold, and the personalized acoustic model containing the speaker feature adaptation layer parameters is saved. Furthermore, to accelerate the training process, multi-threaded parallel computing technology is used to train personalized acoustic models for multiple speakers simultaneously.

[0031] Specifically, a pre-trained core acoustic model (such as FastSpeech2 or Tacotron) is first loaded, and a trainable speaker adaptation layer (such as a SpeakerEmbedding layer) is added to its front end. During training, the core model parameters are fixed, and only the adaptation layer is fine-tuned: 1. Features extracted by S2 are used as input; 2. Phoneme-level mean squared error (MSE) and fundamental frequency similarity (PCC) are used as loss functions; 3) Training stops when the validation set accuracy (e.g., MOS ≥ 3.5) is achieved. For multi-threading implementation, a task scheduling service is built using Flask, and multi-threaded workers are started via JavaScript. Each thread manages a computing node (e.g., 4 GPU server nodes), dynamically allocating training tasks for different speakers. For example, when processing children and elderly speakers simultaneously, the system automatically distributes their training data to different nodes for parallel execution. The innovation of this step is as follows: 1. Fine-tuning of the adaptation layer enables each speaker to obtain a high-quality model with only a small amount of data, such as 20 data points (traditional methods require hundreds of data points); 2. Multi-threaded task distribution improves training efficiency by more than 3 times (in actual tests, the training time for 10 speakers in parallel was reduced from 8 hours to 2.5 hours); 3) Modular design supports flexible replacement of the backbone model (such as replacing it with VITS).

[0032] This hierarchical training method can significantly improve the learning effect of individual pronunciation features while maintaining the model's generality, and accelerates the training speed through distributed computing.

[0033] As described in step S4 above, once the personalized acoustic model training is completed, it can be applied to actual speech synthesis tasks. First, the target text is converted into a phoneme sequence and corresponding prosody feature tags are generated. Then, the featureized text is input into the personalized acoustic model, and acoustic parameters including spectral features and fundamental frequency features are output. Next, a neural vocoder is used to convert these acoustic parameters into the original waveform audio. Finally, the generated original waveform audio is smoothed to eliminate possible unnatural fluctuations and ensure that the final synthesized speech is fluent and natural. For example, when we need to synthesize a welcome speech for a specific speaker, we only need to input the corresponding text, and the system can automatically generate speech content that conforms to the accent and style of that speaker.

[0034] Specifically, first convert the target text (such as "The temperature today is 28 degrees") into a phoneme sequence and prosody tags (such as the tone and stress position of "今 / jin1 / 日 / ri4"), then input it into the trained personalized acoustic model, and output frame-level Mel spectrogram and fundamental frequency parameters. These parameters are converted into waveforms by a neural vocoder such as HiFi-GAN, and finally smoothed by dynamic range compression (DRC) for output. For example, when synthesizing this text for an elderly speaker, the model will automatically enhance the low-frequency formants to reflect the age characteristics. This step supports two types of synthesis methods: 1. Traditional cascaded (text → front-end analysis → acoustic model → vocoder); end-to-end (such as directly outputting waveforms using VITS). Its technical advantages are reflected in: 1. The speech synthesized by the personalized acoustic model has a similarity of over 90% with the original speaker (ABX test results); 2. Supports real-time synthesis (single sentence <200ms latency) to meet the requirements of interactive applications.

[0035] This method allows for the rapid and efficient generation of high-quality speech corpora, greatly enhancing the flexibility and efficiency of speech synthesis.

[0036] As described in S5 above, first identify the new text data for which speech needs to be synthesized and perform format standardization on it. Then, call the personalized acoustic model combined with the vocoder to batch-synthesize the speech corpus corresponding to the new text. After each synthesis is completed, add a speaker identifier to the generated speech corpus and generate the associated text content. For example, if you want to add speech materials about product introductions, you only need to prepare the corresponding product description document, and the system can automatically match it with a suitable speech style, thereby continuously enriching the content of the corpus. This method enables the corpus to grow dynamically according to needs without having to re-record or purchase new audio resources.

[0037] Specifically, when new corpus needs to be added (such as adding the dialect word "嗰度"), the system first standardizes the text (converting to pinyin / phonetic notation), and then calls the acoustic model of the corresponding speaker for batch synthesis. For example, to add 10 hours of data to a Cantonese speaker with 100 hours of existing corpus, only the text needs to be prepared and the synthesis task is started (no need for retraining). The extended corpus is automatically associated with the speaker ID and text metadata, and stored in a standardized format (such as FLAC audio + JSON annotation). The core value of this step lies in: 1. The cost of corpus expansion is reduced to 1 / 10 of the traditional recording method; 2. It supports the on-demand generation of corpus in specific fields (such as medical terms), solving the problem of insufficient data coverage in traditional methods.

[0038] By simply changing the text content, a large amount of speech data can be quickly generated, greatly reducing the cost of maintaining and updating the corpus, and at the same time improving the diversity and coverage of the corpus.

[0039] In one embodiment, the step of performing preprocessing including data annotation based on the collected audio of the target speaker and the corresponding text includes: S201. Collect no less than a first preset number of voice command word audios based on a recording device, where each audio has a corresponding text transcription; S202. Perform environmental noise cancellation processing on the collected audio, and resample the audio to a unified sampling rate and bit depth; S203. Align the audio with the text, generate timestamp annotations including phoneme boundaries, and annotate the fundamental frequency trajectory and energy change characteristics, and output an annotation file annotating phoneme boundary, timestamp, fundamental frequency, and energy characteristic information.

[0040] In this embodiment, the preprocessing step ensures data quality through a standardized process. The recording device can be a common device such as a smartphone, and the collected voice command words are common expressions such as "turn on the air conditioner", and each audio synchronously records the corresponding text to avoid transcribing errors. Noise cancellation uses a conventional noise reduction algorithm to process environmental noise, and resampling unifies the audio format to eliminate device differences. The alignment of the audio with the text is achieved through automated speech recognition technology, accurately annotating the start and end times, fundamental frequency changes, and energy intensities of each phoneme. For example, when processing "hello", the system will annotate the time positions and tone characteristics of the four phonemes / n / , / i / , / h / , / ao / . This automated processing greatly improves the annotation efficiency and provides high-quality basic data for subsequent model training.​​​​​​S302. Based on the phoneme boundary annotation information generated in the preprocessing stage, extract the corresponding local acoustic features for each phoneme segment; S303. Based on the first acoustic feature and the local acoustic feature, the acoustic feature is obtained.

[0042] In this embodiment, feature extraction is divided into two levels: global and local. First, general features such as Mel-frequency cepstral coefficients, fundamental frequency, and energy are extracted from the entire audio segment to reflect the overall characteristics of the speech. Then, combined with preprocessed phoneme boundary information, local features are extracted for each phoneme segment, such as the frictional noise characteristics of initials or the formant distribution of finals. For example, when processing "Beijing," the unique features of the four phonemes / b / , / ei / , / j / , and / ing / are extracted separately. This multi-level feature extraction method can more comprehensively capture the detailed characteristics of speech, providing richer input information for personalized acoustic model training.

[0043] In one embodiment, the step of personalizing the pre-trained acoustic model using labeled data and acoustic features includes: S401. Load the pre-trained acoustic model and add a speaker feature adaptation layer to the pre-trained acoustic model. S402. Fix the main structural parameters of the pre-trained model and use labeled data and acoustic features to conduct supervised training on the speaker feature adaptation layer; S403. When the accuracy of the validation set reaches a preset threshold, stop training and save the personalized acoustic model containing the speaker feature adaptation layer parameters.

[0044] This embodiment achieves personalized adaptation through transfer learning. Based on an open-source speech synthesis model, only a few parameters are added to the adaptation layer to learn the acoustic features of a specific speaker. The speaker feature adaptation layer is a 1D convolutional network whose input dimension matches the output dimension of the hidden layer of the pre-trained acoustic model, and whose output dimension is an embedding vector in the prosodic feature space. During training, most parameters of the base model are fixed, and only the adaptation layer is updated. This preserves the model's general capabilities while quickly adapting to new speakers. For example, when processing children's speech, the adaptation layer learns to adjust the parameters of the fundamental frequency prediction module to adapt to higher pitches. This method achieves good personalization results with only a small number of samples, significantly reducing data requirements.

[0045] In one embodiment, the step of simultaneously training personalized acoustic models for multiple speakers through multi-threaded parallel computation includes: S501. Configure multiple computing servers within the local area network and initialize the computing environment of each server; S502. The personalized acoustic model training tasks for different speakers are dynamically allocated to each computing server, and the training tasks are executed in parallel through multi-threading. S503. Training tasks for each computing server are managed through independent subprocesses to avoid request conflicts; S504. After training is complete, collect the personalized acoustic model results from all computing servers.

[0046] This embodiment utilizes parallel computing to accelerate the training process. Training requests are received via a Flask-based web service. JavaScript dynamically creates worker threads, each bound to independent GPU resources, and a heartbeat mechanism monitors task status. For example, when server S003 completes its current task, it automatically retrieves the next speaker's data (e.g., Speaker_005) from the task queue, preventing resource idleness. Multiple computing servers simultaneously process training tasks for different speakers, with each server running an independent training process. A task scheduler dynamically allocates tasks to ensure balanced load across servers. For instance, when training a model for 10 speakers, it can be distributed across 5 servers, each handling 2 speakers. This simultaneous training method significantly improves overall efficiency and shortens the model development cycle.

[0047] In one embodiment, the step of calling the trained personalized acoustic model and combining it with a vocoder to synthesize the speech corpus of the target text includes: S601. Convert the target text into a phoneme sequence and generate corresponding prosodic feature tags; S602. Input the featured text into the personalized acoustic model and output acoustic parameters containing spectral features and fundamental frequency features; S603. Input the generated acoustic parameters into the neural vocoder to generate the original waveform audio; S604. Smooth the original waveform audio to obtain the speech corpus of the target text.

[0048] This embodiment achieves high-quality speech synthesis. First, the input text is converted into a phoneme sequence and its prosodic features are annotated. Then, a personalized acoustic model predicts the corresponding spectral and fundamental frequency parameters. Finally, a vocoder converts these parameters into a natural and fluent speech waveform. For example, when synthesizing "Welcome," the system generates speech that matches the timbre and intonation of the target speaker based on their acoustic characteristics. The entire process is highly automated and supports the generation of large volumes of speech data.

[0049] In one embodiment, the step of continuously expanding the corpus by changing text based on a trained personalized acoustic model includes: S701. Recognize new text data that needs to be synthesized into speech, and perform format standardization processing on the new text data; S702. Call the trained personalized acoustic model and combine it with the vocoder to synthesize the speech corpus corresponding to the new text in batches; S703. Add speaker identifiers to the synthesized speech corpus, generate and associate the corresponding text content, and complete the current expansion operation of the corpus.

[0050] This embodiment supports flexible expansion of the corpus. When new corpus is needed, simply prepare the new text content, and the system will automatically call existing speaker models for batch synthesis. For example, to add 1,000 response sentences to the customer service system, simply input the text to quickly generate the corresponding speech. The synthesized corpus will automatically associate speaker information with the original text, facilitating subsequent management and use. This method greatly simplifies the corpus maintenance and update process.

[0051] Reference Figure 2 This is a block diagram of a speech synthesis-based corpus augmentation system according to an embodiment of this application. The system includes: The preprocessing module 100 is used to perform preprocessing, including data annotation, based on the collected audio and corresponding text of the target speaker. Feature extraction module 200 is used to extract acoustic features from preprocessed audio; The personalized training module 300 is used to perform personalized fine-tuning based on pre-trained acoustic models using labeled data and acoustic features, and to train personalized acoustic models of multiple speakers simultaneously through multi-threaded parallel computing. The corpus synthesis module 400 is used to call the trained personalized acoustic model and combine it with a vocoder to synthesize the speech corpus of the target text. The text augmentation module 500 is used to continuously expand the corpus by changing the text based on the trained personalized acoustic model.

[0052] In one embodiment, the audio acquisition module 100 includes: An audio acquisition unit is used to acquire no less than a first preset number of audio commands based on a recording device, wherein each audio has a corresponding text transcription. The audio processing unit is used to perform environmental noise cancellation on the acquired audio and resample the audio to a uniform sampling rate and bit depth. The text alignment unit is used to align audio with text, generate timestamp annotations containing phoneme boundaries, and annotate the fundamental frequency trajectory and energy change characteristics, outputting an annotation file with annotations of phoneme boundaries, timestamps, fundamental frequency, and energy characteristic information.

[0053] In one embodiment, the feature extraction module 200 includes: The global feature extraction unit is used to extract the first acoustic features of each audio track, including Mel frequency cepstral coefficients, fundamental frequency, and energy feature information. The local feature extraction unit is used to extract corresponding local acoustic features for each phoneme segment based on the phoneme boundary annotation information generated in the preprocessing stage. The feature fusion unit is used to obtain the acoustic features based on the first acoustic features and the local acoustic features.

[0054] In one embodiment, the personalized training module 300 includes: The model loading unit is used to load a pre-trained acoustic model and add a speaker feature adaptation layer to the pre-trained acoustic model. The fine-tuning training unit is used to fix the main structural parameters of the pre-trained model and to perform supervised training on the speaker feature adaptation layer using labeled data and acoustic features. The model saving unit is used to stop training when the validation set accuracy reaches a preset threshold and save the personalized acoustic model containing speaker feature adaptation layer parameters.

[0055] In one embodiment, the personalized training module 300 further includes: The server configuration unit is used to configure multiple computing servers within a local area network and initialize the computing environment of each server. The task allocation unit is used to dynamically allocate the personalized acoustic model training tasks of different speakers to each computing server and execute the training tasks in parallel through multi-threading. The process management unit is used to manage the training tasks of each computing server through independent sub-processes to avoid request conflicts; The results collection unit is used to collect the personalized acoustic model results from all computing servers after training is complete.

[0056] In one embodiment, the corpus synthesis module 400 includes: The text processing unit is used to convert the target text into a phoneme sequence and generate corresponding prosodic feature tags; The acoustic prediction unit is used to input featured text into a personalized acoustic model and output acoustic parameters containing spectral features and fundamental frequency features. The waveform generation unit is used to input the generated acoustic parameters into the neural vocoder to generate the original waveform audio. The post-processing unit is used to smooth the original waveform audio to obtain the speech corpus of the target text.

[0057] In one embodiment, the text expansion module 500 includes: The text recognition unit is used to recognize new text data that needs to be synthesized into speech and to perform format standardization processing on the new text data; The batch synthesis unit is used to call the trained personalized acoustic model and combine it with the vocoder to synthesize the speech corpus corresponding to the new text in batches. The data management unit is used to add speaker identifiers to the synthesized speech corpus, generate and associate the corresponding text content, and complete the current expansion operation of the corpus.

[0058] Reference Figure 3 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in the speech synthesis-based corpus augmentation method. The network interface allows communication with external terminals via a network connection. Furthermore, the computer device may also include input devices and a display screen. When the aforementioned computer program is executed by a processor, it implements a corpus expansion method based on speech synthesis, including the following steps: preprocessing, including data annotation, based on the acquired audio of the target speaker and the corresponding text; extracting acoustic features from the preprocessed audio; performing personalized fine-tuning based on a pre-trained acoustic model using labeled data and acoustic features, and simultaneously training personalized acoustic models for multiple speakers through multi-threaded parallel computation; calling the trained personalized acoustic model and combining it with a vocoder to synthesize the speech corpus of the target text; and continuously expanding the corpus by changing the text based on the trained personalized acoustic model. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0059] One embodiment of this application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements a corpus expansion method based on speech synthesis, including the following steps: preprocessing, including data annotation, based on the acquired audio of a target speaker and its corresponding text; extracting acoustic features from the preprocessed audio; performing personalized fine-tuning based on a pre-trained acoustic model using labeled data and acoustic features, and simultaneously training personalized acoustic models for multiple speakers through multi-threaded parallel computation; calling the trained personalized acoustic model and combining it with a vocoder to synthesize a speech corpus of the target text; and continuously expanding the corpus by changing the text based on the trained personalized acoustic model. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0060] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0061] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0062] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A corpus augmentation method based on speech synthesis, characterized in that, include: Based on the collected audio and corresponding text of the target speaker, preprocessing including data annotation is performed. Extract acoustic features from the preprocessed audio; Based on the pre-trained acoustic model, personalized fine-tuning is performed using labeled data and acoustic features, and personalized acoustic models of multiple speakers are trained simultaneously through multi-threaded parallel computing. The trained personalized acoustic model is invoked and combined with a vocoder to synthesize the speech corpus of the target text. Based on a trained personalized acoustic model, the corpus is continuously expanded by changing the text.

2. The corpus expansion method based on speech synthesis according to claim 1, characterized in that, The preprocessing steps, including data annotation, based on the collected audio and corresponding text of the target speaker include: The recording device collects no less than a first preset number of voice command words audio, where each audio has a corresponding text transcription; The acquired audio is processed to remove environmental noise and then resampled to a uniform sampling rate and bit depth. Align the audio with the text, generate timestamp annotations containing phoneme boundaries, and annotate the fundamental frequency trajectory and energy change characteristics. Output an annotation file containing phoneme boundaries, timestamps, fundamental frequency, and energy characteristic information.

3. The corpus expansion method based on speech synthesis according to claim 1, characterized in that, The step of extracting acoustic features from the preprocessed audio includes: Extract the first acoustic features of each audio track, including Mel frequency cepstral coefficients, fundamental frequency, and energy characteristic information; Based on the phoneme boundary annotation information generated in the preprocessing stage, corresponding local acoustic features are extracted for each phoneme segment. The acoustic features are obtained based on the first acoustic feature and the local acoustic features.

4. The corpus expansion method based on speech synthesis according to claim 1, characterized in that, The steps for personalized fine-tuning of the pre-trained acoustic model using labeled data and acoustic features include: Load the pre-trained acoustic model and add a speaker feature adaptation layer to the pre-trained acoustic model; The main structural parameters of the pre-trained model are fixed, and the speaker feature adaptation layer is trained in a supervised manner using labeled data and acoustic features; Training stops when the validation set accuracy reaches a preset threshold, and the personalized acoustic model containing speaker feature adaptation layer parameters is saved.

5. The corpus augmentation method based on speech synthesis according to claim 1, characterized in that, The step of simultaneously training personalized acoustic models for multiple speakers through multi-threaded parallel computation includes: Configure multiple computing servers within the local area network and initialize the computing environment of each server. Use Flask to build a network service to connect the computing servers and use JavaScript to implement multi-threaded task allocation and communication. The training tasks of personalized acoustic models for different speakers are dynamically allocated to each computing server, and the training tasks are executed in parallel through multi-threading. Training tasks for each computing server are managed by independent subprocesses to avoid request conflicts; After training is complete, collect the results of the personalized acoustic models from all computing servers.

6. The corpus expansion method based on speech synthesis according to claim 1, characterized in that, The step of calling the trained personalized acoustic model and combining it with a vocoder to synthesize the speech corpus of the target text includes: The target text is converted into a phoneme sequence, and corresponding prosodic feature tags are generated. Input the featured text into the personalized acoustic model and output acoustic parameters containing spectral and fundamental frequency features; The generated acoustic parameters are input into a neural vocoder to generate the original waveform audio. The original waveform audio is smoothed to obtain the speech corpus of the target text.

7. The corpus expansion method based on speech synthesis according to claim 1, characterized in that, The step of continuously expanding the corpus by changing text based on the trained personalized acoustic model includes: Identify new text data that needs to be synthesized into speech, and standardize the format of the new text data; The trained personalized acoustic model is invoked and combined with a vocoder to synthesize speech corpora corresponding to new texts in batches. Add speaker identifiers to the synthesized speech corpus, generate and associate the corresponding text content, and complete the current expansion operation of the corpus.

8. A corpus augmentation system based on speech synthesis, characterized in that, include: The preprocessing module is used to perform preprocessing, including data annotation, based on the collected audio of the target speaker and the corresponding text. The feature extraction module is used to extract acoustic features from the preprocessed audio. The personalized training module is used to fine-tune the pre-trained acoustic model using labeled data and acoustic features, and train personalized acoustic models for multiple speakers simultaneously through multi-threaded parallel computing. The corpus synthesis module is used to call the trained personalized acoustic model and combine it with a vocoder to synthesize the speech corpus of the target text. The text augmentation module is used to continuously expand the corpus by changing the text, based on the trained personalized acoustic model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images