Voice data set generation method and device

Abstract

The invention discloses a voice data set generation method and device. The method comprises the following steps: acquiring a voice data set of a standard general language, and converting the voice data set of the standard general language into a target language text by adopting a large language model; a target language statement text is generated in a retrieval enhancement generation mode; target language voice is generated according to the target language text and the target language statement text, a target voice data set is constructed according to the target language voice, and voice features of the target language voice are consistent with those of the standard universal language voice data set. According to the invention, the problem that the accuracy of translating the target language by the translation model is low due to the fact that the voice data volume of the target language voice database is small in the related technology is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for generating a speech dataset. Background Technology

[0002] Currently, many existing multilingual speech datasets focus on resource-rich languages ​​within the Indo-European (e.g., English, German) and East Asian (e.g., Chinese, Japanese) language families. Low-resource languages, such as domestic dialects, have received little attention. Furthermore, these datasets often lack corresponding translation annotations, making them difficult to use for training speech translation models. Recent research has shown that large language models excel at handling general instructions and have demonstrated potential in text generation tasks. Current research aims to combine large language models with instructions and examples from the original text training data, enabling them to generate novel and diverse samples. Currently, much research focuses on using linguistic theory and machine translation techniques for synthetic data augmentation of mixed-language texts. However, for low-resource languages, the effectiveness of this approach remains limited, often requiring retrieval-enhanced generation methods with additional knowledge bases to improve performance. Summary of the Invention

[0003] This application provides a method and apparatus for generating a speech dataset, which at least solves the technical problem in the related art that the translation model has low accuracy in translating the target language due to the small amount of speech data in the target language speech database.

[0004] According to one aspect of the embodiments of this application, a method for generating a speech dataset is provided, comprising: acquiring a speech dataset of a standard general language, and converting the speech dataset of the standard general language into text in a target language using a large language model; generating text in the target language using a retrieval-enhanced generation method; generating speech in the target language based on the text in the target language and the text in the target language, and constructing a target speech dataset based on the speech in the target language, wherein the speech features of the target language speech are consistent with the speech features of the speech dataset of the standard general language.

[0005] Optionally, generating target language sentence text based on search terms includes: converting vocabulary in a preset target language database into dense vectors, wherein the preset target language database contains multiple target language vocabulary words; obtaining multiple sentence topics, and generating query text based on each sentence topic to obtain multiple query texts; converting the multiple query texts into multiple query vectors; comparing each query vector with the dense vectors in the preset target language database, filtering out multiple dense vectors related to each query vector, and obtaining multiple search terms corresponding to the multiple dense vectors related to each query vector; and generating the target language sentence text based on the multiple search terms corresponding to each query vector.

[0006] Optionally, generating the target language sentence text based on multiple search terms corresponding to each query vector includes: generating target prompt words based on multiple target language words corresponding to each query vector and the query text corresponding to each query vector; and analyzing the target prompt words using a large language model to generate the target language sentence text.

[0007] Optionally, generating target language speech based on the target language text and the target language sentence text includes: normalizing the target language text and the target language sentence text to obtain processed text; annotating the processed text with pronunciation rules to obtain annotated text; identifying speech segments related to the annotated text from the speech dataset of the standard general language as reference audio; extracting the speech features from the reference audio, wherein the speech features include at least timbre and intonation; and analyzing the speech features and the annotated text using a speech generation model to generate the target language speech.

[0008] Optionally, a speech generation model is used to analyze the speech features and the annotated text to generate the target language speech, including: using the speech generation model to analyze the speech features and the annotated text to generate preliminary speech; eliminating silent segments in the preliminary speech whose duration exceeds a preset threshold, and adjusting the volume of all time periods in the preliminary speech to be consistent to obtain adjusted speech; using a speech evaluation model to detect the adjusted speech, and if the detection passes, determining that the adjusted speech is the target language speech.

[0009] Optionally, the adjusted speech is detected using a speech evaluation model, including: detecting the adjusted speech using the speech evaluation model to obtain a naturalness score and a fluency score for the adjusted speech; if the naturalness score is greater than a first threshold and the fluency score is greater than a second threshold, converting the adjusted speech into text to be verified; comparing the text to be verified with the processed text to obtain a character error rate; and if the character error rate is lower than a preset error rate threshold, determining that the adjusted speech has passed detection.

[0010] Optionally, the method further includes: if the adjusted speech detection fails, using a backup speech generation model to generate backup speech based on the speech features and the annotated text; using a speech evaluation model to detect the backup speech, and if the detection passes, determining that the backup speech is the target language speech.

[0011] According to another aspect of the embodiments of this application, a speech dataset generation apparatus is also provided, comprising: an acquisition module, configured to acquire a speech dataset of a standard general language and convert the speech dataset of the standard general language into target language text using a large language model; a generation module, configured to generate target language sentence text using a retrieval-enhanced generation method; and a construction module, configured to generate target language speech based on the target language text and the target language sentence text, and construct a target speech dataset based on the target language speech, wherein the speech features of the target language speech are consistent with the speech features of the standard general language speech dataset.

[0012] According to another aspect of the embodiments of this application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described method for generating a voice dataset.

[0013] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described method for generating a speech dataset.

[0014] In this embodiment, a standard general language speech dataset is obtained, and a large language model is used to convert the standard general language speech dataset into target language text. Target language sentence text is generated using a retrieval-enhanced generation method. Target language speech is generated based on the target language text and the target language sentence text, and a target language speech dataset is constructed based on the target language speech. The speech features of the target language speech are consistent with those of the standard general language speech dataset. The target language speech dataset is constructed by combining the directly converted target language text and the target language text generated using the retrieval-enhanced generation method, thereby expanding the target language speech dataset and increasing the amount of target language speech data. This solves the technical problem in related technologies where the accuracy of translation models in translating target languages ​​is low due to the small amount of speech data in the target language speech database. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a method for generating a voice dataset according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a method for generating a speech dataset according to an embodiment of this application;

[0018] Figure 3 This is a flowchart of another method for generating a speech dataset according to an embodiment of this application;

[0019] Figure 4 This is a flowchart of a method for generating target language sentences based on a search-enhanced generation method according to an embodiment of this application;

[0020] Figure 5 This is a flowchart of a target language speech generation method according to an embodiment of this application;

[0021] Figure 6 This is a structural diagram of a speech dataset generation apparatus according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0025] To address the problems existing in related technologies, embodiments of this application provide a method for generating a speech dataset, which can be run on... Figure 1 The computer terminal shown is explained below.

[0026] The speech dataset generation method provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a method for generating speech datasets is shown. Figure 1As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the voice dataset generation method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned voice dataset generation method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0030] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0031] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware components (including circuitry), software components (including computer code stored on a computer-readable medium), or a combination of both hardware and software components. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0032] In the above operating environment, this application provides an embodiment of a method for generating a voice dataset. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] Figure 2 This is a flowchart of a method for generating a speech dataset according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0034] Step S202: Obtain the speech dataset of the standard general language and use a large language model to convert the speech dataset of the standard general language into text in the target language.

[0035] In step S202, the original text of the standard general language speech dataset is extracted from the standard general language speech dataset and used as the input source for the large language model. A pre-trained LLM (Large Language Model) is then used for direct translation from the standard general language to the target language, such as translating Mandarin into Cantonese. During the translation process, specific prompts can be designed to emphasize the use of colloquial Cantonese expressions and localized vocabulary, avoiding mechanical character conversion. For example, "Translate the following Mandarin (standard general language) text into colloquial Cantonese (target language), using common local expressions, avoiding direct word-for-word translation." Finally, the translation results are initially screened, removing sentences that clearly do not conform to Cantonese expression habits. The LLM is a very large-scale deep learning model pre-trained on a large amount of data. The underlying converter is a set of neural networks consisting of encoders and decoders with self-attention capabilities. The encoders and decoders extract meaning from a series of texts and understand the relationships between words and phrases within them.

[0036] Step S204: The target language sentence text is generated using RAG (Retrieval-Augmented Generation). RAG refers to optimizing the output of a large language model so that it references authoritative knowledge bases beyond the training data source before generating the final response. LLMs are trained on massive amounts of data, containing billions of parameters, and can generate raw outputs for tasks such as question answering, language translation, and content generation. Building upon the already powerful capabilities of LLMs, RAGs can leverage domain-specific or organizational internal knowledge bases without retraining the model. This is a cost-effective way to improve LLM outputs, ensuring relevance, accuracy, and usability across various contexts.

[0037] It should be noted that the target language text generated in step S202 was not enhanced by retrieval and was directly converted by the large language model, while the target language sentence text generated in step S204 was generated by retrieval enhancement.

[0038] In step S204, the preset target language database can be constructed in the following way: multiple target language words are obtained from multiple data sources, and the obtained multiple target language words are combined with words in the well-known target language standard general language dictionary to form the preset target language database.

[0039] Step S206: Generate target language speech based on the target language text and the target language sentence text, and construct a target language speech dataset based on the target language speech, wherein the speech features of the target language speech are consistent with the speech features of the standard general language speech dataset.

[0040] It should also be noted that the target language includes multiple languages, such as Cantonese and other languages.

[0041] Through steps S202 to S206 above, a standard general-purpose language speech dataset is obtained, and a large language model is used to convert the standard general-purpose language speech dataset into target language text; a retrieval-enhanced generation method is used to generate target language sentence text; target language speech is generated based on the target language text and the target language sentence text, and a target language speech dataset is constructed based on the target language speech. The speech features of the target language speech are consistent with those of the standard general-purpose language speech dataset. The target language speech dataset is constructed by combining the directly converted target language text and the target language text generated based on the retrieval-enhanced generation method, thereby expanding the target language speech dataset and increasing the amount of target language speech data. This solves the technical problem in related technologies where the accuracy of translation models in translating target languages ​​is low due to the small amount of speech data in the target language speech database. The following is a detailed explanation.

[0042] In some embodiments of this application, generating target language sentence text based on search terms includes: converting vocabulary in a preset target language database into dense vectors, wherein the preset target language database contains multiple target language vocabulary; obtaining multiple sentence topics, and generating query text based on each sentence topic to obtain multiple query texts; converting the multiple query texts into multiple query vectors; comparing each query vector with the dense vectors in the preset target language database, filtering out multiple dense vectors related to each query vector, and obtaining multiple search terms corresponding to the multiple dense vectors related to each query vector; and generating the target language sentence text based on the multiple search terms corresponding to each query vector.

[0043] Specifically, a natural speech processing model can be used to convert words in a pre-defined target language database into dense vectors for semantic retrieval.

[0044] Query vectors are generated based on specific scenarios corresponding to the topic of the statement, such as casual conversation at dinner, and sentence structures, such as interrogative and declarative sentences. The most relevant Cantonese words and phrases are retrieved from the knowledge base and sorted by relevance and diversity.

[0045] The specific steps for generating the target language sentence text based on multiple search terms corresponding to each query vector include: generating target prompt words based on multiple target language words corresponding to each query vector and the query text corresponding to each query vector; and analyzing the target prompt words using a large language model to generate the target language sentence text.

[0046] Specifically, the search results are combined with the original query to generate enhanced prompt words (target prompt words), which are then input into the large language model. An example prompt word is: "Based on the content in the following preset database (search results: {search term}), generate a sentence in the target language that conforms to everyday conversational style." Finally, automatic generation is achieved: the large language model generates diverse text that conforms to the expression habits of the target language based on the enhanced prompt words. The generated results are post-processed to ensure that the sentences are fluent and conform to the grammar of the target language.

[0047] The above method extracts the original text of Standard General Language (SGML) from the SGML speech dataset, uses a pre-trained large language model for direct translation from SGML to the target language, and designs specific prompt words to emphasize spoken expressions and localized vocabulary in the target language, avoiding mechanical character conversion and effectively improving the accuracy of the target language text translation. This makes the generated target language text more in line with the actual usage habits of Cantonese. In the retrieval enhancement generation stage, a target language vocabulary knowledge base is constructed based on the target language to SGML dictionary and supplementary vocabulary. Semantic retrieval is performed in conjunction with the model, and query vectors are generated according to specific scenarios and sentence structures. Relevant target language words and phrases are retrieved from the knowledge base and sorted. These are combined with the original query to generate enhanced prompt words, which are then input into the large language model. Finally, diverse texts that conform to the expression habits of the target language are generated, enriching the expressive forms of the target language text. In the final stage of text generation, the generated results are post-processed to ensure that the sentences are fluent and conform to the grammar of the target language, further improving the quality and readability of the target language text. The entire text generation process involves multi-stage processing and optimization to make the generated target language text more closely resemble real-world application scenarios, such as different scenarios in daily conversations (e.g., "dinner chat") and sentence structures (e.g., interrogative sentences and declarative sentences), thereby improving the text's practicality and applicability in actual communication.

[0048] In some embodiments of this application, the specific steps for generating target language speech based on the target language text and the target language sentence text include: normalizing the target language text and the target language sentence text to obtain processed text; annotating the processed text with pronunciation rules to obtain annotated text; identifying speech segments related to the annotated text from the speech dataset of the standard general language as reference audio; extracting the speech features from the reference audio, wherein the speech features include at least: timbre and intonation; and analyzing the speech features and the annotated text using a speech generation model to generate the target language speech.

[0049] In practical applications, the target language text and target language sentences output during the text generation stage are standardized, including text normalization (such as digit transcription and punctuation processing) and target language-specific pronunciation annotation (such as entering tone characters and tone sandhi rules). Speech segments from a standard general-purpose language speech dataset are used as reference audio, and speaker speech features (such as timbre and intonation) are extracted through voiceprint encoding. The reference audio features are then compared with the annotated text and input into the speech generation model to ensure that the synthesized speech matches the original speaker features.

[0050] In some embodiments of this application, the specific steps for generating the target language speech by analyzing the speech features and the annotated text using a speech generation model are as follows: The speech features and the annotated text are analyzed using the speech generation model to generate preliminary speech; silent segments in the preliminary speech with a duration exceeding a preset threshold are eliminated, and the volume of all time periods in the preliminary speech is adjusted to be consistent to obtain adjusted speech; the adjusted speech is detected using a speech evaluation model, and if the detection passes, the adjusted speech is determined to be the target language speech.

[0051] The speech synthesis process involves inputting the annotated text into a speech generation model to generate an initial speech waveform. Basic quality checks are then performed on the synthesized speech, including: silence detection (to avoid abnormal pauses) and volume equalization (to ensure consistent volume).

[0052] In some embodiments of this application, a speech evaluation model is used to detect the adjusted speech, including: using the speech evaluation model to detect the adjusted speech and obtaining a naturalness score and a fluency score for the adjusted speech; if the naturalness score is greater than a first threshold and the fluency score is greater than a second threshold, converting the adjusted speech into text to be verified; comparing the text to be verified with the processed text to obtain a character error rate; and if the character error rate is lower than a preset error rate threshold, determining that the adjusted speech has passed detection.

[0053] For example, the DNSMOS model (a speech evaluation model) (threshold > 3.0) is used to evaluate the naturalness and fluency of the speech; the character error rate (CER) between the synthesized speech and the original text is calculated using a speech recognition model, with a threshold set to < 15%. Then, manual verification is performed, with random sampling and listening tests on the selected speech data, focusing on checking the accuracy of Cantonese pronunciation (e.g., avoiding "lazy sounds") and the naturalness of emotional expression (e.g., intonation of interrogative sentences).

[0054] The above methods accurately assess speech quality: Quantitative evaluation of speech naturalness and fluency, using a threshold >3.0 as the standard, precisely filters out speech data with high naturalness and fluency, effectively distinguishing between good and bad speech synthesis quality. This ensures consistency between speech and text: Using a speech recognition model to calculate the character error rate (CER) between the synthesized speech and the original text, setting a threshold <15%, accurately measures the degree of matching between speech and text, ensuring that the generated speech accurately conveys the original information and reducing speech recognition bias. It also improves speech accuracy and emotional expression: Random sampling and listening tests are conducted on the filtered speech data through manual verification, focusing on the accuracy of Cantonese pronunciation to avoid errors such as "lazy pronunciation" and ensure that the speech conforms to Cantonese pronunciation norms; simultaneously, the naturalness of emotional expression is checked, such as whether the intonation of interrogative sentences is appropriate, making the generated speech more realistic and engaging, improving the comprehensibility and acceptability of the speech in actual communication. Construct a multi-level evaluation system: Combine automatic evaluation models with manual verification to form a multi-level, comprehensive speech quality evaluation system. This will give full play to the advantages of the efficiency of automatic evaluation and the meticulousness of manual verification, improve the overall quality of speech data, and provide a reliable basis for the optimization and improvement of subsequent speech synthesis.

[0055] It's important to note that the naturalness score primarily assesses whether synthesized speech sounds similar to human speech, including timbre, intonation, pronunciation accuracy, and emotional delivery. Evaluation can be conducted from several aspects, such as: Timbre: Evaluating the quality of the audio signal, checking its clarity and subtlety to resemble natural speech. Intonation and Rhythm: Examining the prosodic characteristics of speech, including pitch variation, speech rate changes, and the natural distribution of pauses. Pronunciation Authenticity: Assessing the accuracy and fluency of pronunciation. Emotional Delivery: Measuring whether the speech appropriately conveys emotion and tone.

[0056] Fluency assessment primarily evaluates the continuity and fluency of speech, mainly examining the smoothness of transitions between pronunciations and the natural flow of the entire sentence or paragraph. For example: Pronunciation transitions: checking the naturalness of connections between syllables and words, avoiding the discontinuities or abruptness common in synthesized speech. Sentence coherence: assessing the smoothness of connections between sentences within a passage, whether it forms a logically complete and pleasant-sounding dialogue or narrative. Pauses and breathing: considering the appropriateness of pauses and breathing points, which directly relates to the natural fluency and auditory experience of the speech.

[0057] If the adjusted speech detection fails, a backup speech generation model is used to generate backup speech based on the speech features and the annotated text; a speech evaluation model is used to detect the backup speech, and if the detection passes, the backup speech is determined to be the target language speech.

[0058] If the speech quality output by Cosyvoice2 (a speech generation model) is substandard (e.g., mechanical sounds or grammatical errors), the system switches to the GPT-SovITS model (a speech synthesis model that integrates a generative pre-trained transformer with a speech cloning and conversion system, serving as a backup speech generation model) for re-synthesis. Finally, the optimal synthesis result is selected through a multi-model voting mechanism.

[0059] The speech dataset generation method provided in this application generates natural, fluent, and near-human-like speech data in the target language based on the target language text output during the text generation stage, using advanced text-to-speech technology. This improves the quality and usability of target language speech synthesis. Maintaining speaker feature consistency: By using speech segments from a standard general language speech dataset as reference audio, speaker features (such as timbre and intonation) are extracted and input into the speech generation model along with the annotated text (target language). This ensures that the synthesized target language speech maintains consistency with the original speaker's speech features, enhancing the personalization and realism of the speech synthesis. The main synthesizer uses the Cosyvoice2 model, which boasts excellent naturalness and timbre fidelity, while also equipped with a backup synthesizer, the GPT-SovITS model. This ensures the stability of the synthesis process in long sentences or complex intonation scenarios, reducing the risk of speech synthesis failure or anomalies and improving the reliability of speech synthesis. The target language text output during the text generation stage is standardized, including text normalization and Cantonese-specific pronunciation annotation. This makes the text input to the speech generation model more standardized and accurate, helping to generate speech that better conforms to Cantonese pronunciation rules and language habits, further improving the accuracy and naturalness of speech synthesis. A multi-model collaboration mechanism is designed. First, the main synthesizer model generates an initial speech waveform and performs basic quality checks. If the quality is not up to standard, it switches to a backup model for resynthesis. Finally, a multi-model voting mechanism selects the optimal synthesis result, effectively ensuring the high quality of the final output target language speech data and providing users with accurate and high-quality speech synthesis services.

[0060] Figure 3 Another method for generating speech datasets is shown, such as... Figure 3 As shown, the process includes: taking Mandarin and Cantonese as examples, using a large language model to directly translate Mandarin text into Cantonese text, and generating Cantonese text based on retrieval enhancement. Finally, the Cantonese text generated by the two methods is synthesized into speech, and the synthesized speech is tested for performance metrics. After passing the tests, a speech dataset is constructed.

[0061] Figure 4 This paper demonstrates a method for generating target language sentences based on retrieval-enhanced generation, such as... Figure 4 As shown, a query vector (question) is generated based on the statement topic, relevant documents are retrieved, and the retrieved documents are combined with the question to generate a statement in the target language.

[0062] Figure 5 A flowchart of a speech generation method is shown, such as Figure 5As shown, the process includes: generating target language speech based on annotated text and reference speech through a main synthesizer; evaluating the generated target language speech using a speech evaluation model; directly outputting the speech if the evaluation result is satisfactory; and generating new target language speech using a backup speech generation model if the evaluation result is unsatisfactory.

[0063] The speech dataset is generated based on the method for generating speech datasets proposed in the embodiments of this application. The embodiments of this application also provide comparative experimental results with other publicly available datasets:

[0064] Experimental Data Groups: Control Group: CVHK: Using a publicly available dataset as a baseline; DTDT: A Cantonese dataset generated solely through direct translation. Experimental Group: RFDT: A Cantonese dataset generated using retrieval-enhanced translation; AUDT: A Cantonese dataset constructed using an automatic generation method. Test Set Division: ASR Test Set: AUDT_test (Test Set 1): 500 sentences randomly selected from AUDT; CVHK_test (Test Set 2): 500 sentences randomly selected from CVHK. S2TT (Speech-to-Text Translation) Test Set: 800 samples from the MDT-ASR-D001 dataset. Model Training: The baseline model selected was an open-source speech recognition model, maintaining the original model's multilingual recognition and translation capabilities. Fine-tuning Settings: Training cycle of 5 epochs, batch size of 16, and learning rate of 5e-6. Comparative experimental design: Experiment 1: Fine-tuning the model using DTDT, RFDT, and AUDT datasets respectively; Experiment 2: Fine-tuning the model using a combination of RFDT and AUDT datasets; Experiment 3: Performance comparison under different dataset sizes (20h / 50h / 80h). Evaluation metrics: Character error rate was used for speech recognition tasks, while various evaluation metrics were used for speech translation tasks, such as BLEU-1 / BLEU-2, SacreBLEU, and BERTScore.

[0065] Experimental results show that the AUDT dataset generated using the RAG method reduces the CER to 18.7% in the ASR task, which is close to the effect of real data; the hybrid dataset (RFDT+AUDT) achieves a BERTS score of 0.77 in the translation task, which is significantly better than the single dataset; the data volume experiment shows that the performance improves with the increase of data volume and tends to stabilize at 80h of data volume.

[0066] Table 1 shows the performance comparison of ASR (CER / %). As shown in Table 1, the CER of the AUDT dataset in the ASR task is reduced to 18.7%, which is close to the effect of real data.

[0067] Table 1

[0068]

[0069] Table 2 shows the performance comparison of S2TT. As shown in Table 2, the hybrid dataset (RFDT+AUDT) achieved a BERTS score of 0.77 in the translation task, which is significantly better than the single dataset. It can be understood that the hybrid dataset is the dataset generated by the method in this application.

[0070] Table 2

[0071]

[0072] Figure 6 An apparatus for generating a speech dataset is shown, the apparatus comprising:

[0073] The acquisition module 60 is used to acquire a speech dataset of a standard general language and use a large language model to convert the speech dataset of the standard general language into text in the target language.

[0074] Generation module 62 is used to generate target language sentence text using a search-enhanced generation method;

[0075] The construction module 64 is used to generate target language speech based on the target language text and the target language sentence text, and to construct a target language speech dataset based on the target language speech, wherein the speech features of the target language speech are consistent with the speech features of the standard general language speech dataset.

[0076] The aforementioned speech dataset generation device acquires a speech dataset of a standard general language and uses a large language model to convert the speech dataset of the standard general language into target language text; it then generates target language sentence text using a retrieval-enhanced generation method; based on the target language text and the target language sentence text, it generates target language speech and constructs a target language speech dataset based on the target language speech. The speech features of the target language speech are consistent with those of the standard general language speech dataset. The target language speech dataset is constructed by combining the directly converted target language text and the target language text generated using the retrieval-enhanced generation method, thereby expanding the target language speech dataset and increasing the amount of target language speech data. This solves the technical problem in related technologies where the accuracy of translation models in translating target languages ​​is low due to the small amount of speech data in the target language speech database.

[0077] The generation module 62 includes a generation submodule, used to generate target language sentence text using a retrieval-enhanced generation method, including: converting vocabulary in a preset target language database into dense vectors, wherein the preset target language database contains multiple target language vocabulary; obtaining multiple sentence topics, and generating query text based on each sentence topic to obtain multiple query texts; converting the multiple query texts into multiple query vectors; comparing each query vector with the dense vectors in the preset target language database, filtering out multiple dense vectors related to each query vector, and obtaining multiple search terms corresponding to the multiple dense vectors related to each query vector; and generating the target language sentence text based on the multiple search terms corresponding to each query vector.

[0078] The generation submodule includes a generation unit, which is used to generate the target language sentence text based on multiple search terms corresponding to each query vector, including: generating target prompt words based on multiple target language words corresponding to each query vector and the query text corresponding to each query vector; and analyzing the target prompt words using a large language model to generate the target language sentence text.

[0079] The construction module 64 includes a speech submodule, used to generate target language speech based on the target language text and the target language sentence text, including: normalizing the target language text and the target language sentence text to obtain processed text; annotating the processed text with pronunciation rules to obtain annotated text; finding speech segments related to the annotated text from the speech dataset of the standard general language as reference audio; extracting the speech features from the reference audio, wherein the speech features include at least: timbre and intonation; and analyzing the speech features and the annotated text using a speech generation model to generate the target language speech.

[0080] The speech submodule includes a speech unit, used to analyze the speech features and the annotated text using a speech generation model to generate the target language speech, including: analyzing the speech features and the annotated text using the speech generation model to generate preliminary speech; eliminating silent segments in the preliminary speech whose duration exceeds a preset threshold, and adjusting the volume of all time periods in the preliminary speech to be consistent to obtain adjusted speech; and using a speech evaluation model to detect the adjusted speech, and if the detection passes, determining that the adjusted speech is the target language speech.

[0081] The speech unit includes a detection subunit, used to detect the adjusted speech using a speech evaluation model, including: detecting the adjusted speech using the speech evaluation model to obtain a naturalness score and a fluency score for the adjusted speech; if the naturalness score is greater than a first threshold and the fluency score is greater than a second threshold, converting the adjusted speech into text to be verified; comparing the text to be verified with the processed text to obtain a character error rate; and if the character error rate is lower than a preset error rate threshold, determining that the adjusted speech has passed detection.

[0082] The speech unit further includes: a backup subunit, used to generate backup speech based on the speech features and the annotated text using a backup speech generation model when the adjusted speech detection fails; and to detect the backup speech using a speech evaluation model, and determine the backup speech as the target language speech when the detection passes.

[0083] It should be noted that, Figure 6 The device for generating the speech dataset shown is used to perform... Figure 2 The method for generating the speech dataset shown above also applies to the device for generating the speech dataset, and will not be repeated here.

[0084] This application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described method for generating the speech dataset.

[0085] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the speech dataset generation method in this application.

[0086] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0087] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0092] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for generating a speech dataset, characterized in that, include: Obtain a speech dataset of a standard general language and use a large language model to convert the speech dataset of the standard general language into text in the target language; The target language sentence text is generated using a search-enhanced generation method. The target language speech is generated based on the target language text and the target language sentence text, and a target language speech dataset is constructed based on the target language speech, wherein the speech features of the target language speech are consistent with the speech features of the standard general language speech dataset.

2. The method according to claim 1, characterized in that, The target language sentence text is generated using a search-enhanced generation method, including: The vocabulary in the preset target language database is converted into dense vectors, wherein the preset target language database contains multiple target language vocabulary; Obtain multiple statement topics, and generate query text based on each statement topic to obtain multiple query texts; Convert the multiple query texts into multiple query vectors; Each query vector is compared with the dense vectors in the preset target language database to filter out multiple dense vectors related to each query vector, and multiple search terms corresponding to the multiple dense vectors related to each query vector are obtained. The target language sentence text is generated based on the multiple search terms corresponding to each query vector.

3. The method according to claim 2, characterized in that, Generate the target language sentence text based on multiple search terms corresponding to each query vector, including: Target prompt words are generated based on the multiple target language words corresponding to each query vector and the query text corresponding to each query vector; The target prompt words are analyzed using a large language model to generate the target language sentence text.

4. The method according to claim 1, characterized in that, Generating target language speech based on the target language text and the target language sentence text includes: The target language text and the target language sentence text are normalized to obtain the processed text. The pronunciation rules are annotated in the processed text to obtain the annotated text; Identify the speech segments related to the annotated text from the speech dataset of the standard general language, and use them as reference audio; The speech features are extracted from the reference audio, wherein the speech features include at least: timbre and intonation; The speech features and the annotated text are analyzed using a speech generation model to generate the speech in the target language.

5. The method according to claim 4, characterized in that, The speech features and the annotated text are analyzed using a speech generation model to generate the target language speech, including: The speech generation model is used to analyze the speech features and the annotated text to generate preliminary speech. Eliminate silent segments in the initial speech that are longer than a preset threshold, and adjust the volume of all time periods in the initial speech to be consistent to obtain the adjusted speech; The adjusted speech is detected using a speech evaluation model. If the detection is successful, the adjusted speech is determined to be the target language speech.

6. The method according to claim 5, characterized in that, The adjusted speech is detected using a speech evaluation model, including: The adjusted speech is detected using a speech evaluation model to obtain a naturalness score and a fluency score for the adjusted speech; If the naturalness score is greater than the first threshold and the fluency score is greater than the second threshold, the adjusted speech is converted into text to be verified. The text to be verified is compared with the processed text to obtain the character error rate; If the character error rate is lower than a preset error rate threshold, the adjusted speech detection is deemed to have passed.

7. The method according to claim 6, characterized in that, The method further includes: If the adjusted speech detection fails, an alternative speech generation model is used to generate alternative speech based on the speech features and the annotated text. The backup speech is detected using a speech evaluation model. If the detection passes, the backup speech is determined to be the target language speech.

8. An apparatus for generating a speech dataset, characterized in that, include: The acquisition module is used to acquire a speech dataset of a standard general language and to convert the speech dataset of the standard general language into text in the target language using a large language model. The generation module is used to generate target language sentence text using a search-enhanced generation method. A construction module is used to generate target language speech based on the target language text and the target language sentence text, and to construct a target language speech dataset based on the target language speech, wherein the speech features of the target language speech are consistent with the speech features of the standard general language speech dataset.

9. A computer device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the method for generating a speech dataset according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method for generating the speech dataset according to any one of claims 1 to 7.

Images