A training method for a dialect voice cloning model, a dialect voice cloning method, a device, a terminal device, and a storage medium

Abstract

The present invention discloses a training method for a dialect speech cloning model, a dialect speech cloning method, an apparatus, a terminal device, and a storage medium. The training method comprises: obtaining a dialect audio sample and corresponding text information; setting parameters of a text encoder, a speech segmenter, a large language sub-model, and a stream matching sub-model to fixed parameters, initializing a first low-rank matrix and a second low-rank matrix of a Lora module; iteratively training the dialect speech cloning model using the dialect audio sample and text information as input and the cloned speech corresponding to the text information as output until a loss function converges; during training, if the loss function does not converge, updating the first low-rank matrix and the second low-rank matrix, and adding the product of the updated first low-rank matrix and the second low-rank matrix as a weight matrix update to the large language sub-model. The implementation of the present invention can improve the realism of dialect cloning and the training efficiency of the model.

Description

Technical Field

[0001] The present invention relates to the field of voice cloning technology, and in particular to a method for training a dialect voice cloning model, a dialect voice cloning method, an apparatus, a terminal device, and a storage medium. Background Art

[0002] In today's information-based society, speech synthesis and cloning technologies have become an integral part of human-computer interaction, widely used in industries such as customer service, entertainment, and education. Voice cloning technology, in particular, replicates the voice characteristics of a specific individual, enabling the generation of personalized speech.

[0003] However, existing voice cloning models are often optimized for universal languages ​​(such as Mandarin). While they excel at cloning standard Mandarin and other common speech, they fail to adequately preserve the specific phonetic features of local dialects, such as tones and accents, resulting in cloned voices lacking authenticity. This is particularly true for Cantonese dialects like Foshanese, whose unique phonetic rhythms, pronunciation patterns, and vocabulary usage make it difficult for traditional voice cloning models to accurately capture and reproduce them. Furthermore, existing voice cloning models have poor adaptability when dealing with different dialects, often requiring retraining with large amounts of dialect data, resulting in low efficiency. Summary of the Invention

[0004] The embodiments of the present invention provide a dialect speech cloning model training method, a dialect speech cloning method, an apparatus, a terminal device, and a storage medium, which can improve the realism of dialect cloning and the training efficiency of the model.

[0005] An embodiment of the present invention provides a method for training a dialect speech cloning model, which is applicable to the dialect speech cloning model. The dialect speech cloning model includes: a text encoder, a speech word segmenter, a large language sub-model, a stream matching sub-model, a Lora module, and a vocoder. The training method includes:

[0006] Obtain dialect audio samples and corresponding text information;

[0007] Set the parameters of the text encoder, speech segmenter, large language sub-model, and stream matching sub-model to corresponding fixed parameters, and initialize the first low-rank matrix and the second low-rank matrix of the Lora module;

[0008] The initial dialect speech cloning model is iteratively trained with dialect audio samples and corresponding text information as input and cloned speech corresponding to the text information as output until the loss function of the dialect speech cloning model converges. During training, the cloned speech output by the vocoder is compared with the dialect audio sample, and the loss function value is calculated according to the comparison result. If the loss function value does not converge, the first low-rank matrix and the second low-rank matrix of the Lora module are updated, and the product of the updated first low-rank matrix and the second low-rank matrix is ​​used as the weight matrix update amount and added to the large language sub-model.

[0009] Furthermore, obtaining the dialect audio sample and the corresponding text information includes:

[0010] Obtaining initial dialect audio, and standardizing the initial dialect audio according to a preset sampling rate and format to obtain standardized dialect audio;

[0011] Performing noise reduction processing on the standardized dialect audio to obtain noise-reduced dialect audio;

[0012] Cut the de-noised dialect audio into segments according to a preset duration to obtain several dialect audio segments;

[0013] Using speaker recognition technology, the voices of different speakers in the dialect audio segment are separated to obtain separated dialect audio segments, so that each separated dialect audio segment corresponds to the voice of a single speaker;

[0014] Removing the silent part from the separated dialect audio segment to obtain the dialect audio sample;

[0015] Perform text transcription on the dialect audio sample to generate corresponding text information.

[0016] Furthermore, the step of normalizing the initial dialect audio according to a preset sampling rate and format to obtain the standardized dialect audio includes:

[0017] The initial dialect audio is converted into a 16000 Hz WAV format to obtain a standardized dialect audio.

[0018] Furthermore, the preset duration is 5 to 10 seconds.

[0019] Based on the above method embodiment, the present invention provides a corresponding device embodiment;

[0020] An embodiment of the present invention provides a training device for a dialect speech cloning model, comprising: a data acquisition module, a parameter setting module, and an iterative training module;

[0021] The data acquisition module is used to acquire dialect audio samples and corresponding text information;

[0022] The parameter setting module is used to set the parameters of the text encoder, speech segmenter, large language sub-model and stream matching sub-model to corresponding fixed parameters, and initialize the first low-rank matrix and the second low-rank matrix of the Lora module;

[0023] The iterative training module is configured to iteratively train an initial dialect voice cloning model using a dialect audio sample and corresponding text information as input and a cloned voice corresponding to the text information as output until the loss function of the dialect voice cloning model converges. During training, the cloned voice output by the vocoder is compared with the dialect audio sample, and a loss function value is calculated based on the comparison result. If the loss function value does not converge, the first low-rank matrix and the second low-rank matrix of the Lora module are updated, and the product of the updated first low-rank matrix and the second low-rank matrix is ​​used as a weight matrix update amount and added to the large language sub-model.

[0024] Furthermore, the data acquisition module includes: a standardization processing unit, a noise reduction unit, an audio cutting unit, an audio separation unit, a silence removal unit, and a text transcription unit;

[0025] The standardization processing unit is used to obtain the initial dialect audio, and perform standardization processing on the initial dialect audio according to a preset sampling rate and format to obtain standardized dialect audio;

[0026] The noise reduction unit is configured to perform noise reduction processing on the standardized dialect audio to obtain noise-reduced dialect audio;

[0027] The audio cutting unit is used to cut the de-noised dialect audio according to a preset duration to obtain a plurality of dialect audio segments;

[0028] The audio separation unit is used to separate the voices of different speakers in the dialect audio segment by using speaker recognition technology to obtain separated dialect audio segments, so that each separated dialect audio segment corresponds to the voice of a single speaker;

[0029] The silence removal unit is used to remove the silence part in the separated dialect audio segment to obtain the dialect audio sample;

[0030] The text transcription unit is used to transcribe the dialect audio sample into text to generate corresponding text information.

[0031] Based on the above method embodiment, the present invention provides a corresponding terminal device embodiment;

[0032] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for training a dialect speech cloning model as described in any one of the embodiments is implemented.

[0033] Based on the above method embodiment, the present invention provides a corresponding storage medium embodiment;

[0034] Another embodiment of the present invention provides a storage medium, which includes a stored computer program. When the computer program is executed, the device where the storage medium is located is controlled to execute a method for training a dialect speech cloning model as described in any one of the above embodiments.

[0035] In addition, another embodiment of the present invention further provides a dialect voice cloning method, comprising:

[0036] Obtain the dialect speech to be cloned and the text to be synthesized;

[0037] The dialect speech to be cloned and the text to be synthesized are input into a dialect speech cloning model trained according to a training method for a dialect speech cloning model, so that the dialect speech cloning model extracts semantic features of the text to be synthesized through a text encoder and extracts timbre features of the dialect speech to be cloned through a speech segmenter. The semantic features and timbre features are input into a large language sub-model, and a speech token sequence corresponding to the text to be synthesized is generated through the large speech sub-model. The token sequence is converted into a mel-spectrogram through a stream matching sub-model, and sound wave synthesis is performed through a vocoder based on the mel-spectrogram to generate a cloned speech with the same timbre as the dialect speech to be cloned and corresponding to the text content of the text to be synthesized.

[0038] The following beneficial effects are achieved by implementing the present invention:

[0039] The present invention discloses a training method, apparatus, terminal device, and storage medium for a dialect speech cloning model. The training method first obtains a dialect audio sample and corresponding text information; then sets the parameters of a text encoder, a speech segmenter, a large language sub-model, and a stream matching sub-model to corresponding fixed parameters, and initializes the first low-rank matrix and the second low-rank matrix of a Lora module; then, iteratively trains the initial dialect speech cloning model using the dialect audio sample and the corresponding text information as input and the cloned speech corresponding to the text information as output until the loss function of the dialect speech cloning model converges; during training, the cloned speech output by the vocoder is compared with the dialect audio sample, and a loss function value is calculated based on the comparison result. If the loss function value does not converge, the first low-rank matrix and the second low-rank matrix of the Lora module are updated, and the product of the updated first low-rank matrix and the second low-rank matrix is ​​used as a weight matrix update and added to the large language sub-model. During the training process, the parameters of the four modules (the text encoder, the speech segmenter, the large speech sub-model, and the stream matching model) are kept unchanged. On this basis, we focus on training the Lora module and optimize its first and second low-rank matrices using training samples. The product of the optimized first and second low-rank matrices is used as the weight matrix update amount, and the weight matrix update amount of the model is formed as the product of the two low-rank matrices. Since the dimensions of the two low-rank matrices are much smaller than the original weight matrix update amount, this strategy significantly reduces the number of parameters required in the training process and the demand for training data. This allows the model to maintain high cloning quality even with limited data resources, thereby improving the training efficiency of the model. In addition, this low-rank matrix decomposition method is not only structurally simple, but also enhances the model's adaptability to specific dialect cloning, effectively capturing and reproducing the unique speech characteristics of the dialect, and improving the regional characteristics and realism of the dialect speech cloning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flow chart of a method for training a dialect speech cloning model provided by one embodiment of the present invention.

[0041] Figure 2 1 is a schematic diagram of the structure of a dialect speech cloning model provided by one embodiment of the present invention.

[0042] Figure 3 The figure is a schematic diagram showing the principle of fine-tuning the Lora module in a training method for a dialect speech cloning model provided by one embodiment of the present invention.

[0043] Figure 4 The figure is a schematic diagram of the structure of a training device for a dialect speech cloning model provided by one embodiment of the present invention.

[0044] Figure 5 The figure is a schematic diagram of the principle of a dialect voice cloning method provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0047] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0048] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0050] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0051] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0052] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for training a dialect speech cloning model, which is applicable to the dialect speech cloning model. The dialect speech cloning model includes: a text encoder, a speech word segmenter, a large language sub-model, a stream matching sub-model, a Lora module, and a vocoder. The training method includes the following steps:

[0053] S1. Obtain dialect audio samples and corresponding text information.

[0054] In a preferred embodiment, obtaining the dialect audio sample and the corresponding text information includes:

[0055] Obtaining initial dialect audio, and standardizing the initial dialect audio according to a preset sampling rate and format to obtain standardized dialect audio;

[0056] Performing noise reduction processing on the standardized dialect audio to obtain noise-reduced dialect audio;

[0057] Cut the de-noised dialect audio into segments according to a preset duration to obtain several dialect audio segments;

[0058] Using speaker recognition technology, the voices of different speakers in the dialect audio segment are separated to obtain separated dialect audio segments, so that each separated dialect audio segment corresponds to the voice of a single speaker;

[0059] Removing the silent part from the separated dialect audio segment to obtain the dialect audio sample;

[0060] Perform text transcription on the dialect audio sample to generate corresponding text information.

[0061] In a preferred embodiment, the standardizing of the initial dialect audio according to a preset sampling rate and format to obtain the standardized dialect audio includes: converting the initial dialect audio into a 16000 Hz WAV format to obtain the standardized dialect audio.

[0062] In a preferred embodiment, the preset duration is 5 to 10 seconds.

[0063] For example, taking Foshan dialect as an example, more than 500 hours of Foshan dialect audio materials can be collected from online video platforms to obtain the above-mentioned initial dialect audio.

[0064] The collected Foshan dialect audio materials were then uniformly converted into a WAV format with a sampling rate of 16,000 Hz to obtain the above-mentioned standardized dialect audio; through standardization processing, the audio collected from different online video platforms was converted into a unified format to maintain data consistency for subsequent processing and model training.

[0065] Since the collected audio data is often accompanied by background noise, in order to avoid interference caused by this noise during the training process, we use noise reduction technology to perform noise reduction on the standardized dialect audio to obtain the above-mentioned noise-reduced dialect audio, so as to extract the pure human voice part and ensure the audio quality.

[0066] Next, in order to improve training efficiency and reduce computing resource consumption, the denoised dialect audio is cut into short audio clips of 5 to 10 seconds to obtain the above-mentioned dialect audio clips to meet the needs of model training.

[0067] For the obtained dialect audio clips, speaker recognition technology is used to separate the voices of the different speakers in the multi-speaker dialect audio clips to obtain separated dialect audio clips. This ensures that the separated dialect audio clips contain only the voice of a single speaker, thereby improving the accuracy of timbre recognition. It is understood that if there is only one speaker in the dialect audio clip, separation processing is not required.

[0068] Through silence detection technology, the silent parts in the separated dialect audio clips are identified and removed to obtain the final dialect audio samples. The audio data is further optimized through silence processing and the data volume is reduced, thereby improving the efficiency and quality of the training data.

[0069] Finally, accurate speech recognition is performed on each dialect audio sample, and the recognized audio content is transcribed into text information to obtain the text information corresponding to each dialect audio sample.

[0070] By processing the dialect audio samples used for training through the above steps, the quality of the training samples can be greatly improved, thereby improving the accuracy and efficiency of the trained model.

[0071] S2. Set the parameters of the text encoder, speech segmenter, large language sub-model, and stream matching sub-model to corresponding fixed parameters, and initialize the first low-rank matrix and the second low-rank matrix of the Lora module.

[0072] Specifically, in the present invention, a pre-training model without the Lora module is first constructed. The pre-training model includes the above-mentioned text encoder, speech segmenter, large language sub-model, stream matching sub-model and Hifi GAN vocoder;

[0073] Next, the pre-training model is trained using audio samples of a specific dialect (such as audio samples of the Foshan dialect) and corresponding text information until the pre-training model converges. When the pre-training model converges, the parameters of the text encoder, the parameters of the speech segmenter, the parameters of the large language sub-model, and the parameters of the stream matching sub-model are used as the corresponding fixed parameters of the present invention.

[0074] Next, a dialect speech cloning model is constructed, which includes a Lora module, a text encoder, a speech segmenter, a large language sub-model, a stream matching sub-model, and a Hifi GAN vocoder. The text encoder, speech segmenter, large language sub-model, and stream matching sub-model are set to corresponding fixed parameters and remain unchanged. Then, the first low-rank matrix and the second low-rank matrix of the Lora module are initialized to obtain an initial dialect speech cloning model.

[0075] S3. Using the dialect audio sample and the corresponding text information as input and the cloned speech corresponding to the text information as output, the initial dialect speech cloning model is iteratively trained until the loss function of the dialect speech cloning model converges. During training, the cloned speech output by the vocoder is compared with the dialect audio sample, and the loss function value is calculated based on the comparison result. If the loss function value has not converged, the first low-rank matrix and the second low-rank matrix of the Lora module are updated, and the product of the updated first low-rank matrix and the second low-rank matrix is ​​used as the weight matrix update amount and added to the large language sub-model.

[0076] After obtaining the initial dialect speech cloning model, the dialect audio sample and the corresponding text information are used as input, and the cloned speech corresponding to the text information is used as output. The initial dialect speech cloning model is iteratively trained until the loss function of the dialect speech cloning model converges. The trained dialect speech cloning model can be obtained.

[0077] Specifically, such as Figure 3 As shown in the figure, during the training process, in order to maximize the savings in computing resources, we introduced two low-rank matrices A (i.e., the first low-rank matrix) and B (i.e., the first low-rank matrix) of the Lora module based on the aforementioned large speech sub-model to approximate the original weight matrix W of the large speech sub-model. Since the sizes of A and B are much smaller than W, this significantly reduces the number of parameters and computational cost while maintaining model performance, thereby achieving efficient resource utilization.

[0078] In actual application, by modifying the weight matrix in the linear layer Δ W to achieve:

[0079] (W+ΔW)x=Wx+ABx

[0080] Here, W represents the original weight matrix, and Δ W represents the weight matrix update. In the LoRA framework, the weight matrix update ΔW is decomposed into the product of two low-rank matrices A and B. Since the dimensions of matrices A and B are much smaller than ΔW, this strategy significantly reduces the number of parameters required during training.

[0081] This low-rank matrix factorization method is not only structurally simple but also enhances the model's adaptability to specific new tasks or datasets while maintaining the original capabilities of the pre-trained model. This approach effectively balances the model's generalization ability with rapid adaptation to new situations, providing an effective way to efficiently fine-tune pre-trained models.

[0082] Compared to traditional models, our proposed speech cloning model based on the Lora framework demonstrates significant advantages. This model significantly reduces the amount of training data required, maintaining high synthesis quality even with limited data resources. Furthermore, due to the reduced model complexity, it also has more tolerant requirements on the performance of the running device, thereby broadening the model's application range. Furthermore, the model's training process is more efficient, significantly reducing the required time, which not only improves R&D efficiency but also reduces computing resource consumption. In summary, the Lora-based speech synthesis model demonstrates excellent performance in terms of data dependency, device compatibility, and training efficiency.

[0083] Based on the implementation of the above training method, another embodiment of the present invention provides a training device for a dialect speech cloning model.

[0084] like Figure 4 As shown, an embodiment of the present invention provides a training device for a dialect speech cloning model, comprising: a data acquisition module, a parameter setting module, and an iterative training module;

[0085] The data acquisition module is used to acquire dialect audio samples and corresponding text information;

[0086] The parameter setting module is used to set the parameters of the text encoder, speech segmenter, large language sub-model and stream matching sub-model to corresponding fixed parameters, and initialize the first low-rank matrix and the second low-rank matrix of the Lora module;

[0087] The iterative training module is configured to iteratively train an initial dialect voice cloning model using a dialect audio sample and corresponding text information as input and a cloned voice corresponding to the text information as output until the loss function of the dialect voice cloning model converges. During training, the cloned voice output by the vocoder is compared with the dialect audio sample, and a loss function value is calculated based on the comparison result. If the loss function value does not converge, the first low-rank matrix and the second low-rank matrix of the Lora module are updated, and the product of the updated first low-rank matrix and the second low-rank matrix is ​​used as a weight matrix update amount and added to the large language sub-model.

[0088] In a preferred embodiment, the data acquisition module includes: a standardization processing unit, a noise reduction unit, an audio cutting unit, an audio separation unit, a silence removal unit, and a text transcription unit;

[0089] The standardization processing unit is used to obtain the initial dialect audio, and perform standardization processing on the initial dialect audio according to a preset sampling rate and format to obtain standardized dialect audio;

[0090] The noise reduction unit is configured to perform noise reduction processing on the standardized dialect audio to obtain noise-reduced dialect audio;

[0091] The audio cutting unit is used to cut the de-noised dialect audio according to a preset duration to obtain a plurality of dialect audio segments;

[0092] The audio separation unit is used to separate the voices of different speakers in the dialect audio segment by using speaker recognition technology to obtain separated dialect audio segments, so that each separated dialect audio segment corresponds to the voice of a single speaker;

[0093] The silence removal unit is used to remove the silence part in the separated dialect audio segment to obtain the dialect audio sample;

[0094] The text transcription unit is used to transcribe the dialect audio sample into text to generate corresponding text information.

[0095] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0096] Those skilled in the art will clearly understand that for the sake of convenience and brevity, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0097] Another preferred embodiment of the present invention provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for training a dialect speech cloning model as described in any one of the above embodiments is implemented.

[0098] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0099] The processor may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and connects various parts of the entire terminal device using various interfaces and lines.

[0100] The memory can be used to store the computer program, and the processor realizes various functions of the terminal device by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0101] Another preferred embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is run, the device where the storage medium is located is controlled to execute any one of the methods for training a dialect speech cloning model described in the present invention.

[0102] The storage medium is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. The computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0103] Based on the above embodiment of the method for training a dialect speech cloning model, another embodiment of the present invention provides a dialect speech cloning method, comprising:

[0104] Obtain the dialect speech to be cloned and the text to be synthesized;

[0105] The dialect speech to be cloned and the text to be synthesized are input into a dialect speech cloning model trained by the training method for a dialect speech cloning model according to any one of claims 1 to 4, so that the dialect speech cloning model extracts semantic features of the text to be synthesized through a text encoder and extracts timbre features of the dialect speech to be cloned through a speech segmenter. The semantic features and timbre features are input into a large language sub-model, and a speech token sequence corresponding to the text to be synthesized is generated through the large speech sub-model. The token sequence is converted into a mel-spectrogram through a stream matching sub-model, and sound wave synthesis is performed through a vocoder based on the mel-spectrogram to generate a cloned speech with the same timbre as the dialect speech to be cloned and corresponding to the text content of the text to be synthesized.

[0106] Indicative, such as Figure 5 As shown, the Foshan dialect speech to be cloned and the text to be synthesized are input into the trained dialect speech cloning model. The model combines the processing results of the speech segmenter and text encoder, introduces the trained Lora parameters, and feeds them into a large speech sub-model for in-depth processing. The processed data is then fed into a streaming matching model for matching optimization. Ultimately, the system outputs a Foshan dialect clone speech corresponding to the text to be synthesized. The audio content of the output Foshan dialect clone speech is the text to be synthesized. Assuming that the text to be synthesized is "I am from Foshan" and the input Foshan dialect speech to be cloned is any Foshan dialect content, the generated clone speech is the speech of "I am from Foshan" in the form of Foshan dialect.

[0107] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for training a dialect speech cloning model, characterized in that: Applicable to the dialect speech cloning model, the dialect speech cloning model includes: a text encoder, a speech word segmenter, a large language sub-model, a stream matching sub-model, a Lora module and a vocoder; The training method comprises: Obtain dialect audio samples and corresponding text information; Set the parameters of the text encoder, speech segmenter, large language sub-model, and stream matching sub-model to corresponding fixed parameters, and initialize the first low-rank matrix and the second low-rank matrix of the Lora module; The method uses a dialect audio sample and corresponding text information as input and a cloned speech corresponding to the text information as output, iteratively training the initial dialect speech cloning model until the loss function of the dialect speech cloning model converges. During the training, the cloned speech output by the vocoder is compared with the dialect audio sample, and the loss function value is calculated based on the comparison result. If the loss function value does not converge, the first low-rank matrix and the second low-rank matrix of the Lora module are updated, and the product of the updated first low-rank matrix and the second low-rank matrix is ​​used as the weight matrix update amount and added to the large language sub-model. The method obtains the dialect audio sample and the corresponding text information, including: Obtaining initial dialect audio, and standardizing the initial dialect audio according to a preset sampling rate and format to obtain standardized dialect audio; Performing noise reduction processing on the standardized dialect audio to obtain noise-reduced dialect audio; Cut the de-noised dialect audio into segments according to a preset duration to obtain several dialect audio segments; Using speaker recognition technology, the voices of different speakers in the dialect audio segment are separated to obtain separated dialect audio segments, so that each separated dialect audio segment corresponds to the voice of a single speaker; Removing the silent part from the separated dialect audio segment to obtain the dialect audio sample; Perform text transcription on the dialect audio sample to generate corresponding text information.

2. The method for training a dialect speech cloning model according to claim 1, wherein: The step of normalizing the initial dialect audio according to a preset sampling rate and format to obtain the standardized dialect audio includes: The initial dialect audio is converted into a 16000 Hz WAV format to obtain a standardized dialect audio.

3. The method for training a dialect speech cloning model according to claim 2, wherein: The preset duration is 5 to 10 seconds.

4. A training device for a dialect speech cloning model, characterized in that: Suitable for training a dialect speech cloning model, the dialect speech cloning model includes: a text encoder, a speech word segmenter, a large language sub-model, a stream matching sub-model, a Lora module and a vocoder; The training device includes: a data acquisition module, a parameter setting module and an iterative training module; The data acquisition module is used to acquire dialect audio samples and corresponding text information; The parameter setting module is used to set the parameters of the text encoder, speech segmenter, large language sub-model and stream matching sub-model to corresponding fixed parameters, and initialize the first low-rank matrix and the second low-rank matrix of the Lora module; The iterative training module is used to iteratively train the initial dialect voice cloning model using the dialect audio sample and the corresponding text information as input and the cloned voice corresponding to the text information as output until the loss function of the dialect voice cloning model converges. During the training, the cloned voice output by the vocoder is compared with the dialect audio sample, and the loss function value is calculated based on the comparison result. If the loss function value does not converge, the first low-rank matrix and the second low-rank matrix of the Lora module are updated, and the product of the updated first low-rank matrix and the second low-rank matrix is ​​used as the weight matrix update amount and added to the large language sub-model. The data acquisition module includes: a normalization processing unit, a noise reduction unit, an audio segmentation unit, an audio separation unit, a silence removal unit, and a text transcription unit. The standardization processing unit is used to obtain the initial dialect audio, and perform standardization processing on the initial dialect audio according to a preset sampling rate and format to obtain standardized dialect audio; The noise reduction unit is configured to perform noise reduction processing on the standardized dialect audio to obtain noise-reduced dialect audio; The audio cutting unit is used to cut the de-noised dialect audio according to a preset duration to obtain a plurality of dialect audio segments; The audio separation unit is used to separate the voices of different speakers in the dialect audio segment by using speaker recognition technology to obtain separated dialect audio segments, so that each separated dialect audio segment corresponds to the voice of a single speaker; The silence removal unit is used to remove the silence part in the separated dialect audio segment to obtain the dialect audio sample; The text transcription unit is used to transcribe the dialect audio sample into text to generate corresponding text information.

5. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for training a dialect speech cloning model according to any one of claims 1 to 3 is implemented.

6. A storage medium, characterized in that The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the method for training a dialect speech cloning model according to any one of claims 1 to 3.

7. A dialect voice cloning method, characterized in that: include: Obtain the dialect speech to be cloned and the text to be synthesized; The dialect speech to be cloned and the text to be synthesized are input into a dialect speech cloning model trained by the dialect speech cloning model training method according to any one of claims 1 to 3, so that the dialect speech cloning model extracts semantic features of the text to be synthesized through a text encoder and extracts timbre features of the dialect speech to be cloned through a speech segmenter. The semantic features and timbre features are input into a large language sub-model, and a speech token sequence corresponding to the text to be synthesized is generated through the large speech sub-model. The token sequence is converted into a mel-spectrogram through a stream matching sub-model, and sound wave synthesis is performed through a vocoder based on the mel-spectrogram to generate a cloned speech with the same timbre as the dialect speech to be cloned and corresponding to the text content of the text to be synthesized.

Images