Real-time sound duplicating method and system based on end-cloud fusion

Abstract

The invention provides a real-time sound copying method and system based on end-cloud fusion. The method comprises the following steps: a cloud end carries out real-time tone copying and voice synthesis on a small amount of voice data of a user based on an AI large model; timbre samples are collected when a user registers voice audio data, and user timbre voice data of a preset text are synchronously generated by a large model and are used as fine tuning training data of an end-side voice synthesis model; user tone voice data of a preset text and voice audio data registered by a user are utilized to carry out migration fine tuning training on an end-side voice synthesis model to adapt to the personalized tone of the user, so that high-quality output of the end-side voice synthesis model is ensured, and personalized voice replication is realized; and issuing the trained end-side speech synthesis model to the user equipment, and independently completing speech replication in a network-free or weak network environment. According to the invention, through automatic generation of the user tone data and adaptive fine tuning of the model, the tone of the user is deployed to the end side after fine tuning, and high-quality and high-adaptability sound replication of end-cloud collaboration is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of speech synthesis, artificial intelligence and mobile terminals, in particular, to a real-time voice cloning technology supporting end-cloud collaboration; specifically, to a real-time voice cloning method and system based on end-cloud fusion. BACKGROUND

[0002] Current AI-based speech synthesis and voice cloning technologies are widely used in voice assistants and other scenarios, especially zero-shot voice cloning based on large models, which can generate personalized timbre with only a small amount of input voice samples, for example, a user can register about 10 seconds of voice data to clone his own voice in real time.

[0003] However, the existing voice cloning scheme relies on cloud high-performance GPU inference, and users cannot enjoy the exclusive timbre experience in a network-free or weak network environment, which greatly affects product reliability and user immersion.

[0004] Traditional end-side speech synthesis models are limited by small sample adaptability and cannot achieve high-quality personalized voice cloning. SUMMARY

[0005] Therefore, the present application aims to provide a real-time voice cloning method and system based on end-cloud fusion, which combines cloud real-time voice cloning models and end-side speech synthesis models, uses zero-shot voice cloning technology supported by cloud strong computing power to achieve real-time high-quality cloning and synthesis of user timbre; At the same time, through user timbre data automatic generation and model adaptive fine-tuning technology, the user's timbre is fine-tuned and deployed to the end side, realizing high-quality and high-adaptability voice cloning in end-cloud collaboration, ensuring voice cloning effect in a network-free or weak network environment; In actual application scenarios (such as vehicle-mounted voice assistants), users can obtain real-time voice cloning experience close to their original timbre in any network environment, balancing high quality, low latency and network adaptability, and improving user experience in all-time and all-scenarios.

[0006] The present application provides a real-time voice cloning method based on end-cloud fusion, comprising the following steps:

[0007] S1, based on AI large model (such as zero-shot voice cloning model) in the cloud, real-time timbre cloning and speech synthesis are performed on a small amount of voice data of a user;

[0008] The "real-time timbre cloning" and "speech synthesis" of the present application are key technologies for realizing personalized voice output, and they work closely together to achieve full-scenario and high-quality voice cloning effect with the help of end-cloud collaboration architecture.

[0009] S2, collect the voice tone samples of the user when the user registers voice audio data, synchronously generate the voice tone voice data of the user of the preset text by a large model, and use the voice tone voice data as fine-tuning training data of an end-side voice synthesis model (an end-side user voice cloning model);

[0010] Specifically, the content of the text annotation (preset text) is pre-set, and then based on the voice audio data registered by the user, the zero-shot cloning technology based on the large model, such as cosyvoice2, spark-tts, F5, etc., is used to generate the audio data (user voice tone voice data) corresponding to the text. The audio data is very close to the voice tone of the user.

[0011] Specifically, the fine-tuning training data is a specific data set selected to optimize the performance of the end-side model and make targeted adjustments to the pre-trained model. The data source of the fine-tuning training data is: when the user registers, the cloud generates a large amount of user voice tone voice data of the preset text based on a small amount of voice tone samples provided by the user through the zero-shot voice cloning technology. This is an important part of the fine-tuning training data, which solves the limitation of the small sample adaptability of the end-side voice synthesis model. In addition, the collected user voice tone voice data-text annotation and the voice audio data registered by the user are also included in the fine-tuning training data category. These data contain rich user voice tone information and the corresponding relationship between voice and text.

[0012] In an embodiment of the present application, in the user registration stage, the cloud generates a large amount of user voice tone voice data of the preset text based on a small amount of voice tone samples of the user by using the zero-shot voice cloning technology. These data are used for fine-tuning training of the end-side voice synthesis model (end-side user voice cloning model). At the same time, the collected “user voice tone data-text annotation” and the voice data registered by the user are also put into the migration fine-tuning training of the end-side voice synthesis model. Through multiple rounds of fine-tuning training, the end-side voice synthesis model can deeply learn the user voice tone characteristics and automatically adapt to the user personalized voice tone, so as to realize high-quality personalized voice cloning. Even in a network-free or weak network environment, the cloned voice close to the original voice tone of the user can be stably outputted;

[0013] S3, use the collected user voice tone voice data (user voice tone data-text annotation) of the preset text and the voice audio data registered by the user to perform migration fine-tuning training on the end-side voice synthesis model, automatically adapt to the user personalized voice tone, ensure the high-quality output of the end-side voice synthesis model, and realize high-quality personalized voice cloning of the end-side voice synthesis model;

[0014] Adapting to user's personalized timbre is the core goal of achieving high-quality personalized voice replication. Specifically, personalized voice replication refers to using advanced speech processing and artificial intelligence technologies to accurately simulate the speech characteristics of a specific individual, generating speech content that is highly similar to the person's real voice. Personalized voice replication not only replicates the timbre, but also restores the rhythm, tone, and habitual expressions of the speaker. Some implementations of personalized voice replication are as follows:

[0015] For example, in the production of film and television works in the field of film and television dubbing, sometimes actors cannot personally dub for a role due to special reasons, or they want to give the character a more unique voice style. In this case, personalized voice replication technology can be used to collect a small amount of voice samples from the actor, and through cloud-based AI large models and on-device fine-tuning training, the actor's voice can be replicated. This way, even if the actor is not present, the voice replication can complete the dubbing work, ensuring the coherence and consistency of the character's voice. For example, dubbing for characters in science fiction movies who cannot be recorded in sync due to special special effects; audio book creation, well-known hosts or writers often have unique voices that are highly recognizable and loved by listeners. When creating an audio book, the host's or writer's voice can be replicated using personalized voice replication technology. Based on the replicated voice, a large amount of text content can be efficiently converted into an audio work, meeting the market's demand for the host's or writer's voice work, while also saving the host's or writer's time and effort, allowing listeners to hear more stories narrated by familiar voices.

[0016] In the customization of intelligent voice assistants, users can provide their own voice samples to devices such as smart speakers and mobile voice assistants, and use zero-shot cloning and real-time voice replication technologies to customize the voice of the voice assistant, making the voice assistant sound similar to the user. This way, when interacting with the voice assistant, the user is like talking to another "self", increasing the fun and intimacy of use.

[0017] In the telephone customer service system, large enterprise customer service centers handle a large number of consultation calls every day. Using personalized voice replication technology, the voices of professional customer service personnel can be replicated and applied to the automatic voice response system. This not only ensures the professionalism and affinity of the customer service voice, but also improves service efficiency, while also reducing labor costs to some extent.

[0018] S4, the trained end-side speech synthesis model is sent to the user device, so that the end-side speech synthesis model can independently complete voice replication in a networkless or weak network environment.

[0019] The trained end-side speech synthesis model is sent to the user equipment, and the user's voice tone is fine-tuned and deployed to the end side; in a network-free or weak network environment, the speech synthesis work is independently completed by the end-side speech synthesis model; the end-side speech synthesis model processes the input text according to the user voice tone features learned by the previous fine-tuning training and the built-in speech synthesis algorithm. Although the computing power of the end-side speech synthesis model is relatively weak compared to the cloud AI large model, through the previous optimization training, it can still synthesize speech close to the user's original voice tone, ensuring that users can still enjoy personalized voice services when the network is poor.

[0020] The application supports seamless switching between the end and the cloud: the cloud model is used in a good network environment to ensure the highest synthesis quality, and the local end model is switched to in a weak network or offline scene, so that users can always experience personalized voice tones. If the computing power, delay and other problems are considered, only the end-side sound replication model can be used to generate audio data of the user's voice tone.

[0021] Further, the method of migration fine-tuning training of the S3 step comprises:

[0022] S301. A pre-trained model that has completed pre-training on large-scale general speech data is selected, and the pre-trained model has learned rich speech features, basic knowledge of acoustic rules, and specific understanding and expression ability of the basic structure of speech and pronunciation rules;

[0023] The pre-trained model, such as a common speech synthesis pre-trained model, is like a “student” who has already acquired a certain knowledge reserve, laying a foundation for subsequent personalized training;

[0024] S302, collect speech data related to a specific user, which contains the personalized speech features of the unique voice tone, tone and speed of the specific user;

[0025] The speech data related to the specific user is the key material for the large model to learn the unique voice style of the user; the data sources of the speech data related to the specific user include: the speech samples provided by the user during registration, the user voice tone speech data of a large number of preset texts generated by the cloud based on a small number of samples through the zero-shot voice cloning technology, and “user voice tone data-text annotation”;

[0026] S303, determine the key parameters in the fine-tuning training process, including the learning rate and the number of training rounds; at the same time, develop a training strategy, adopt the transfer learning method, retain the parameters related to the commonality of speech in the pre-trained model, and only adjust and update the parameters related to the personalized voice tone and speech habits;

[0027] Specifically, the learning rate determines the step size of parameter updates during training, and a smaller learning rate leads to more stable but slower training, while a larger learning rate can cause the model to not converge; the number of training rounds determines the number of times the model learns from the data.

[0028] S304, input the prepared personalized data into the pre-trained model and train according to the determined training parameters and strategies; during the training process, the pre-trained model adjusts its parameters according to the input voice data and corresponding text annotations to minimize the difference between the predicted voice and the real user voice; for example, by using a backpropagation algorithm to calculate the error and feeding the error back to each level of the pre-trained model to update the parameters, so that the pre-trained model gradually learns to generate voice that meets the user's personalized characteristics.

[0029] S305, after the training is completed, evaluate the fine-tuned pre-trained model using special evaluation indicators and test data, such as subjective evaluation (invite users or professionals to score the synthesized voice) and objective evaluation (calculate the acoustic feature similarity between the synthesized voice and the real voice, etc.) to judge the quality and personalization degree of the evaluation model generated voice; if the evaluation result is not satisfactory, adjust the training parameters or increase the training data, retrain, and continuously optimize the model until the desired personalized sound replication effect is achieved.

[0030] Further, the method for synchronously generating user voice tone voice data of preset text by the large model in the S2 step comprises:

[0031] During the user registration phase, a large amount of user voice tone voice data of preset text is synchronously generated based on the zero-shot voice cloning technology of the cloud-side large model using the small amount of voice tone samples provided by the user as the end-side voice synthesis model fine-tuning training data.

[0032] "zero-shot cloning" or zero-sample cloning, such as cosyvoice2 and Spark-TTS, is a voice cloning technology based on AI large models, which can generate synthesized voice with high consistency in voice tone in real time based on very short voice samples (usually 5-10 seconds) without the need for specific training data of the target speaker, achieving personalized voice generation and playing a key role in the real-time sound replication system in this document.

[0033] Due to the limitations of small sample adaptability of traditional end-side voice synthesis models, it is difficult to achieve high-quality personalized sound replication, while the large amount of data generated by the zero-shot cloning technology provides rich learning materials for the end-side voice synthesis model, enabling it to better learn the characteristics of the user's voice, thereby solving the problem of small sample adaptability of the end-side voice synthesis model and achieving high-quality personalized sound replication.

[0034] Further, the method of evaluating the fine-tuned pre-trained model using special evaluation indicators and test data in the S305 step includes:

[0035] In terms of subjective evaluation, users or professionals are invited to listen to the synthesized speech of the model and score and evaluate it from the dimensions of timbre similarity and naturalness. In terms of objective evaluation, the similarity index of the synthesized speech and the real user speech in terms of fundamental frequency and formant is calculated to quantitatively evaluate the adaptation effect of the model. According to the evaluation results, if it is found that there is a gap between the synthesized speech and the user's real timbre, the training parameters are adjusted, such as appropriately increasing or decreasing the learning rate or the number of training rounds, or more user speech data is collected, and the transfer fine-tuning training is performed again. The model is iteratively optimized until the model can accurately adapt to the user's personalized timbre and generate highly similar speech content.

[0036] Further, the transfer fine-tuning training in the S3 step is: with the help of the knowledge of the pre-trained model, through specific data and training strategies, the model is adapted to personalized needs; through fine-tuning training data, the model learns the personalized information of the user's timbre characteristics and speech habits.

[0037] Since the pre-trained model is trained on general speech data, although it has certain speech processing capabilities, it lacks adaptation to the timbre of a specific user, so the transfer fine-tuning training plays a key role in personalized voice replication.

[0038] For example, the model learns the user's unique intonation fluctuations and pronunciation habits through specific data, thereby realizing the transformation from general speech synthesis capability to personalized voice replication capability, so that the model can output speech that is more consistent with the user's real timbre when processing speech synthesis tasks related to the user.

[0039] Further, the method of real-time timbre replication of the user's small amount of speech data in the S1 step includes:

[0040] Real-time timbre replication is carried out on the cloud with the help of an AI large model; a small amount of speech data samples are provided when the user registers, and the AI large model accurately extracts the key acoustic features of the user's timbre from the limited speech data samples, including fundamental frequency, formant, etc.; the AI large model generates a highly similar replicated timbre based on the key acoustic features of the user's timbre.

[0041] For example, when a user registers, 10 seconds or so of voice data is provided, and a large model in the cloud (such as a zero-shot voice cloning model) can use the 10 seconds or so of voice data as a basis, and the large model can extract unique features of the user's voice timbre from the limited voice samples by virtue of its powerful learning and generalization capabilities, and then synthesize voice with the same timbre characteristics, real-time voice timbre replication, and further complete voice synthesis, and the synthesis quality is superior.

[0042] Further, the voice synthesis method of the S1 step comprises:

[0043] After the cloud completes real-time voice timbre replication and obtains the replicated voice timbre of the user, the AI large model combines the text to be synthesized, uses a deep learning algorithm to perform semantic understanding and prosody analysis on the text; for example, judges the emotional tendency of the text, sentence breaks, pauses, etc., and then combines the replicated voice timbre with the text content to generate natural and smooth high-quality voice.

[0044] Specifically, in the case of good network, the cloud large model is preferentially used for voice synthesis to ensure the best synthesis effect.

[0045] The application also provides a real-time voice replication system based on end-to-cloud fusion, which executes the real-time voice replication method based on end-to-cloud fusion as described above, comprising:

[0046] The cloud sample voice timbre replication and voice synthesis module is used for real-time voice timbre replication and voice synthesis of a user's small amount of voice data based on an AI large model in the cloud;

[0047] The user voice timbre voice data generation module is used for collecting voice timbre samples of a user when the user registers voice audio data, and synchronously generating user voice timbre voice data of a preset text by a large model, and using the user voice timbre voice data as fine-tuning training data of an end-side voice synthesis model;

[0048] The end-side voice synthesis model fine-tuning training module is used for performing migration fine-tuning training on the end-side voice synthesis model by using the collected user voice timbre voice data of the preset text and the voice audio data registered by the user, automatically adapting the user's personalized voice timbre, ensuring high-quality output of the end-side voice synthesis model, and realizing high-quality personalized voice replication of the end-side voice synthesis model;

[0049] The end-side voice synthesis model distribution module is used for distributing the trained end-side voice synthesis model to a user device, so that the end-side voice synthesis model can independently complete voice replication in a networkless or weak network environment.

[0050] The application also provides a computer readable storage medium, which stores a computer program, and the program, when executed by a processor, implements the steps of the real-time sound replication method based on end-cloud fusion.

[0051] The application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the real-time sound replication method based on end-cloud fusion when executing the program.

[0052] Compared with the prior art, the application has the following beneficial effects:

[0053] The real-time sound replication method and system based on end-cloud fusion provided by the application adopt an end-cloud fusion architecture, fuse a cloud-side real-time sound replication model and an end-side speech synthesis model, utilize a zero-shot speech cloning technology supported by strong cloud-side computing power, realize consistent and high-quality replication of timbre in all-time and all-scenarios (online and offline), and implement an end-cloud collaborative high-quality and high-adaptability sound replication system by automatically generating user timbre data and model adaptive fine-tuning technology, thereby guaranteeing the speech replication effect in a network-free or weak network environment; based on the zero-shot cloning of a large model and the migration of a small-sample end model, the user's unique timbre can be quickly and efficiently adapted, and the system is personalized; the end-side sound replication generates a large amount of user timbre sound data by means of the zero-shot technology based on a large model, and the data is used to fine-tune the model, so that the effect of the cloud-side sound replication can be approached, the timbre difference is further reduced when switching between the end and the cloud, the user has no perception, and the user experience is good; the cloud-side high quality and the end-side usability are combined, and the end-side speech synthesis model delivery mechanism can guarantee that the user can still use "his own voice" in a weak network / offline scenario, so that the user can obtain real-time and close-to-original timbre speech cloning experience in any network environment, the high quality, low delay and network adaptability are considered, and the product experience of the user in all-time and all-scenarios is improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not considered a limitation of the application.

[0055] In the drawings:

[0056] Figure 1 is a cloud-side real-time sound replication scheme of an embodiment of the application;

[0057] Figure 2 is a real-time sound replication system architecture based on end-cloud fusion of an embodiment of the application;

[0058] Figure 3 A flow chart of a real-time sound replication method based on end-cloud fusion according to an embodiment of the present application;

[0059] Figure 4 A structural schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar components. The following exemplary embodiments are not representative of all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and articles of manufacture consistent with some aspects of the present disclosure as detailed in the appended claims.

[0061] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0062] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish different sets of information from one another. For example, a first information can be termed a second information, and similarly, a second information can also be termed a first information, without departing from the scope of the present disclosure. As used herein, the word "if' can be construed to mean "when" or "in response to determining" depending on the context.

[0063] The embodiments of the present application are further described below.

[0064] An embodiment of the present application provides a real-time sound replication method based on end-cloud fusion, referring to Figure 3 as shown, comprising the following steps:

[0065] S1, based on an AI large model (zero-shot voice cloning model) in the cloud, real-time timbre replication and speech synthesis are performed on a small amount of voice data of a user;

[0066] The method for real-time timbre replication of a small amount of voice data of a user comprises:

[0067] The AI large model is used to perform real-time timbre replication in the cloud. When a user registers, a small amount of voice data sample is provided, and the AI large model is used to accurately extract the key acoustic features of the user's timbre from the limited voice data sample, including the fundamental frequency and the formant. The AI large model generates a replicated timbre that is highly similar to the user's original timbre in real time based on the key acoustic features of the user's timbre.

[0068] In this embodiment, the user provides about 10 seconds of voice data when registering. The zero-shot voice cloning model in the cloud can extract the unique features of the user's timbre from the limited voice sample based on the 10 seconds of voice data. Then, the zero-shot voice cloning model can synthesize voice with the same timbre characteristics based on these features. The process of real-time timbre replication and further voice synthesis can be completed in a short time, meeting the real-time requirements. Figure 1 The cloud real-time sound replication scheme of this embodiment is shown.

[0069] The method for voice synthesis includes: after the cloud completes real-time timbre replication and obtains the replicated timbre of the user, the AI large model combines the text to be synthesized and uses deep learning algorithms for semantic understanding and prosodic analysis of the text, such as determining the emotional tendency of the text, sentence segmentation, and pauses, and then combines the replicated timbre with the text content to generate natural and smooth high-quality voice. In good network conditions, the cloud's large model is preferred for voice synthesis to ensure the best synthesis effect.

[0070] S2, when the user registers voice audio data, collect the user's timbre sample, and generate the user's timbre voice data of the preset text by the large model at the same time, use the user's timbre voice data as the fine-tuning training data of the end-side voice synthesis model;

[0071] The method for generating the user's timbre voice data of the preset text by the large model at the same time includes:

[0072] In the user registration stage, a small amount of timbre sample provided by the user is used to generate a large amount of user timbre voice data of the preset text based on the zero-shot voice cloning technology of the cloud large model, which is used as the fine-tuning training data of the end-side voice synthesis model.

[0073] The large amount of data generated by the zero-shot cloning technology provides rich learning materials for the end-side voice synthesis model, enabling the end-side voice synthesis model to better learn the characteristics of the user's timbre, thereby solving the problem of small sample adaptability of the end-side voice synthesis model and achieving high-quality personalized sound replication.

[0074] The method for evaluating the fine-tuned pre-trained model using special evaluation indicators and test data includes:

[0075] In terms of subjective evaluation, users or professionals are invited to listen to the synthesized speech of the model and score and evaluate it from the dimensions of timbre similarity and naturalness. In terms of objective evaluation, the similarity index of the synthesized speech and the real user speech in terms of fundamental frequency, formant and other acoustic characteristics is calculated to quantitatively evaluate the adaptation effect of the model. According to the evaluation results, if it is found that there is a gap between the synthesized speech and the user's real timbre, the training parameters are adjusted, such as appropriately increasing or decreasing the learning rate or the number of training rounds, or more user speech data is collected, and the transfer fine-tuning training is performed again. The model is iteratively optimized until the model can accurately adapt to the user's personalized timbre and generate highly similar speech content.

[0076] The content of the text label (preset text) is pre-set, and then based on the user-registered voice audio data, the zero-shot cloning technology based on large models, such as cosyvoice2, spark-tts, F5, etc., is used to generate audio data (user timbre voice data) corresponding to these texts. These audio data are very close to the user's timbre.

[0077] The data source of the fine-tuning training data is: during user registration, the cloud generates a large amount of user timbre voice data of preset texts based on a small amount of timbre samples provided by the user through zero-shot voice cloning technology, solving the limitation of small sample adaptability of the end-side speech synthesis model; in addition, the collected user timbre voice data-text label and user-registered voice audio data are also included in the fine-tuning training data category; these data contain rich user timbre information and the corresponding relationship between speech and text;

[0078] In this embodiment, during the user registration stage, the cloud generates a large amount of user timbre voice data of preset texts based on a small amount of timbre samples provided by the user using zero-shot voice cloning technology; these data are used for fine-tuning training of the end-side speech synthesis model (end-side user voice replication model). At the same time, the collected "user timbre data-text label" and user-registered voice data are also put into the transfer fine-tuning training of the end-side speech synthesis model; through multiple rounds of fine-tuning training, the end-side speech synthesis model can deeply learn the user timbre characteristics and automatically adapt to the user's personalized timbre, thereby realizing high-quality personalized voice replication. Even in a network-free or weak network environment, the user's original timbre can also be stably outputted.

[0079] S3, using the collected user voice tone voice data of the preset text and the user registered voice audio data, performing fine-tuning training on the end-side voice synthesis model, automatically adapting the user personalized tone, ensuring high-quality output of the end-side voice synthesis model, and realizing high-quality personalized sound replication of the end-side voice synthesis model;

[0080] The fine-tuning training is to adapt the model to personalized needs by means of the knowledge of the pre-trained model and through specific data and training strategies, and to make the model learn the personalized information of the tone characteristics and speech habits of a specific user through fine-tuning training data.

[0081] Since the pre-trained model is trained on general voice data, although it has certain speech processing capabilities, it lacks adaptation to the tone of a specific user, so the fine-tuning training plays a key role in personalized sound replication. The pre-trained model learns the unique intonation fluctuations and pronunciation habits of a certain user through specific data, thereby realizing the transformation from general voice synthesis capability to personalized voice replication capability, so that the model can output voice that is more consistent with the real tone of the user when processing voice synthesis tasks related to the user.

[0082] The method of fine-tuning training includes the following steps:

[0083] S301. Select a pre-trained model that has been pre-trained on large-scale general voice data, the pre-trained model having learned rich voice features and basic knowledge of acoustic rules, and having specific understanding and expression ability for the basic structure of voice and pronunciation rules;

[0084] S302, collect voice data related to a specific user, the voice data related to the specific user containing personalized voice features of the unique tone, intonation and speech rate of the specific user;

[0085] The voice data related to the specific user is the key material for the large model to learn the unique sound style of the user. The data sources of the voice data related to the specific user include: voice samples provided by the user during registration, a large amount of user voice tone voice data of preset texts generated by the cloud based on a small amount of samples through zero-shot voice cloning technology, and "user voice tone data-text annotation";

[0086] S303, determine the key parameters in the fine-tuning training process, the key parameters including learning rate and training rounds; at the same time, develop a training strategy, adopt the transfer learning method, retain the parameters related to the commonality of voice in the pre-trained model, and only adjust and update the parameters related to personalized tone and speech habits;

[0087] S304, input the prepared personalized data into the pre-trained model, and train according to the determined training parameters and strategies; during the training process, the pre-trained model adjusts its parameters according to the input voice data and corresponding text labels, so as to minimize the difference between the predicted voice and the real user voice;

[0088] In this embodiment, the error is calculated by the back propagation algorithm, and the error is fed back to each level of the pre-trained model to update the parameters, so that the pre-trained model gradually learns to generate voice that meets the personalized characteristics of the user.

[0089] S305, after the training is completed, the pre-trained model after fine-tuning is evaluated using special evaluation indicators and test data to judge the quality and personalization degree of the voice generated by the evaluation model; if the evaluation result is not ideal, the training parameters or the training data are adjusted, and the training is re-performed, and the model is continuously optimized until the satisfactory personalized sound replication effect is achieved.

[0090] The personalized sound replication can not only replicate the timbre, but also restore the rhythm, intonation, habitual expressions and other personalized characteristics of the speaker.

[0091] S4, the end-side speech synthesis model after completing the training is issued to the user equipment, so that the end-side speech synthesis model independently completes the voice replication in a network-free or weak network environment.

[0092] The end-side speech synthesis model after completing the training is issued to the user equipment, and the user's timbre is fine-tuned and deployed to the end side; in a network-free or weak network environment, the end-side speech synthesis model independently completes the speech synthesis work; the end-side speech synthesis model processes the input text according to the user timbre characteristics learned by the previous fine-tuning and training and the built-in speech synthesis algorithm. Although the computing power of the end-side speech synthesis model is relatively weak compared with the cloud AI large model, through the previous optimization training, it can still synthesize voice close to the user's original timbre, ensuring that the user can still enjoy personalized voice service when the network is poor.

[0093] This embodiment supports seamless switching between the end and the cloud: the cloud model is used in the case of good network to ensure the highest synthesis quality, and the local end model is switched to in the weak network or offline scene, so that the user can always experience personalized timbre. If the computing power, delay and other problems are considered, only the end-side sound replication model can be used to generate audio data of the user's timbre.

[0094] The embodiment of the application also provides a real-time sound replication system based on end-cloud fusion, which executes the real-time sound replication method based on end-cloud fusion as described above, comprising:

[0095] The cloud sample timbre replication and speech synthesis module is used for real-time timbre replication and speech synthesis of a small amount of voice data of a user based on an AI large model in the cloud.

[0096] A user voice timbre voice data generation module is configured to collect a user's voice timbre sample when the user registers voice audio data, and to generate user voice timbre voice data of preset text synchronously by a large model, and use the user voice timbre voice data as fine-tuning training data of an end-side voice synthesis model;

[0097] An end-side voice synthesis model fine-tuning training module is configured to perform migration fine-tuning training on the end-side voice synthesis model by using the collected user voice timbre voice data of preset text and voice audio data registered by the user, to automatically adapt the user's personalized voice timbre, to ensure high-quality output of the end-side voice synthesis model, and to realize high-quality personalized sound replication of the end-side voice synthesis model;

[0098] An end-side voice synthesis model issuing module is configured to issue the trained end-side voice synthesis model to a user device, so that the end-side voice synthesis model can independently complete voice replication in a network-free or weak network environment.

[0099] Figure 2 The end-cloud fusion-based real-time sound replication system architecture of this embodiment is shown.

[0100] The end-cloud fusion-based real-time sound replication method and system of this embodiment adopt an end-cloud fusion architecture, fuse a cloud-side real-time sound replication model and an end-side voice synthesis model, use a zero-shot voice cloning technology supported by cloud-side strong computing power, realize consistent and high-quality replication of voice timbre in all-time and all-scenarios (online and offline), and through user voice timbre data automatic generation and model adaptive fine-tuning technology, the user's voice timbre is fine-tuned and deployed to the end side, realizing an end-cloud collaborative high-quality and highly adaptive sound replication system, and guaranteeing voice replication effect in a network-free or weak network environment; based on the zero-shot cloning of a large model and the migration of a small-sample end model, the user's unique voice timbre can be quickly and efficiently adapted, and the system is highly personalized; the end-side sound replication generates a large amount of user voice timbre sound data by means of the zero-shot technology based on a large model, to fine-tune the model, which can approach the effect of cloud-side sound replication, further reduce the difference in voice timbre when switching between the end side and the cloud side, and provide a good user experience; the cloud-side high quality and the end-side usability are combined, and the end-side voice synthesis model issuing mechanism can guarantee that the user can still use "his own voice" in a weak network / offline scenario, and guarantee that the user can obtain real-time voice cloning experience close to the original voice timbre in any network environment, and high quality, low delay and network adaptability are considered.

[0101] The embodiment of the application further provides a computer device, Figure 4 is a structural schematic diagram of a computer device provided by the embodiment of the application; see the accompanying drawings Figure 4As shown, the computer device includes: an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is configured to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the real-time sound replication method based on end-cloud fusion provided by the above-mentioned embodiments; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 4 The bus connection is taken as an example.

[0102] The memory 22 is a readable and writable storage medium of a computer device, which can be used to store software programs, computer executable programs, such as program instructions of the real-time sound replication method based on end-cloud fusion provided by the embodiments of the present application; the memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the device, etc.; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some examples, the memory 22 can further include a memory remotely arranged with respect to the processor 21, and these remote memories can be connected to the device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0103] The input system 23 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the device; the output system 24 can include a display device such as a display screen.

[0104] The processor 21 executes various function applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, that is, implements the real-time sound replication method based on end-cloud fusion described above.

[0105] The computer device provided above can be used to execute the real-time sound replication method based on end-cloud fusion provided by the above-mentioned embodiments, and has corresponding functions and beneficial effects.

[0106] The embodiment of the present application also provides a storage medium containing computer executable instructions, which are used for executing the real-time sound replication method based on end-cloud fusion provided by the above embodiment when executed by a computer processor. The storage medium is any various type of memory device or storage device, and the storage medium includes: installation medium such as CD-ROM, floppy disk or tape system; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM and the like; non-volatile memory such as flash memory, magnetic medium (for example, hard disk or optical storage); register or other similar type of memory element and the like; the storage medium can also include other types of memory or combinations thereof; in addition, the storage medium can be located in a first computer system in which the program is executed, or can be located in a different second computer system, and the second computer system is connected to the first computer system through a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected through a network). The storage medium can store program instructions (for example, computer programs) executable by one or more processors.

[0107] Of course, the storage medium containing computer executable instructions provided by the embodiment of the present application is not limited to the real-time sound replication method based on end-cloud fusion described in the above embodiment, and can also execute the related operations in the real-time sound replication method based on end-cloud fusion provided by any embodiment of the present application.

[0108] So far, the technical solutions of the present application have been described in combination with preferred embodiments, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will fall within the protection scope of the present application.

[0109] The above description is only the preferred embodiments of the present application and is not used to limit the present application; for those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time sound replication method based on edge-cloud fusion, characterized in that, Includes the following steps: S1. Based on a large AI model in the cloud, real-time timbre replication and speech synthesis are performed on a small amount of user voice data. S2. When a user registers voice audio data, the user's timbre samples are collected, and the user's timbre voice data is generated synchronously by the large model. The user's timbre voice data is used as fine-tuning training data for the edge speech synthesis model. S3. Using the collected user voice data of preset text and user registered voice audio data, the terminal speech synthesis model is transferred and fine-tuned to automatically adapt to the user's personalized voice, ensuring the high-quality output of the terminal speech synthesis model and realizing high-quality personalized voice replication of the terminal speech synthesis model. S4. The trained edge speech synthesis model is sent to the user equipment so that the edge speech synthesis model can independently complete speech replication in environments without network or with weak network.

2. The real-time sound replication method based on edge-cloud fusion according to claim 1, characterized in that, The transfer fine-tuning training method in step S3 includes: S301. Select a pre-trained model that has been pre-trained on large-scale general speech data. The pre-trained model has learned rich speech features and basic knowledge of acoustic laws, and has a specific understanding and expression ability of the basic structure of speech and pronunciation rules. S302. Collect voice data related to a specific user, wherein the voice data related to the specific user includes personalized voice features of the specific user, such as unique timbre, intonation, and speech rate; S303. Determine the key parameters in the fine-tuning training process, including the learning rate and the number of training rounds; at the same time, formulate a training strategy, adopting the transfer learning approach, retaining the parameters related to speech commonality in the pre-trained model, and only adjusting and updating the parameters related to personalized timbre and speech habits. S304. Input the prepared personalized data into the pre-trained model and train it according to the determined training parameters and strategies. During the training process, the pre-trained model continuously adjusts its own parameters based on the input speech data and corresponding text annotations to minimize the difference between the predicted speech and the real user speech. S305. After training is completed, the fine-tuned pre-trained model is evaluated using specialized evaluation metrics and test data to determine the quality and personalization of the speech generated by the evaluation model. If the evaluation results are not ideal, the training parameters are adjusted or training data is increased, and the model is retrained to continuously optimize the model until a satisfactory personalized voice replication effect is achieved.

3. The real-time sound replication method based on edge-cloud fusion according to claim 1, characterized in that, The method for synchronously generating user voice data for preset text from a large model in step S2 includes: During the user registration phase, a large amount of user voice data with preset text is generated simultaneously using a small number of voice samples provided by the user and zero-shot speech cloning technology based on a large cloud model. This data serves as training data for fine-tuning the edge speech synthesis model.

4. The real-time sound replication method based on edge-cloud fusion according to claim 2, characterized in that, The method for evaluating the fine-tuned pre-trained model using specific evaluation metrics and test data in step S305 includes: In terms of subjective evaluation, users or professionals are invited to listen to the synthesized speech and score it based on timbre similarity and naturalness. In terms of objective evaluation, the similarity index between the synthesized speech and the real user's speech in terms of fundamental frequency and formant acoustic features is calculated to quantify the model's adaptation effect. Based on the evaluation results, if a gap is found between the synthesized speech and the user's real timbre, the training parameters are adjusted, or more user speech data is collected for further transfer fine-tuning training. The model is continuously iterated and optimized until it can accurately adapt to the user's personalized timbre and generate highly similar speech content.

5. The real-time sound replication method based on edge-cloud fusion according to claim 1, characterized in that, The transfer fine-tuning training in step S3 is as follows: with the help of the knowledge of the pre-trained model, the model is adapted to personalized needs through specific data and training strategies; by fine-tuning the training data, the model learns personalized information such as the timbre characteristics and speech habits of specific users.

6. The real-time sound replication method based on edge-cloud fusion according to claim 1, characterized in that, The method for real-time timbre replication of a small amount of user voice data in step S1 includes: The system utilizes a large AI model in the cloud to perform real-time timbre replication. When a user registers, they provide a small amount of voice data samples. The large AI model then accurately extracts the key acoustic features of the user's timbre from these limited voice data samples. These key acoustic features include the fundamental frequency and formants. Based on these key acoustic features, the large AI model generates a replicated timbre that is highly similar to the user's original timbre in real time.

7. The real-time sound replication method based on edge-cloud fusion according to claim 1, characterized in that, The speech synthesis method in step S1 includes: After the cloud completes real-time timbre replication and obtains the user's replicated timbre, the AI ​​big model combines it with the text to be synthesized and uses deep learning algorithms to perform semantic understanding and speech prosody analysis on the text; then, it combines the replicated timbre with the text content to generate natural, fluent, and high-quality speech.

8. A real-time sound replication system based on edge-cloud fusion, characterized in that, Performing the real-time sound replication method based on edge-cloud fusion as described in any one of claims 1-7 includes: Cloud-based sample timbre replication and speech synthesis module: used to perform real-time timbre replication and speech synthesis on small amounts of user speech data in the cloud based on a large AI model; User voice data generation module: used to collect user voice samples when registering user voice audio data, and generate user voice data of preset text synchronously by a large model, and use the user voice data as fine-tuning training data for the edge speech synthesis model; The edge speech synthesis model fine-tuning training module is used to perform transfer fine-tuning training on the edge speech synthesis model using the collected user voice data of preset text and the user's registered voice audio data. It automatically adapts to the user's personalized voice, ensures the high-quality output of the edge speech synthesis model, and realizes high-quality personalized voice replication of the edge speech synthesis model. End-side speech synthesis model distribution module: used to distribute the trained end-side speech synthesis model to the user equipment, enabling the end-side speech synthesis model to independently complete speech replication in environments without network or with weak network.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the real-time sound replication method based on edge-cloud fusion as described in any one of claims 1-7.

10. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the real-time sound replication method based on edge-cloud fusion as described in any one of claims 1-7.

Images