Sound replication method and related apparatus

Abstract

The application discloses a sound replication method and related device, and relates to the technical field of speech synthesis. In one aspect, a hybrid expert sound replication model is adopted, each expert sub-model corresponds to only one acoustic category, and all the expert sub-models adopt a lightweight structure. Therefore, the end-side device only needs to activate a single expert sub-model matched with target acoustic category information for reasoning, which greatly reduces the model parameter quantity and computing cost, and thus successfully runs on the low-resource end-side device. In another aspect, the target speaker personalized acoustic features pre-delivered by the cloud and locally stored are directly used in the replication process. The features accurately depict the details of the speaker's timbre, accent, prosody, etc., and make up for the insufficient expression ability of the lightweight model. Therefore, the scheme can generate a highly similar replicated voice to the target speaker on the low-resource end-side device without relying on real-time calculation of the cloud large model, and takes into account the feasibility of end-side deployment and the high fidelity of synthesized sound quality.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a method and apparatus for voice replication. Background Technology

[0002] Voice cloning is a cutting-edge technology in the field of text-to-speech (TTS). It requires only a small amount of speech samples from the target speaker (e.g., 5 to 30 seconds) and combines them with any text input to learn and imitate the speaker's timbre, intonation, pronunciation habits, and other features, thereby generating synthesized speech that is highly similar to the target speaker's voice.

[0003] With the rapid development of artificial intelligence, deep learning-based voice replication technology has been widely used, reaching a level that is difficult to distinguish from the real thing. This effectively makes up for the shortcomings of earlier voice replication technologies, generating more realistic and richer speech textures, with a high degree of similarity to real people in timbre, pitch, and tone. Although the audio quality synthesized by deep learning-based voice replication technology can reach a level that is difficult to distinguish from the real thing, its model parameters are huge, its structure is complex, and its computation is time-consuming. It must rely on expensive high-performance graphics cards (such as professional-grade GPUs) to run. The unit price of such hardware is usually tens of thousands to hundreds of thousands of yuan, resulting in extremely high overall deployment costs. Therefore, it is difficult to deploy on low-resource edge devices (such as mobile phones and IoT terminals).

[0004] Therefore, how to provide a sound replication technology that enables high-fidelity sound replication on low-resource edge devices has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this application provides a sound replication method and related apparatus to achieve high-fidelity sound replication on low-resource end-side devices. The specific solution is as follows:

[0006] The first aspect of this application provides a sound replication method applied to an edge device. The edge device deploys a hybrid expert sound replication model, which includes multiple expert sub-models. Different expert sub-models correspond to different acoustic categories. Each expert sub-model adopts a lightweight model structure, including:

[0007] Retrieve the text to be copied sent by the registered target speaker;

[0008] Acquire target acoustic category information corresponding to the target speaker and personalized acoustic features of the target speaker, which are sent from the cloud device to local storage.

[0009] Using the target expert sub-model in the hybrid expert voice replication model that corresponds to the target acoustic category information, the replicated speech of the target speaker is generated based on the text to be replicated and the personalized acoustic features of the target speaker.

[0010] In one possible implementation, prior to obtaining the text to be copied sent by the registered target speaker, the method further includes:

[0011] Obtain the prompting voice of the target speaker;

[0012] The prompt voice is sent to the cloud device, so that the cloud device can extract the personalized acoustic features of the target speaker from the prompt voice, perform cluster analysis on the personalized acoustic features to obtain the target acoustic category information corresponding to the target speaker, and then send the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker to the end device.

[0013] The system receives and stores the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker sent by the cloud device.

[0014] In one possible implementation, the step of employing the target expert sub-model corresponding to the target acoustic category information in the hybrid expert voice replication model to generate the replicated speech of the target speaker based on the text to be replicated and the personalized acoustic features of the target speaker includes:

[0015] Only activate the target expert sub-model in the hybrid expert voice replication model that corresponds to the target acoustic category information;

[0016] The text to be replicated and the personalized acoustic features of the target speaker are input into the activated target expert sub-model to obtain the target acoustic representation output by the target expert sub-model.

[0017] The target acoustic representation is converted into a replica of the target speaker's speech.

[0018] In one possible implementation, the hybrid expert voice replication model is constructed as follows:

[0019] Acquire training sample data, which includes prompt audio and corresponding text;

[0020] Clustering is performed on representative sample data from the training sample data to obtain multiple acoustic categories;

[0021] The hybrid expert voice replication model is constructed based on the multiple acoustic categories. The hybrid expert voice replication model includes multiple expert sub-models, and different expert sub-models correspond to different acoustic categories.

[0022] In one possible implementation, the hybrid expert voice replication model is trained using a preset large voice replication model as the teacher model and the hybrid expert voice replication model as the student model. During training, the parameters of the large voice replication model are fixed, and the parameters of the hybrid expert voice replication model are updated with the goal of fitting the output of the large voice replication model to the output of the hybrid expert voice replication model and fitting the real audio labels obtained based on the output of the hybrid expert voice replication model.

[0023] In one possible implementation, the training process of the hybrid expert voice replication model specifically includes:

[0024] According to different acoustic categories, the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model are trained sequentially.

[0025] In one possible implementation, the step of sequentially training the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model according to different acoustic categories includes:

[0026] By fixing the parameters of the sound replication model, inputting the prompt audio and text from the training sample data into the sound replication model, the acoustic features of the prompt audio, the acoustic category of the prompt audio, and the acoustic representation output by the sound replication model are obtained.

[0027] The acoustic features and acoustic category of the prompt audio are input into the expert sub-model to be trained in the hybrid expert voice replication model, which corresponds to the acoustic category of the prompt audio, to obtain the acoustic representation output by the expert sub-model to be trained.

[0028] The parameters of the expert sub-model to be trained are updated by fitting the acoustic representation output by the expert sub-model to the sound representation output by the sound replication large model, and by fitting the real audio labels obtained based on the acoustic representation output by the expert sub-model to the sound replication large model, while the parameters of the other expert sub-models remain unchanged.

[0029] A second aspect of this application provides a sound replication device applied to an end-side device. The end-side device deploys a hybrid expert sound replication model, which includes multiple expert sub-models. Different expert sub-models correspond to different acoustic categories. Each expert sub-model adopts a lightweight model structure, including:

[0030] The text to be copied acquisition unit is used to acquire the text to be copied sent by the registered target speaker;

[0031] The target acoustic category information and personalized acoustic feature acquisition unit is used to acquire the target acoustic category information corresponding to the target speaker and the personalized acoustic features of the target speaker, which are sent from the cloud device to the local storage.

[0032] The replicated speech generation unit is used to generate the replicated speech of the target speaker based on the text to be replicated and the personalized acoustic features of the target speaker by employing the target expert sub-model corresponding to the target acoustic category information in the hybrid expert voice replication model.

[0033] In one possible implementation, the device further includes:

[0034] The speaker registration unit is configured to: acquire the prompt voice of the target speaker before acquiring the text to be copied sent by the registered target speaker; send the prompt voice to the cloud device so that the cloud device can extract the personalized acoustic features of the target speaker from the prompt voice, perform cluster analysis on the personalized acoustic features to obtain the target acoustic category information corresponding to the target speaker; and send the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker to the end device; and receive and store the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker sent by the cloud device.

[0035] In one possible implementation, the replicated speech generation unit is specifically used for:

[0036] Only activate the target expert sub-model in the hybrid expert voice replication model that corresponds to the target acoustic category information;

[0037] The text to be replicated and the personalized acoustic features of the target speaker are input into the activated target expert sub-model to obtain the target acoustic representation output by the target expert sub-model.

[0038] The target acoustic representation is converted into a replica of the target speaker's speech.

[0039] In one possible implementation, the device further includes: a hybrid expert voice replication model building unit;

[0040] The hybrid expert voice replication model construction unit is specifically used for:

[0041] Acquire training sample data, which includes prompt audio and corresponding text;

[0042] Clustering is performed on representative sample data from the training sample data to obtain multiple acoustic categories;

[0043] The hybrid expert voice replication model is constructed based on the multiple acoustic categories. The hybrid expert voice replication model includes multiple expert sub-models, and different expert sub-models correspond to different acoustic categories.

[0044] In one possible implementation, the hybrid expert voice replication model is trained using a preset large voice replication model as the teacher model and the hybrid expert voice replication model as the student model. During training, the parameters of the large voice replication model are fixed, and the parameters of the hybrid expert voice replication model are updated with the goal of fitting the output of the large voice replication model to the output of the hybrid expert voice replication model and fitting the real audio labels obtained based on the output of the hybrid expert voice replication model.

[0045] In one possible implementation, the apparatus further includes: a hybrid expert voice replication model training unit, used to sequentially train the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model according to different acoustic categories.

[0046] In one possible implementation, the hybrid expert voice replication model training unit is specifically used for:

[0047] By fixing the parameters of the sound replication model, inputting the prompt audio and text from the training sample data into the sound replication model, the acoustic features of the prompt audio, the acoustic category of the prompt audio, and the acoustic representation output by the sound replication model are obtained.

[0048] The acoustic features and acoustic category of the prompt audio are input into the expert sub-model to be trained in the hybrid expert voice replication model, which corresponds to the acoustic category of the prompt audio, to obtain the acoustic representation output by the expert sub-model to be trained.

[0049] The parameters of the expert sub-model to be trained are updated by fitting the acoustic representation output by the expert sub-model to the sound representation output by the sound replication large model, and by fitting the real audio labels obtained based on the acoustic representation output by the expert sub-model to the sound replication large model, while the parameters of the other expert sub-models remain unchanged.

[0050] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the sound replication method described in the first aspect or any implementation thereof.

[0051] A fourth aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0052] The memory is used to store computer programs;

[0053] The processor is used to execute the computer program so that the electronic device can implement the sound replication method of the first aspect or any implementation thereof.

[0054] The fifth aspect of this application provides a computer-readable storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the sound replication method described in the first aspect or any implementation thereof.

[0055] By employing the aforementioned technical solutions, the voice replication method and related apparatus provided in this application, on the one hand, utilize a hybrid expert voice replication model, where each expert sub-model corresponds to only one acoustic category and all adopt a lightweight structure. This allows the edge device to activate only a single expert sub-model matching the target acoustic category information for inference, significantly reducing the number of model parameters and computational overhead, thus enabling smooth operation on edge devices with limited memory and low computing power. On the other hand, the replication process directly uses the personalized acoustic features of the target speaker, pre-delivered from the cloud and stored locally. These features accurately depict the speaker's timbre, accent, rhythm, and other details, compensating for the limitations of the lightweight model's expressive capabilities. Therefore, this solution does not rely on real-time computation of large cloud models to generate replicated speech highly similar to the target speaker on low-resource edge devices, balancing the feasibility of edge deployment with the high fidelity of synthesized sound quality. Attached Figure Description

[0056] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0057] Figure 1 A schematic flowchart illustrating a sound replication method provided in an embodiment of this application;

[0058] Figure 2 This is a schematic diagram of the structure of a sound replication device provided in an embodiment of this application;

[0059] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0061] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0062] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0063] Voice cloning is a cutting-edge technology in the field of text-to-speech (TTS). It requires only a small amount of speech samples from the target speaker (e.g., 5 to 30 seconds) and combines them with any text input to learn and imitate the speaker's timbre, intonation, pronunciation habits, and other features, thereby generating synthesized speech that is highly similar to the target speaker's voice.

[0064] Early voice replication techniques primarily relied on splicing synthesis and Hidden Markov Models (HMMs). These methods required recording and splicing large amounts of speech segments, resulting in high costs for personalized replication and less-than-expected results. Because splicing speech segments requires massive amounts of data and finding corresponding audio materials is difficult, the technology is very challenging to implement. Furthermore, stuttering and other issues easily occur at the splicing points, leading to low overall quality of the synthesized audio that fails to meet practical application requirements.

[0065] With the rapid development of artificial intelligence, deep learning-based voice replication technology has been widely used, reaching a level that is difficult to distinguish from the real thing. It effectively makes up for the shortcomings of early voice replication technology, and the generated voice texture is more realistic and rich, with a high degree of similarity to the real person in timbre, pitch and tone.

[0066] Currently, deep learning-based voice replication technologies employ massive training data and high-parameter models. Training data typically exceeds 200,000 hours, and the number of speaker types needs to be over 50,000. The model parameters are usually over 10 bytes. During replication, only 5 to 30 seconds of audio from the speaker are recorded and registered. Acoustic features are extracted from the audio, and the model can synthesize highly similar audio using a small amount of acoustic features combined with arbitrary text. It can also achieve a high degree of similarity to the speaker in terms of prosody, timbre, pitch, and accent. The implementation process is as follows: First, an audio compression module compresses and encodes continuous audio into a discrete space to obtain discrete codes for subsequent modeling, reducing the overall computational load. Then, an autoregressive model predicts the discrete codes frame by frame for further modeling. Next, a target speaker timbre prediction structure is used to obtain information such as the target speaker's timbre. Then, a fusion modeling model integrates the encoded information such as text, discrete codes, and timbre to obtain acoustic representation information. Finally, the acoustic representation is processed by a vocoder to obtain the synthesized audio. Among them, most models in the discrete code module currently use the GPT autoregressive prediction scheme, with 36 layers in the model. Each autoregression predicts only one code, which requires a lot of computing resources and takes a long time.

[0067] Although the audio quality synthesized by deep learning-based sound replication technology can reach a level that is difficult to distinguish from the real thing, its model parameters are huge, its structure is complex, and its computation is time-consuming. It must rely on expensive high-performance graphics cards (such as professional-grade GPUs) to run. The unit price of such hardware is usually tens of thousands to hundreds of thousands of yuan, resulting in extremely high overall deployment costs. Therefore, it is difficult to deploy on low-resource edge devices (such as mobile phones and IoT terminals).

[0068] Therefore, how to provide a sound replication technology that enables high-fidelity sound replication on low-resource edge devices has become a technical problem that urgently needs to be solved by those skilled in the art.

[0069] To address the aforementioned issues, this application provides a sound replication method. This sound replication is applied to edge devices. Edge devices refer to terminal devices with relatively limited computing and storage resources, such as smartphones, smart vehicle terminals, smart speakers, and IoT voice modules. These devices typically possess basic voice acquisition and playback capabilities, but struggle to run large models with a massive number of parameters (e.g., billions of parameters).

[0070] The edge device is equipped with a hybrid expert voice replication model, which includes multiple expert sub-models. Different expert sub-models correspond to different acoustic categories, and each expert sub-model adopts a lightweight model structure. This model does not contain a traditional gate network, but directly uses the category information obtained from clustering as a hard switch to activate the corresponding expert sub-model based on the acoustic category of the input speaker.

[0071] In this application, each expert sub-model adopts a lightweight model structure, which can greatly reduce the number of parameters of the hybrid expert voice replication model, making the number of parameters of the hybrid expert voice replication model about one-thousandth of the number of parameters of the large voice replication model, and the overall model's computational cost is only one-thousandth of that of the large voice replication model, enabling deployment on a large number of low-resource edge devices.

[0072] In one possible implementation, the structure of each expert sub-model can be a simple structure based on Conv (Convolution) and LSTM (Long Short-Term Memory). Each expert sub-model can contain N layers. For example, the FFN (Feed-Forward Network) in the Transformer model can be copied N times as an expert sub-model. The specific value of N can be set according to the needs of the scenario, and this application does not impose any restrictions on it.

[0073] Different expert sub-models correspond to different acoustic categories, meaning each sub-model is specifically responsible for replicating a particular sound style. For example, expert sub-model A excels at replicating deep male voices, while expert sub-model B excels at replicating clear female voices. Since each sub-model only needs to learn the distribution of one type of sound, its parameters can be highly simplified, resulting in extremely low computational cost.

[0074] The sound replication method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0075] Reference Figure 1 , Figure 1 This is a flowchart illustrating a sound replication method provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiment of this application, a sound replication method may include the following steps, which are described in detail below.

[0076] S101: Obtain the text to be copied sent by the registered target speaker;

[0077] In this application, a registered target speaker refers to a user whose voice characteristics (personalized acoustic characteristics, acoustic category) have been extracted and stored in the edge device through a cloud registration process. Only registered users have their corresponding acoustic category information and personalized acoustic characteristics in the edge device. The text to be replicated is the content used to provide voice replication; it is the text content that the user wants to read aloud in the voice of the target speaker. It can be a sentence, a paragraph, or text of any length. For example, "The weather is so nice today, let's go for a walk together."

[0078] The edge device receives the text to be replicated from the user through an application interface (such as an input box, speech-to-text, file import, etc.). For example, suppose the target speaker, "Zhang San," has already registered. One day, Zhang San wants to send a voice message to a friend using his own voice (the replicated voice). He opens the voice replication app on his phone, writes in the input box: "Friends, see you at our usual spot at 7 PM," and then clicks the "Generate Replicated Voice" button. At this point, the edge device obtains the text to be replicated: "Friends, see you at our usual spot at 7 PM."

[0079] S102: Obtain the target acoustic category information corresponding to the target speaker and the personalized acoustic characteristics of the target speaker, which are sent from the cloud device to the local storage.

[0080] In this application, the cloud device can be a server cluster deployed with high-performance AI large models (such as a zero-shot speech clone model based on Transformer) and is responsible for handling acoustic feature extraction and cluster analysis during user registration.

[0081] The non-volatile memory of the edge device (such as the flash memory of a mobile phone or a dedicated storage partition) is used to persistently store the relevant data of each registered speaker. In this embodiment, local storage can be stored in the form of key-value pairs: the key is the speaker ID (such as a user account or voiceprint identifier), and the value is "acoustic category information" and "personalized acoustic features".

[0082] The target acoustic category information can be a discrete identifier (such as an integer ID or one-hot encoding) that represents the category to which the speaker belongs in the global acoustic space. This category is obtained by unsupervised clustering (such as K-Means, GMM) of the acoustic features of a large number of speakers, and each category corresponds to a similar timbre style (for example, category 0 corresponds to a deep male voice, category 1 corresponds to a clear female voice, category 2 corresponds to a child's voice, etc.).

[0083] Personalized acoustic features can be a continuous vector (such as a 256-dimensional or 512-dimensional embedding vector) extracted by cloud devices from the target speaker's prompts, representing the speaker's unique timbre, accent, prosody, and other personalized attributes. This feature works in conjunction with category information: the category information determines which expert sub-model to use, while the personalized acoustic features provide fine-grained timbre tuning information.

[0084] Typically, the edge device first identifies the current user's identity (e.g., through a login account, voiceprint verification, or user selection), and then reads the corresponding acoustic category information and personalized acoustic features based on that identity.

[0085] S103: Using the target expert sub-model corresponding to the target acoustic category information in the hybrid expert voice replication model, generate the replicated speech of the target speaker based on the text to be replicated and the personalized acoustic features of the target speaker.

[0086] In this application, the replicated speech of the target speaker conforms to the speaking characteristics of the target speaker, and its timbre, accent, rhythm and other features are highly similar to those of the target speaker, reaching a level that makes it difficult to distinguish between the real and the fake.

[0087] For ease of understanding, this embodiment uses a smartphone as an example. The smartphone has already communicated with the cloud server via a wireless network (such as 4G / 5G or Wi-Fi) and has pre-registered with the target speaker. The registration process is completed by the cloud: the target speaker records a 5-30 second prompt voice message via a mobile app. The cloud server uses a large model to extract the speaker's personalized acoustic features (such as a 256-dimensional speaker embedding vector, which implicitly contains information such as timbre, accent, and prosody). Simultaneously, it uses cluster analysis to categorize the speaker into a specific acoustic category (for example, dividing a large number of speakers into 1000 categories based on timbre similarity, with each category corresponding to a specific voice style). Then, the "acoustic category information" (such as category ID 335) and "personalized acoustic features" (such as a 256-dimensional floating-point vector) are sent to the phone's local storage. Afterward, the phone gains the ability to replicate the speaker's voice offline.

[0088] This solution employs a hybrid expert voice replication model, where each expert sub-model corresponds to only one acoustic category and utilizes a lightweight structure. This allows edge devices to activate only the single expert sub-model matching the target acoustic category for inference, significantly reducing the number of model parameters and computational overhead, thus enabling smooth operation on edge devices with limited memory and computing power. Furthermore, the replication process directly utilizes the personalized acoustic features of the target speaker, pre-delivered from the cloud and stored locally. These features accurately depict the speaker's timbre, accent, rhythm, and other details, compensating for the limitations of the lightweight model's expressive capabilities. Therefore, this solution can generate highly similar replicated speech to the target speaker on low-resource edge devices without relying on real-time computation of large cloud models, balancing the feasibility of edge deployment with high fidelity of synthesized sound quality.

[0089] In one possible implementation, prior to obtaining the text to be copied sent by the registered target speaker, the method further includes:

[0090] S201: Obtain the prompting voice of the target speaker;

[0091] In this application, the prompt voice is a short audio clip used by the target speaker for registration, typically ranging from 5 to 30 seconds in length. The content can be any statement (such as "Hello, I am Zhang San, this is my voice"), or the system can randomly assign a text clip for the user to read aloud. The voice should be as clear as possible, free of background noise, to fully reflect the speaker's timbre, accent, rhythm, and other characteristics. Of course, in this application, the prompt voice can also be a large-volume audio clip; no specific limitations are imposed here.

[0092] The device can record the user's voice in real time via microphone or select an existing recording file from local files. For example, a voice cloning app on a smartphone can pop up a recording interface. After the user clicks the "Start Recording" button, they can read a passage aloud. Once the recording is complete, the app will obtain the audio data of the prompt (such as a 16kHz sampling rate, 16-bit quantized WAV format).

[0093] S202: The prompt voice is sent to the cloud device, so that the cloud device extracts the personalized acoustic features of the target speaker from the prompt voice, performs cluster analysis on the personalized acoustic features to obtain the target acoustic category information corresponding to the target speaker, and sends the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker to the end device.

[0094] Cloud-based devices are deployed with server clusters containing high-performance AI models, possessing powerful computing capabilities and storage capacity. These cloud-based models are typically autoregressive or diffusion models based on Transformers, capable of accurately extracting deep speaker-related features from extremely short audio clips.

[0095] Personalized acoustic features can be a fixed-dimensional embedding vector (e.g., 256-dimensional or 512-dimensional) extracted from the prompt speech by a large cloud-based model through an encoder (such as a speaker verification model or a speech self-supervised model). This vector implicitly contains personalized information such as the speaker's unique timbre spectrum, fundamental frequency distribution, formant patterns, speech rate and rhythm, and pronunciation habits. It can be regarded as the "acoustic fingerprint" of the target speaker.

[0096] The cloud can pre-process the personalized acoustic features of a massive number of speakers using unsupervised clustering (e.g., K-Means, Gaussian mixture models), forming several clusters (e.g., 1000 clusters), each corresponding to an acoustic category. When a new user's personalized acoustic features are received, the cloud calculates the distance (cosine similarity or Euclidean distance) between that feature and the centers of each cluster, assigning it to the nearest cluster, thus obtaining the user's target acoustic category information (i.e., cluster ID). Clustering analysis allows users with similar voice styles to share the same expert sub-model, significantly reducing the number of model branches that need to be maintained on the edge.

[0097] For example, the mobile app compresses Li Si's 8-second prompt voice and uploads it to a cloud server. The cloud server first preprocesses the audio, removing the beginning and end silences, and then inputs it into a speaker encoder based on a WavLM pre-trained model, obtaining a 256-dimensional personalized acoustic feature vector. Subsequently, the cloud server calculates the cosine similarity of this vector with each of 500 pre-trained cluster centers, finding that the highest similarity (0.92) is with cluster center number 237. Therefore, Li Si's target acoustic category information is determined to be 237.

[0098] This step fully leverages the powerful computing capabilities and large modeling power of the cloud to accurately decouple the speaker's personalized features from extremely short audio clips. It then uses clustering to compress a massive number of voice styles into a limited number of categories, providing the necessary conditions for lightweight models on the device. The entire process is completed within seconds, with virtually no impact on the user.

[0099] S203: Receive and store the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker sent by the cloud device.

[0100] The cloud-based device can send the processing results (personalized acoustic feature vectors and acoustic category information) to the edge device via the network. The amount of data sent is extremely small (e.g., 256-dimensional floating-point numbers occupy 1KB, and the category ID occupies 4 bytes), requiring very little network bandwidth. The edge device saves the received data to local non-volatile storage (such as the phone's internal flash memory or SD card). Storage is typically linked to the user account or device unique identifier, forming key-value pairs. For example, using the user ID "LiSi_001" as the key, "Acoustic Category=237" and "Personalized Feature=[0.23, -0.45, …, 0.67] (256 dimensions)" are saved. The data remains available even if the device goes offline or restarts.

[0101] For example, after calculating Li Si's personalized acoustic features (256-dimensional vector) and acoustic category 237 in the cloud, these two pieces of data are packaged into a JSON object: {"user_id":"LiSi_001", "class_id":237, "speaker_embedding":[0.23,-0.45,...,0.67]}. This data is returned to the mobile app via an HTTPS response. Upon receiving this data, the app stores it in the "Registered User Table" in the phone's local database. Simultaneously, the app interface displays a message: "Registration successful. You can now use the offline voice cloning function."

[0102] This step requires only a one-time network interaction; all subsequent replication operations are completed offline on the device, no longer relying on the network. Because the amount of data stored is extremely small (approximately 1KB per user), a single mobile phone can easily store the acoustic information of hundreds or thousands of registered users. Simultaneously, the cloud does not require separate deployment or fine-tuning of models for each user, significantly reducing cloud operation and maintenance costs.

[0103] This embodiment introduces the speaker registration process, illustrating how the edge device collaborates with the cloud to complete the registration of the target speaker, acquiring and storing the personalized acoustic features and acoustic category information required for subsequent voice replication. Through the above registration process, the edge device only needs a simple network interaction to permanently obtain the target speaker's personalized acoustic features and acoustic category information. This information is the key input for subsequent lightweight voice replication. Compared to traditional solutions, this solution concentrates complex calculations in a one-time processing phase during registration. The registration backend works completely offline, protecting user privacy (acoustic features can be stored in anonymized form) while achieving high-fidelity voice replication on low-resource devices. Users only need to spend a few seconds recording a voice once to use their replicated voice an unlimited number of times at any time and on any text.

[0104] In one possible implementation, the step of employing the target expert sub-model corresponding to the target acoustic category information in the hybrid expert voice replication model to generate the replicated speech of the target speaker based on the text to be replicated and the personalized acoustic features of the target speaker includes:

[0105] S301: Only activate the target expert sub-model in the hybrid expert voice replication model that corresponds to the target acoustic category information;

[0106] In the hybrid expert voice replication model, multiple expert sub-models coexist in memory, but only one is selected to run during inference, while the others remain dormant (not participating in computation). This "sparse activation" mechanism significantly reduces the actual computational load. Since only one lightweight expert sub-model is activated at a time, the computational load on the edge device is comparable to that of a single model, regardless of the total number of models. This allows for smooth operation even on low-end devices with only a few hundred MB of memory and limited computing power.

[0107] S302: Input the text to be replicated and the personalized acoustic features of the target speaker into the activated target expert sub-model to obtain the target acoustic representation output by the target expert sub-model;

[0108] The target acoustic representation is an intermediate feature output by the expert sub-model, typically a Mel-spectrogram or linear spectrum with dimensions of (number of time frames × number of frequency bands). This representation describes the energy distribution of the synthesized speech at various time points and is a compact representation prior to waveform generation.

[0109] S303: Convert the target acoustic representation into a replica of the target speaker's speech.

[0110] In this application, the process of converting the target acoustic representation into replicated speech of the target speaker can be implemented based on a vocoder. A vocoder is a neural network model specifically designed to learn the mapping from acoustic features to waveforms; common examples include HiFi-GAN, MelGAN, and WaveNet. Vocoders are characterized by high speed and high sound quality, making them suitable for edge deployment. The replicated speech is the final generated audio waveform, which can be directly played through a speaker or saved as a file.

[0111] This embodiment focuses on how the edge device completes high-fidelity sound replication through three sub-steps: activating the target expert sub-model, generating acoustic representations, and vocoder conversion. Each inference iteration activates only a single lightweight expert sub-model corresponding to the target acoustic category, avoiding computation of all sub-models. This results in memory consumption being only one percent of that of a large cloud-based model, and computational load controlled within a few hundred MFLOPS, enabling real-time operation on ordinary CPUs or low-end NPUs. Simultaneously, the high-fidelity acoustic representations output by the expert sub-models, after conversion by a neural vocoder, achieve a timbre similarity and naturalness of the replicated speech approaching the level of the large cloud-based model (MOS score difference less than 0.12), realizing high-quality sound replication on low-resource devices. Furthermore, this process is not dependent on a specific vocoder type, exhibiting good hardware adaptability and versatility.

[0112] In one possible implementation, the hybrid expert voice replication model is constructed as follows:

[0113] S401: Obtain training sample data, the training sample data including prompt audio and corresponding text;

[0114] Training sample data refers to a large-scale speech dataset used to train small edge models. This dataset needs to contain a massive amount of cue audio (e.g., more than 200,000 hours of speech) and corresponding text annotations. The cue audio comes from a large number of different speakers (e.g., more than 50,000), covering various ages, genders, accents, and speech styles.

[0115] The prompt audio is a speech segment from the training samples, which can be of any content and is typically 5-30 seconds long, consistent with the audio format provided during user registration. The corresponding text is the text content obtained by automatically transcribing the prompt audio using existing speech recognition models (such as Whisper and Paraformer), used to establish the alignment between text and speech. Simultaneously, prosodic information (such as phoneme duration and fundamental frequency curves) can be extracted using text analysis models, providing richer conditions for subsequent modeling.

[0116] Large-scale and diverse training data provides rich material for the model to learn common speech features and timbre differences, and is the basis for subsequent clustering and expert sub-model training.

[0117] S402: Clustering is performed on representative sample data in the training sample data to obtain multiple acoustic categories;

[0118] In this application, representative sample data is a subset selected from a massive training sample according to certain rules to improve clustering efficiency and effectiveness. Selection criteria may include acoustic attributes such as age, gender, pitch range, and speech rate, or a random sampling plus stratification strategy may be used to ensure that each speaker category has sufficient representativeness.

[0119] Clustering is an unsupervised learning method that divides high-dimensional acoustic feature vectors into several clusters, making the features of samples within the same cluster similar, while the features of samples in different clusters are significantly different. Commonly used algorithms include K-Means, Gaussian Mixture Models (GMM), and spectral clustering. In this application, a pre-trained acoustic encoder (such as WavLM or ECAPA-TDNN) is first used to convert each cue audio segment into a fixed-dimensional embedding vector (such as 256-dimensional or 512-dimensional). This encoded information condenses the speaker's timbre, vocal tract characteristics, and other deep acoustic features. Then, the encoded vectors of all training samples are input into the clustering algorithm to iteratively optimize the cluster centers until convergence. Each sample is assigned a cluster ID.

[0120] S403: Construct the hybrid expert voice replication model based on the multiple acoustic categories. The hybrid expert voice replication model includes multiple expert sub-models, and different expert sub-models correspond to different acoustic categories.

[0121] In this application, the number of expert sub-models is equal to the total number of acoustic categories obtained by clustering.

[0122] In one possible implementation, the hybrid expert voice replication model is trained using a preset large voice replication model as the teacher model and the hybrid expert voice replication model as the student model. During training, the parameters of the large voice replication model are fixed, and the parameters of the hybrid expert voice replication model are updated with the goal of fitting the output of the large voice replication model to the output of the hybrid expert voice replication model and fitting the real audio labels obtained based on the output of the hybrid expert voice replication model.

[0123] In this application, the preset voice replication model can be an autoregressive or diffusion-based large-scale speech generation model with high-fidelity speech synthesis capabilities, which can generate speech highly similar to the speaker in the prompt audio based on the input text and prompt audio. When updating the parameters of the hybrid expert voice replication model by fitting the output of the hybrid expert voice replication model to the output of the large-scale voice replication model, and by fitting the audio to the real audio labels obtained based on the output of the hybrid expert voice replication model, the fitting results of the two can be calculated according to a certain proportional relationship, such as by weighted summation according to preset weights.

[0124] The training process of the hybrid expert voice replication model specifically includes:

[0125] According to different acoustic categories, the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model are trained sequentially.

[0126] In this application, the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model are trained sequentially according to different acoustic categories, which enables the model to better learn the effects of different timbres.

[0127] Specifically, the step of training the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model in sequence according to different acoustic categories includes:

[0128] With the parameters of the fixed sound replication model fixed, the prompt audio and text from the training sample data are input into the sound replication model to obtain the acoustic features of the prompt audio, the acoustic category of the prompt audio, and the acoustic representation output by the sound replication model.

[0129] The acoustic features and acoustic category of the prompt audio are input into the expert sub-model to be trained in the hybrid expert voice replication model, which corresponds to the acoustic category of the prompt audio, to obtain the acoustic representation output by the expert sub-model to be trained.

[0130] The parameters of the expert sub-model to be trained are updated by fitting the acoustic representation output by the expert sub-model to the sound representation output by the sound replication large model, and by fitting the real audio labels obtained based on the acoustic representation output by the expert sub-model to the sound replication large model, while the parameters of the other expert sub-models remain unchanged.

[0131] This training method enables the edge-side hybrid expert voice replication model to achieve near-perfect timbre similarity and naturalness (MOS score difference less than 0.12) in replicated speech, with only one percent of the parameters and a computational cost of a few hundred MFLOPS compared to the larger model. Training by category avoids gradient conflicts, and each expert sub-model focuses on a specific acoustic style, resulting in more accurate output. Soft labels introduced by distillation supplement details that are difficult for the smaller model to learn, while hard labels ensure the authenticity of the speech. The final model can run in real-time on ordinary CPUs or low-end NPUs, achieving high-fidelity voice replication on low-resource devices.

[0132] The above describes a sound replication method provided by the embodiments of this application. The following will describe the apparatus for performing the above sound replication method.

[0133] Please see Figure 2 , Figure 2 This is a schematic diagram of a sound replication device provided in an embodiment of this application. Figure 2 As shown, this sound replication device is applied to an end-side device. The end-side device deploys a hybrid expert sound replication model, which includes multiple expert sub-models. Different expert sub-models correspond to different acoustic categories. Each expert sub-model adopts a lightweight model structure, including:

[0134] The text to be copied acquisition unit 11 is used to acquire the text to be copied sent by the registered target speaker;

[0135] The target acoustic category information and personalized acoustic feature acquisition unit 12 is used to acquire the target acoustic category information corresponding to the target speaker and the personalized acoustic features of the target speaker, which are sent from the cloud device to the local storage.

[0136] The replicated speech generation unit 13 is used to generate the replicated speech of the target speaker based on the text to be replicated and the personalized acoustic features of the target speaker by employing the target expert sub-model corresponding to the target acoustic category information in the hybrid expert voice replication model.

[0137] In one possible implementation, the device further includes:

[0138] The speaker registration unit is configured to: acquire the prompt voice of the target speaker before acquiring the text to be copied sent by the registered target speaker; send the prompt voice to the cloud device so that the cloud device can extract the personalized acoustic features of the target speaker from the prompt voice, perform cluster analysis on the personalized acoustic features to obtain the target acoustic category information corresponding to the target speaker; and send the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker to the end device; and receive and store the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker sent by the cloud device.

[0139] In one possible implementation, the replicated speech generation unit is specifically used for:

[0140] Only activate the target expert sub-model in the hybrid expert voice replication model that corresponds to the target acoustic category information;

[0141] The text to be replicated and the personalized acoustic features of the target speaker are input into the activated target expert sub-model to obtain the target acoustic representation output by the target expert sub-model.

[0142] The target acoustic representation is converted into a replica of the target speaker's speech.

[0143] In one possible implementation, the device further includes: a hybrid expert voice replication model building unit;

[0144] The hybrid expert voice replication model construction unit is specifically used for:

[0145] Acquire training sample data, which includes prompt audio and corresponding text;

[0146] Clustering is performed on representative sample data from the training sample data to obtain multiple acoustic categories;

[0147] The hybrid expert voice replication model is constructed based on the multiple acoustic categories. The hybrid expert voice replication model includes multiple expert sub-models, and different expert sub-models correspond to different acoustic categories.

[0148] In one possible implementation, the hybrid expert voice replication model is trained using a preset large voice replication model as the teacher model and the hybrid expert voice replication model as the student model. During training, the parameters of the large voice replication model are fixed, and the parameters of the hybrid expert voice replication model are updated with the goal of fitting the output of the large voice replication model to the output of the hybrid expert voice replication model and fitting the real audio labels obtained based on the output of the hybrid expert voice replication model.

[0149] In one possible implementation, the apparatus further includes: a hybrid expert voice replication model training unit, used to sequentially train the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model according to different acoustic categories.

[0150] In one possible implementation, the hybrid expert voice replication model training unit is specifically used for:

[0151] By fixing the parameters of the sound replication model, inputting the prompt audio and text from the training sample data into the sound replication model, the acoustic features of the prompt audio, the acoustic category of the prompt audio, and the acoustic representation output by the sound replication model are obtained.

[0152] The acoustic features and acoustic category of the prompt audio are input into the expert sub-model to be trained in the hybrid expert voice replication model, which corresponds to the acoustic category of the prompt audio, to obtain the acoustic representation output by the expert sub-model to be trained.

[0153] The parameters of the expert sub-model to be trained are updated by fitting the acoustic representation output by the expert sub-model to the sound representation output by the sound replication large model, and by fitting the real audio labels obtained based on the acoustic representation output by the expert sub-model to the sound replication large model, while the parameters of the other expert sub-models remain unchanged.

[0154] Each unit in the aforementioned sound replication device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each unit.

[0155] This application also provides an electronic device in its embodiments. (See reference...) Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0156] like Figure 3 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0157] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0158] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the sound replication methods provided in this application.

[0159] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the sound replication methods provided in this application.

[0160] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0162] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0163] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for sound replication, characterized in that, This is applied to an edge device, which deploys a hybrid expert voice replication model. The hybrid expert voice replication model includes multiple expert sub-models, each corresponding to a different acoustic category. Each expert sub-model employs a lightweight model structure, including: Retrieve the text to be copied sent by the registered target speaker; Acquire target acoustic category information corresponding to the target speaker and personalized acoustic features of the target speaker, which are sent from the cloud device to local storage. Only the target expert sub-model in the hybrid expert voice replication model that corresponds to the target acoustic category information is activated. The hybrid expert voice replication model does not contain a traditional gating network. Instead, it is based on a fixed mapping relationship between acoustic categories and expert sub-models determined in advance through clustering. The category information obtained by clustering is directly used as a hard switch to activate the corresponding expert sub-model according to the acoustic category of the input speaker. The text to be replicated and the personalized acoustic features of the target speaker are input into the activated target expert sub-model to obtain the target acoustic representation output by the target expert sub-model. The target acoustic representation is converted into a replica of the target speaker's speech.

2. The method according to claim 1, characterized in that, Before obtaining the text to be copied sent by the registered target speaker, the method further includes: Obtain the prompting voice of the target speaker; The prompt voice is sent to the cloud device, so that the cloud device can extract the personalized acoustic features of the target speaker from the prompt voice, perform cluster analysis on the personalized acoustic features to obtain the target acoustic category information corresponding to the target speaker, and then send the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker to the end device. The system receives and stores the personalized acoustic features of the target speaker and the target acoustic category information corresponding to the target speaker sent by the cloud device.

3. The method according to claim 1, characterized in that, The hybrid expert voice replication model is constructed as follows: Acquire training sample data, which includes prompt audio and corresponding text; Clustering is performed on representative sample data from the training sample data to obtain multiple acoustic categories; The hybrid expert voice replication model is constructed based on the multiple acoustic categories. The hybrid expert voice replication model includes multiple expert sub-models, and different expert sub-models correspond to different acoustic categories.

4. The method according to any one of claims 1 to 3, characterized in that, The hybrid expert voice replication model is trained using a pre-set large voice replication model as the teacher model and the hybrid expert voice replication model as the student model. During training, the parameters of the large voice replication model are fixed, and the parameters of the hybrid expert voice replication model are updated with the goal of fitting the output of the large voice replication model to the output of the hybrid expert voice replication model and fitting the real audio labels obtained based on the output of the hybrid expert voice replication model.

5. The method according to claim 4, characterized in that, The training process of the hybrid expert voice replication model specifically includes: According to different acoustic categories, the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model are trained sequentially.

6. The method according to claim 5, characterized in that, The step of training the expert sub-models corresponding to each acoustic category in the hybrid expert voice replication model in sequence, according to different acoustic categories, includes: By fixing the parameters of the sound replication model, inputting the prompt audio and text from the training sample data into the sound replication model, the acoustic features of the prompt audio, the acoustic category of the prompt audio, and the acoustic representation output by the sound replication model are obtained. The acoustic features and acoustic category of the prompt audio are input into the expert sub-model to be trained in the hybrid expert voice replication model, which corresponds to the acoustic category of the prompt audio, to obtain the acoustic representation output by the expert sub-model to be trained. The parameters of the expert sub-model to be trained are updated by fitting the acoustic representation output by the expert sub-model to the sound representation output by the sound replication large model, and by fitting the real audio labels obtained based on the acoustic representation output by the expert sub-model to the sound replication large model, while the parameters of the other expert sub-models remain unchanged.

7. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the sound replication method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the sound replication method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the sound replication method as described in any one of claims 1 to 6.

Images