End side voice model deployment method and device, equipment and storage medium

Abstract

The invention relates to the technical field of end-side model deployment, in particular to an end-side voice model deployment method and device, equipment and a storage medium. Comprising the steps of recording reference audio at an end side; extracting a reference semantic token and an embedded vector of the reference audio; obtaining a training text set, integrating and inputting the training text code, the embedded vector and the reference semantic token corresponding to each training text in the training text set into a preset language model, and outputting a comprehensive token sequence corresponding to the training text; performing Mel spectrum conversion on each comprehensive token sequence to obtain a first spectrum representation corresponding to the comprehensive token sequence; generating an audio signal corresponding to the training text according to each first spectrum representation, and integrating to obtain a training data set; inputting the training data set into a to-be-trained model, and training to obtain a lightweight voice model which only retains timbre modeling parameters; and updating the lightweight voice model to the end side to complete end side deployment. According to the invention, the voice synthesis model deployment of the end-side equipment can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of end-side model deployment, and particularly to an end-side speech model deployment method, device, equipment and storage medium. BACKGROUND

[0002] A speech synthesis model generates speech output conforming to human pronunciation habits by modeling acoustic, prosodic and linguistic features of speech signals, and has been widely used in smart home control, mobile terminal voice assistants, online education audio content generation, and reading aids for visually impaired people, and has become a key technology for human-computer interaction.

[0003] Currently, most speech synthesis tasks rely on cloud server processing. After the device end collects the voice command, it is uploaded to the cloud server, and the large model operation generates voice data and returns. Although this method can take advantage of the powerful computing power and storage resources of the cloud server to achieve high-quality services, it has significant limitations: first, real-time performance is limited by network transmission and server response, and high-concurrency scenarios are prone to delays; second, there is a risk of privacy leakage when uploading user voice data, and the security of sensitive information is a major concern; third, the dependence on network connection makes the device unable to be used in a network-free or weak network environment, reducing usability.

[0004] To alleviate the bottleneck of the cloud server, some solutions attempt to deploy speech models on the end side, but are limited by the computing and storage resources of the end-side device, making it difficult to run large models; and personalized speech synthesis requires a large amount of data support, which is difficult to achieve under the condition of small data sets on the end side, and cannot meet the personalized needs. Therefore, the contradiction between the complexity of the speech synthesis model and the resources of the end-side device is a technical problem that needs to be solved for deploying personalized speech synthesis models on the end side. SUMMARY

[0005] To overcome the deficiencies of the prior art, the present application provides an end-side speech model deployment method, device, equipment and storage medium, which is based on a small data set task migration method to deploy a speech synthesis model on an end-side device.

[0006] The technical solution adopted by the present application to solve its technical problems is:

[0007] In a first aspect, the present application provides an end-side speech model deployment method, which is applied to a cloud server, the cloud server is in communication connection with an end-side device, and the method comprises:

[0008] obtaining a reference audio sent by the end-side device;

[0009] extracting a reference semantic token and an embedding vector of the reference audio; the reference semantic token is a discrete label with semantic information in the reference audio, and the embedding vector is used to represent the vocal characteristics of a target user;

[0010] obtaining a training text set, integrating a training text code corresponding to each training text in the training text set, the embedding vector and the reference semantic token into a preset language model, and outputting a comprehensive token sequence of the corresponding training text from the preset language model according to the embedding vector and the reference semantic token; the comprehensive token sequence comprises a timbre feature of each semantic unit in the current training text;

[0011] performing mel-spectrum conversion on each comprehensive token sequence to obtain a first spectral representation of the corresponding comprehensive token sequence;

[0012] generating an audio signal of each training text according to each first spectral representation, and integrating all the audio signals to obtain a training data set;

[0013] inputting the training data set into a to-be-trained model to obtain a lightweight speech model through training; the lightweight speech model is a speech model that only retains timbre modeling parameters;

[0014] sending the lightweight speech model to the terminal device to enable the terminal device to complete terminal-side speech model deployment based on the lightweight speech model.

[0015] Optionally, in the step of extracting the reference semantic token and the embedding vector of the reference audio, the step of obtaining the reference semantic token comprises:

[0016] performing mel-spectrum extraction on the reference audio to obtain a second spectral representation, and inputting the second spectral representation into a preset semantic token extraction module; the semantic token extraction module comprises a start coding unit, a feature coding unit and a decoding unit;

[0017] inputting the second spectral representation into the start coding unit, wherein the start coding unit comprises a start position encoder and a start feature encoder, and the start position encoder is used to perform timing coding on the second spectral representation to generate a context-aware representation;

[0018] inputting the context-aware representation into the start feature encoder to perform coding to obtain a first-level coding, and performing vector quantization processing on the first-level coding to generate an initial coding sequence;

[0019] performing coding embedding on the initial coding sequence, inputting the initial coding sequence after completing coding embedding into the feature coding unit, and the feature coding unit comprises at least one feature encoder; all the feature encoders in the feature coding unit are sequentially used to perform coding on the initial coding sequence to obtain a final coding sequence;

[0020] inputting the final encoding sequence to the decoding unit to decode the final encoding sequence by the decoding unit to obtain the reference semantic token.

[0021] Optionally, the feature encoding unit comprises a terminal encoding unit, and the terminal encoding unit comprises a terminal position encoder and a terminal feature encoder.

[0022] The step of sequentially encoding the initial encoding sequence through all the feature encoders in the feature encoding unit to obtain a final encoding sequence comprises:

[0023] The step of determining an input encoding sequence and inputting the input encoding sequence to the terminal encoding unit comprises, when the feature encoding unit comprises an intermediate encoding subunit, sequentially inputting the initial encoding sequence to all the intermediate feature encoders in the intermediate encoding subunit to perform encoding, generating an intermediate encoding sequence, and taking the intermediate encoding sequence as the input encoding sequence; the intermediate encoding subunit comprises at least one feature encoder.

[0024] When the feature encoding unit does not comprise an intermediate encoding subunit, taking the initial encoding sequence as the input encoding sequence.

[0025] The step of performing final encoding on the input encoding sequence by the terminal encoding unit comprises performing context-increasing encoding on the input encoding sequence by the terminal position encoder, inputting the enhanced sequence to the terminal feature encoder to perform encoding, and obtaining the final encoding sequence.

[0026] Optionally, the step of inputting each training text in the training text set, the embedding vector and the reference semantic token into a preset language model comprises:

[0027] Initializing a to-be-constructed sequence and adding a start identifier as a sequence start bit of the to-be-constructed sequence.

[0028] Adding the embedding vector, training text encoding of a current training text and the reference semantic token to the to-be-constructed sequence in a preset order, and adding a semantic separation identifier between the training text encoding and the reference semantic token.

[0029] Adding a termination identifier to a sequence end bit of the to-be-constructed sequence to complete input sequence construction, and inputting the input sequence to the preset language model.

[0030] Optionally, the step of performing Mel-frequency spectrum conversion on each comprehensive token sequence to obtain a first frequency spectrum representation of the corresponding comprehensive token sequence comprises:

[0031] input the comprehensive token sequence and the reference semantic token into a pre-trained spectrum generation model, the spectrum generation model learns a mel spectrum distribution through pre-training;

[0032] sample generation of the mel spectrum distribution to generate a spectrum feature corresponding to the comprehensive token sequence in the spectrum generation model conditioned on the reference semantic token;

[0033] output a first spectrum representation corresponding to the comprehensive token sequence according to the spectrum feature.

[0034] Optionally, the step of inputting the training data set into a to-be-trained model to train a lightweight speech model comprises:

[0035] annotating a real timbre label for the audio signal in the training data set;

[0036] inputting the annotated audio signal into an initial speech model to be trained, and outputting a predicted timbre label of the current audio signal by the initial speech model;

[0037] calculating a model loss value according to the predicted timbre and the corresponding real timbre label, and updating parameters of the initial speech model based on the model loss value, returning the step of annotating a real timbre label for the audio signal in the training data set iteratively training until a preset iteration condition is met;

[0038] performing model parameter extraction on the trained initial speech model, screening to obtain the timbre modeling parameter, and constructing the lightweight speech model based on the timbre modeling parameter.

[0039] In a second aspect, the present application provides an end-side speech model deployment method, the method is applied to an end-side device, the end-side device is in communication connection with a cloud-side server, and the method comprises:

[0040] sending a pre-recorded reference audio to the cloud-side server;

[0041] obtaining a lightweight speech model fed back by the cloud-side server and performing end-side speech model deployment; the lightweight speech model is a speech model retaining only timbre modeling parameters, which is trained by the cloud-side server according to the reference audio.

[0042] In a third aspect, the present application provides an end-side speech model deployment device, comprising:

[0043] a reference audio acquisition module configured to acquire a reference audio sent by the end-side device;

[0044] an audio information extraction module configured to extract a reference semantic token and an embedding vector of the reference audio; the reference semantic token is a discrete label with semantic information in the reference audio, and the embedding vector is used to represent a target user voice color feature;

[0045] a token sequence generation module configured to obtain a training text set, integrate a training text code corresponding to each training text in the training text set, the embedding vector, and the reference semantic token into a preset language model, and output a comprehensive token sequence of the corresponding training text by the preset language model according to the embedding vector and the reference semantic token; the comprehensive token sequence includes a voice color feature of each semantic unit in the current training text;

[0046] a spectrum representation conversion module configured to perform mel spectrum conversion on each comprehensive token sequence to obtain a first spectrum representation of the corresponding comprehensive token sequence;

[0047] a training data acquisition module configured to generate an audio signal of the corresponding training text according to each first spectrum representation, and integrate all the audio signals to obtain a training data set;

[0048] a light model generation module configured to input the training data set into a to-be-trained model to obtain a light voice model by training; the light voice model is a voice model that only retains voice color modeling parameters;

[0049] an end-side model deployment module configured to update the light voice model to an end-side device to complete end-side voice model deployment.

[0050] In a fourth aspect, the present application provides an electronic device, comprising:

[0051] one or more processors;

[0052] one or more memories;

[0053] and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the above method.

[0054] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the storage medium stores a program or instructions, and when the program or instructions are executed, the above method is implemented.

[0055] The beneficial effects of the present application are: through the collaborative task migration of the cloud server and the end side, the problem of deploying a personalized speech synthesis model on the end side is solved. Specifically, first, the user records a short (for example, 3-10 seconds) reference audio on the end side and uploads it to the cloud server, extracts the reference semantic token and embedding vector in the reference audio recorded on the end side from the cloud server, and then generates a plurality of comprehensive token sequences of fusion timbre features in combination with the training text set (i.e. the text of the high-frequency words corresponding to the application scenario of the current speech synthesis model). After the Mel frequency spectrum conversion, the comprehensive token sequences construct a training data set, and finally a lightweight speech model is trained which only retains the timbre modeling parameters. Finally, the obtained lightweight speech model is updated to the end side to form an exclusive timbre model on the end side. Subsequent users can be independent of the cloud server and directly use the exclusive timbre model on the end side for real-time speech synthesis.

[0056] In the deployment of the speech synthesis model on the end side described above, on the one hand, the cloud server handles complex tasks, including extracting reference semantic tokens and embedding vectors, training a lightweight speech model, and updating the lightweight speech model to the end side, so that the end side device does not need to directly run a large model, thereby overcoming the difficulty of limited computing resources and storage capacity of the end side device, and also avoiding the real-time operation pressure on the end side device; on the other hand, the present application provides necessary data support for personalized speech synthesis by combining the reference audio and the training text set, so that personalized speech synthesis can be realized under the condition of a small data set. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a flowchart of the end side speech model deployment method provided by the embodiments of the present application;

[0058] Figure 2 is a model diagram of the semantic token extraction module in the end side speech model deployment method provided by the embodiments of the present application;

[0059] Figure 3 is a virtual structure diagram of the end side speech model deployment device provided by the present application;

[0060] Figure 4 is a structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0061] The present application will be further described below in conjunction with the drawings and embodiments.

[0062] The concept, specific structure and generated technical effects of the present application will be described clearly and completely in combination with the embodiments and drawings to fully understand the purpose, features and effects of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments, and based on the embodiments of the present application, other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. In addition, all the coupling / connection relationships involved in the patent do not mean that the components are directly connected, but that a better coupling structure can be composed by adding or reducing coupling accessories according to the specific implementation. The technical features in the creation of the present application can be interactively combined without mutual contradiction and conflict.

[0063] The speech synthesis model is an artificial intelligence model that converts text information into natural speech output, widely used in smart home, mobile terminal voice assistants, online education, and assistive reading for visually impaired people. However, the current speech synthesis task relies on cloud server processing, which has problems such as network delay, privacy leakage risk, and dependence on network connection, making it difficult to meet real-time and privacy needs. The cloud server refers to a remote server or server group, and the task or processing process performed on the cloud server is cloud-side.

[0064] To solve the above problems, the industry has begun to explore the deployment of speech synthesis models on the side. The side refers to the task or processing process performed on the user's local device (such as a smartphone, smart home device, etc.), and the side deployment can effectively reduce latency, improve real-time performance, reduce data transmission, enhance user privacy protection, and support personalized voice services. However, the computing resources and storage capacity of the side device are limited, making it difficult to directly run large speech synthesis models, and it is difficult to achieve personalized speech synthesis under small data set conditions, which is the main difficulty of side deployment. Based on this, the present application proposes a side speech model deployment method to overcome the difficulties of side model deployment.

[0065] Reference Figure 1 , Figure 1 is a flowchart of the side speech model deployment method provided by the embodiments of the present application, which includes a plurality of key steps involved in the side model deployment method provided by the present application, and each key step will be described in detail as follows:

[0066] In step S1, the reference audio sent by the side device is obtained.

[0067] The reference audio is recorded by a user through an end side, i.e., a target human voice sample for voice synthesis recorded by a user on a local device (i.e., an end side device such as a mobile phone, a smart speaker, or a terminal device), usually a short-length audio (such as 10-30 seconds), containing the individualized acoustic characteristics of the speaker such as tone, intonation, and speech rate. Therefore, before step S1, the end side voice model deployment method further includes an end side task, which first sends the pre-recorded reference audio to the cloud side server in step S1, so that the cloud server receives the reference audio.

[0068] Specifically, after the user records a reference audio through an end side device (such as a mobile phone APP), the device uploads the audio data to the cloud server through the network. After the cloud server receives the audio, a preset cloud side task flow is started, and the cloud side task includes multiple key steps such as steps S2 to S7. In addition, during the process of uploading the reference audio to the cloud server by the end side device, compression processing can be performed on the reference audio to reduce the transmission bandwidth.

[0069] More specifically, due to the limited computing power, memory, and battery capacity of the end side device (such as a mobile phone or an embedded device), it is difficult to independently complete complex acoustic feature extraction and model training tasks, while the cloud server can quickly process data through distributed computing and rely on large-scale general models to improve the accuracy and naturalness of personalized voice synthesis. Therefore, this step migrates the heavy task to the cloud server for execution, and uses the computing power advantage of the cloud server to solve the problem of limited resources on the end side. In addition, uploading only the reference audio to the cloud server can also avoid uploading sensitive voice data that may be involved in practical human-computer interaction applications to the cloud server, reducing the risk of privacy leakage.

[0070] Further, the end side device in this embodiment can also be an embedded device based on a memory-computing integrated chip. For a device based on a memory-computing integrated chip, the storage and computing functions are integrated into the same chip, so that data can be directly computed within the chip without the need to read data from the memory, which can reduce data transmission costs, increase parallelism, reduce delay, save energy, and have higher data processing capacity. However, this makes higher requirements for the size of the model for the embedded device or system based on the memory-computing integrated chip, and therefore, the model of the end side device needs to be updated on the cloud side server for execution, so that the new model sent by the cloud side server is deployed on the memory-computing integrated chip.

[0071] In step S2, the reference semantic token and embedding vector of the reference audio are extracted.

[0072] The reference semantic token is a discrete token with semantic information in the reference audio, specifically a discrete symbol sequence carrying semantic information of speech content extracted from the reference audio. The symbols are digital representations of vocabulary, syntax, or pronunciation logic in the speech, such as converting "hello, world" into discrete tokens containing phonemes, word vectors, or semantic units.

[0073] In the embodiments of the present application, the reference semantic token is extracted from the reference audio by S3 Tokenizer, which is a deep learning-based speech semantic feature extraction tool mainly used to extract discrete tokens with semantic information from speech signals. It can convert continuous speech waveform into discrete semantic token sequence, filter non-semantic information (such as background noise, pause), and retain tokens related to vocabulary meaning.

[0074] The embedding vector is used to represent the target user's voice characteristics, which is learned and generated from the reference audio by a pre-trained voiceprint model (i.e. acoustic feature extraction model), and is specifically used to quantify and represent the unique voice characteristics of the target user, such as vocal cord vibration frequency, resonance cavity characteristics, and other personalized acoustic attributes.

[0075] Specifically, the cloud server first pre-processes the reference audio (such as noise reduction, frame division), and then performs the embedding vector extraction and reference semantic token extraction in parallel. In the semantic token extraction step, the S3 Tokenizer tool is used to analyze the audio signal, converting continuous speech waveform into discrete semantic token sequence. Non-semantic information (such as breathing sound, pause) is automatically filtered out, and only tokens related to vocabulary meaning are retained. In the embedding vector extraction step, the acoustic features of the audio (such as mel spectrum, fundamental frequency curve) are encoded through cloning technology. Specifically, the speaker cloning model (such as GE2E model) is used to learn the unique patterns of the user's voice, compressing high-dimensional acoustic data into fixed-dimensional embedding vectors, ensuring that only voice characteristics related to user identity are included in the vectors, excluding environmental noise and pronunciation content interference.

[0076] More specifically, the embodiments of the present application further provide a method for extracting reference semantic tokens. In the step of extracting the reference semantic token and embedding vector of the reference audio, the step of obtaining the reference semantic token includes:

[0077] Mel-spectrum extraction is performed on the reference audio to obtain a second spectral representation, and the second spectral representation is input into a pre-set semantic token extraction module.

[0078] The semantic token extraction module comprises a start encoding unit, a feature encoding unit and a decoding unit. The start encoding unit is the first processing unit of the module and is responsible for adding time sequence information to the spectral features and generating an initial encoding sequence.

[0079] The feature encoding unit is configured to enhance semantic representation capability through multi-layer feature extraction, gradually compress the sequence length and abstract high-level semantic features.

[0080] The decoding unit is the output unit of the module and is composed of a decoder. The decoding unit maps the encoding sequence to a fixed-length discrete semantic token, thereby completing the conversion from the spectral features to the reference semantic token.

[0081] Specifically, the mel-spectrogram obtained by performing mel-spectrum extraction on the reference audio is taken as the second spectral representation, and the second spectral representation is taken as the input feature of the semantic token extraction module.

[0082] The second spectral representation is input into the start encoding unit. The start encoding unit comprises a start position encoder and a start feature encoder. The start position encoder is configured to perform time sequence encoding on the second spectral representation to generate a context-aware representation.

[0083] The start position encoder is a position encoder configured to add time sequence position information to the input sequence. The start position encoder generates a position vector according to a preset rule, so that the model can identify the order of the audio frames on the time axis. The start position encoder can process the second spectral representation into a context-aware representation. The context-aware representation is a spectral feature matrix to which time sequence information is added. The context-aware representation retains the frequency characteristics of the original spectrum and contains the context-dependent relationship (such as the spectral change trend of adjacent frames) of the time dimension, thereby providing time sequence marked input for subsequent encoding.

[0084] The start feature encoder is a feature encoder in the start encoding unit and is configured to encode the second spectral representation after time sequence marking to output a first-level encoding.

[0085] Specifically, the second spectral representation is input into the start encoding unit. First, the start position encoder is used to perform time sequence encoding. For each frame of spectral features, a position vector with a dimension matching the frame is generated. The numerical value of the position vector changes with the time frame index. The position vector and the spectral features of the corresponding frame are added element by element to obtain a feature matrix fused with time sequence information, i.e., a context-aware representation.

[0086] Further, the context-aware representation is input into the start feature encoder to perform encoding to obtain a first-level encoding. The first-level encoding is subjected to vector quantization processing to generate an initial encoding sequence.

[0087] Specifically, the context-aware representation is input into a starting feature encoder to output a first-level encoding through the starting feature encoder. Then, a vector quantization process is performed on the first-level encoding, and for each frame vector in the first-level encoding, an L2 norm distance is calculated between the frame vector and all vectors in a codebook, and the index of the codebook vector with the smallest distance is selected as the discrete token of the frame. The discrete tokens of all frames are arranged in chronological order to finally generate an initial encoding sequence.

[0088] Further, an encoding embedding is performed on the initial encoding sequence, and the initial encoding sequence after the encoding embedding is input into the feature encoding unit, which includes at least one feature encoder. All feature encoders in the feature encoding unit are sequentially passed through the feature encoding unit to perform encoding on the initial encoding sequence to obtain a final encoding sequence.

[0089] Specifically, the initial encoding sequence is input into the feature encoding unit, which includes at least one feature encoder (if there are multiple feature encoders, they are connected in series). Each feature encoder sequentially performs encoding operations on the input sequence to gradually compress the sequence length to reduce the computational cost of subsequent decoding. For example, the initial encoding sequence is sequentially passed through three feature encoders, the first encoder compresses 500 tokens to 300, the second compresses to 200, and the third compresses to 125, finally generating a final encoding sequence with a length of 125.

[0090] More specifically, before the initial encoding sequence is input into the feature encoding unit, the encoding text embedding needs to be fused first. The text information is converted into an embedding vector of the same dimension as the initial encoding sequence by a preset text encoding embedder, and the initial encoding sequence and the embedding vector are added element by element to realize cross-modal feature fusion. Then, the fused initial encoding sequence is input into the feature encoding unit, so that the fusion of the encoding text embedding enables the model to simultaneously model the semantic association of speech and text (such as the semantic alignment of the speech token of "rain" and the text embedding), and improves the accuracy of the subsequent reference semantic token generation.

[0091] More specifically, in the embodiments of the present application, with reference to Figure 2 , Figure 2 is a model schematic diagram of a semantic token extraction module in the end-side speech model deployment method provided by the embodiments of the present application. The following will further describe the step of sequentially passing all feature encoders in the feature encoding unit to perform encoding on the initial encoding sequence to obtain a final encoding sequence: Figure 2

[0092] ​The feature encoding unit consists of an optional intermediate encoding subunit and a terminal encoding unit. Through multiple layers of encoders, the sequence length is gradually compressed and semantic associations are enhanced to output the final encoded sequence. The intermediate encoding subunit can be determined based on the length of the reference audio that the model needs to process in the current scenario, or can be set as needed.

[0093] Determining an input coding sequence, and inputting the input coding sequence into the terminal coding unit, including, when the feature coding unit has an intermediate coding subunit, sequentially inputting the initial coding sequence into all intermediate feature encoders in the intermediate coding subunit to perform encoding, generating an intermediate coding sequence, and using the intermediate coding sequence as the input coding sequence; the intermediate coding subunit includes at least one feature encoder;

[0094] When the feature encoding unit does not have an intermediate encoding subunit, using the initial encoding sequence as the input encoding sequence;

[0095] Specifically, the source of the input coding sequence is first determined. The input coding sequence is the sequence to be processed entering the terminal coding unit. If an intermediate coding subunit exists, it is the intermediate coding sequence; if not, it is the initial coding sequence. Specifically, if the feature coding unit contains intermediate coding subunits, the initial coding sequence is sequentially input into all intermediate feature encoders for processing. Each intermediate feature encoder calculates the inter-token association weights and compresses the sequence length through a self-attention mechanism. The intermediate coding sequence generated after passing through all intermediate feature encoders is the input coding sequence. If no intermediate coding subunit exists, the initial coding sequence is directly used as the input coding sequence.

[0096] Furthermore, the terminal encoding unit performs final encoding on the input encoding sequence, including performing context augmentation encoding on the input encoding sequence through the terminal position encoder, and inputting the enhanced sequence into the terminal feature encoder to perform encoding to obtain the final encoding sequence.

[0097] Specifically, the terminal position encoder adds position information to the input coding sequence. For example, for "The weather is sunny today", the temporal relationship of "today → weather → sunny" is modeled. Then, the input coding sequence after context enhancement is input into the terminal feature encoder to output the final coding sequence. Each token corresponds to a high-level semantic unit, such as "positive weather description" and "pause at the end of the sentence".

[0098] More specifically, the intermediate encoding subunit reduces sequence redundancy through multi-layer compression, improving subsequent processing efficiency; the end position encoder solves the problem of self-attention mechanism being insensitive to sequence order, enhances context awareness, and ensures that the model accurately captures the temporal dependencies between tokens (such as avoiding the semantic reversal of "sunny weather today"), and finally the end feature encoder extracts the final encoding sequence, which not only retains the detailed features of the speech signal but also improves the reliability of semantic representation through compression and abstraction, laying the foundation for the subsequent decoding unit to generate high-quality reference semantic tokens.

[0099] Further, the final encoding sequence is input into the decoding unit to decode the final encoding sequence through the decoding unit to obtain the reference semantic token.

[0100] Among them, the final encoding sequence refers to the discrete token sequence processed by the feature encoding unit, which is generated by gradually compressing and enhancing semantic association by the multi-layer feature encoder (at least including the start feature encoder and the last feature encoder in the feature encoding unit), and contains high-level semantic structures at the sentence level.

[0101] Specifically, the final encoding sequence is input into the decoding unit, and the decoder in the decoding unit generates the optimal token sequence through a preset search strategy (such as beam search). Specifically, each time step receives the current embedding vector and the hidden state at the previous time, and outputs the probability distribution of the next token, selects the candidate token with the highest probability to form the sequence, and finally generates a fixed-length reference semantic token. The reference semantic token directly corresponds to the core semantic elements of the reference audio, including the vocabulary sequence (such as "weather" and "umbrella") and the prosodic features (such as the end pause marker).

[0102] In step S3, a training text set is obtained, and the training text encoding corresponding to each training text in the training text set, the embedding vector and the reference semantic token are integrated and input into a preset language model. The comprehensive token sequence of the corresponding training text is output by the preset language model according to the embedding vector and the reference semantic token.

[0103] Among them, the training text set refers to a text data set composed of common terms, containing multiple frequently used independent words or phrases, such as basic terms frequently appearing in daily life, business communication, instruction interaction, etc., such as "hello", "please confirm", "received" and other common terms. The training text encoding is to convert the common terms in the training text set into a digital form that can be processed by a computer, which retains the semantics and syntax structure of the corresponding text.

[0104] Among them, the preset language model is an LLM (Large Language Model).

[0105] The comprehensive token sequence includes the timbre characteristics of each semantic unit in the current training text. It is a sequence of discrete tokens that contain both semantic information and timbre characteristics. Each token corresponds to a comprehensive representation of a semantic unit (such as a single word) in the training text. A semantic unit is the smallest unit with independent semantic meaning in the text, such as a single word, phrase, or morpheme. It is the basic component of the text's meaning.

[0106] Specifically, a training text set consisting of commonly used terms is first screened and obtained from a general dictionary or a high-frequency vocabulary in the field, ensuring coverage of core scenarios such as daily conversations and service instructions, or presetting common application scenarios of end-side devices involved in user interaction. For each training text, the term is converted into a high-dimensional semantic vector using a preset language model. At the same time, the embedding vector and semantic token from the previous step are retrieved, and the training text encoding, embedding vector, and reference semantic token are integrated into an input sequence. The integrated input sequence is input into the preset language model. The model integrates the semantic meaning of the term and the timbre characteristics of the target user represented by the embedding vector, and finally outputs a comprehensive token sequence. Each token in this sequence contains the textual meaning of the corresponding term and the timbre attributes of the target user.

[0107] More specifically, by combining a training text set consisting of commonly used terms with reference audio, the pre-set language model quickly grasps the representational patterns of basic semantic units even with small data sets. The process of generating comprehensive token sequences integrates the semantics of terms with the user's timbre, enabling the model to transfer personalized timbre characteristics to speech synthesis for different terms.

[0108] For example, suppose the training text set contains multiple high-frequency daily terms such as "hello," "please wait," and "thank you." Taking the term "hello" as an example, it is converted into a high-dimensional training text encoding. Simultaneously, the embedding vector and reference semantic token obtained from the user-provided reference audio are retrieved. The text encoding, embedding vector, and reference semantic token are then concatenated along the feature dimension to form an input sequence. The input sequence is then fed into a preset language model, which can learn the association between the semantics of terms and their timbre features through methods such as self-attention mechanisms, and output a comprehensive token sequence. Each token, such as "你," not only contains the semantic information of the term, but also carries user-specific timbre features such as "smooth tone" and "clear pronunciation." When subsequently synthesizing the new term "welcome," the model can use similar token generation logic to ensure that the synthesized speech accurately conveys the meaning of the term while reproducing the user's timbre.

[0109] More specifically, in an embodiment of the present application, the step of integrating each training text in the training text set, the embedding vector, and the reference semantic token into a preset language model includes:

[0110] initializing a to-be-constructed sequence, adding a start identifier as a sequence start bit of the to-be-constructed sequence;

[0111] wherein the to-be-constructed sequence is an input sequence being assembled, the initial state is empty, various elements are added step by step to form a complete model input, and the start identifier is a special symbol (i.e. Start) marking the beginning of the sequence, used to indicate the start position of the model sequence.

[0112] Further, the embedding vector, the training text encoding of the current training text and the reference semantic token are added to the to-be-constructed sequence in a preset order, and a semantic separation identifier is added between the training text encoding and the reference semantic token;

[0113] wherein the semantic separation identifier is a special symbol (i.e. T) used to distinguish the text encoding from the reference semantic token, because they belong to different semantic levels and need to be clearly bounded by the identifier.

[0114] Further, a termination identifier is added to the sequence end bit of the to-be-constructed sequence to complete the construction of the input sequence, and the input sequence is input to the preset language model.

[0115] wherein the termination identifier is a special symbol (i.e. End) marking the end of the sequence, used to indicate the termination position of the model sequence, ensuring that the model correctly truncates the output.

[0116] Specifically, first, the to-be-constructed sequence is initialized, and the start identifier (Start) is added to the sequence start bit to clearly indicate the beginning of the sequence. Then, elements are added to the sequence in a preset order, first the embedding vector (v) is added to serve as a timbre condition to guide the subsequent generation, then the training text encoding (y) of the current training text is added, and then the semantic separation identifier (T) is inserted between the training text encoding and the reference semantic token to distinguish the level of text semantics and speech semantics, and then the reference semantic token (s) is added, i.e. the semantic token sequence corresponding to the reference audio. Finally, the termination identifier (End) is added to the sequence end bit to complete the construction of the input sequence. At this time, the complete sequence structure is [Start, v, y, T, s, End], and the entire sequence is input to the preset language model for training.

[0117] More specifically, by structuring the sequence, the model can simultaneously receive the text content, speaker characteristics and reference semantic examples, and through the start and termination identifiers, the model can correctly identify the sequence boundary, avoid input confusion, so that the preset language model can generate a high-quality semantic token sequence that conforms to the text content and matches the target timbre and semantic style. In the training process of the preset language model, the training target of the model is to minimize the loss function, and the expression of the loss function is:

[0118]

[0119] wherein L LLM is a loss function of the preset language model (LLM model), L is a sequence length, L+1 is a position where a termination identifier is located; q(μl) represents a posterior probability of the l-th frame.

[0120] Specifically, the loss function mainly focuses on the prediction error caused by the reference semantic token (s), without considering the loss of v and y, because they mainly play a role in assisting to generate an accurate semantic token sequence. Specifically, the difference between the posterior probability of the l-th frame predicted by the model and the actual value constitutes the loss, and the L+1-th frame corresponds to the End position, indicating the end of the sequence.

[0121] More specifically, for example, assuming that the training text is “tomorrow will rain”, the embedding vector (v) is a timbre feature vector extracted from the 3-second speech of user B, and the reference semantic token (s) is the semantic token sequence (containing 15 tokens) when user B says “today has rain”. First, initialize the sequence to be constructed and add Start, obtaining [Start]; add v (embedding vector of user B) and training text encoding y (BPE encoding vector of “tomorrow will rain”) in order, at which time the sequence is [Start, v, y]; insert the semantic separation identifier T, and the sequence becomes [Start, v, y, T]; then add the reference semantic token s (15 tokens of “today has rain” spoken by user B), obtaining [Start, v, y, T, s]; finally, add the termination identifier End, completing the sequence construction: [Start, v, y, T, s, End]. After inputting the sequence into the preset language model, the model will combine the timbre features of user B, the text content of “tomorrow will rain”, and the semantic example of “today has rain” to generate a semantic token sequence corresponding to “tomorrow will rain” that conforms to the timbre and expression habits of user B, providing a basis for subsequent speech synthesis.

[0122] In step S4, a mel-spectrogram conversion is performed on each of the integrated token sequences to obtain a first spectral representation corresponding to the integrated token sequence.

[0123] wherein the mel-spectrogram conversion is a process of converting an audio-related feature sequence into a mel-spectrum diagram, mapping a linear spectrum to a mel-frequency scale to generate a perceptually meaningful spectral representation (i.e., the first spectral representation) by simulating the perceptual sensitivity of the human ear to different frequencies.

[0124] Specifically, the embodiment of the present application proposes to convert the obtained synthesized token sequence into a corresponding first spectral representation through a pre-trained spectral generation model. The synthesized token sequence is converted into a mel-spectrogram form (i.e., the first spectral representation), which quantifies frequency through mel-scale, is closer to human auditory perception, and can preserve key acoustic features of speech (such as pitch and timbre). In speech synthesis, although the synthesized token sequence contains semantic and timbre information, it cannot be directly used to generate speech. However, the mel-spectrogram can be used as the input of a vocoder to synthesize natural and fluent speech signals.

[0125] More specifically, in the embodiment of the present application, the step of performing mel-spectrum conversion on each of the synthesized token sequences to obtain a first spectral representation corresponding to the synthesized token sequence comprises:

[0126] inputting the synthesized token sequence and the reference semantic token into a pre-trained spectral generation model, the spectral generation model learning a mel-spectrum distribution through pre-training;

[0127] sampling the mel-spectrum distribution to generate spectral features corresponding to the synthesized token sequence in the spectral generation model conditioned on the reference semantic token; and outputting the first spectral representation corresponding to the synthesized token sequence according to the spectral features.

[0128] Specifically, the synthesized token sequence and the reference semantic token are input into the spectral generation model as double inputs. The synthesized token sequence is converted into a content feature sequence through a token embedding layer in the spectral generation model, and the reference semantic token is converted into a style condition vector through a style embedding layer in the spectral generation model. Then, the spectral generation model injects the style condition vector into the content feature generation process. For example, the "question intonation mark" in the reference semantic token is mapped to a weight that enhances the energy of the high frequency band, which acts on the question word (such as the word "ma" at the end of the sentence) in the content feature. On this basis, the pre-trained mel-spectrum distribution parameters are called to guide the iterative denoising starting from the standard normal distribution, and the mean of the noise distribution is adjusted according to the current content feature and style condition at each step to gradually generate spectral features close to real speech. Finally, the generated spectral features are frame-aligned and dynamic range-compressed to obtain the first spectral representation.

[0129] Exemplarily, it is assumed that the integrated token sequence corresponds to the text "weekend go hiking?" (containing a questioning sentiment, 80 tokens), and the reference semantic token comes from the reference audio of "holiday go travel?" (questioning intonation, 30 tokens, high-frequency energy proportion of spectrum 38%, interframe difference at the end of the sentence 0.25) said by user A. The model embeds the integrated token sequence into content features (80x512) and embeds the reference semantic token into style conditions (30x512). Through cross attention, the "question mark" weight (0.8) in the style condition is injected into the 65th-80th frame (corresponding to the "question" word) of the content features. During generation, starting from the noise matrix (80x150), the noise is iteratively removed: in the first 500 steps, the low-frequency energy is gradually enhanced (to build the basic spectrum of "weekend go hiking"), and in the last 500 steps, the high-frequency energy of the 65th-80th frame is improved according to the style condition (to simulate the rising of the questioning intonation), and finally a 80x150 spectrum feature matrix is generated. After normalization, the high-frequency energy proportion of the first spectrum representation reaches 37% (close to the reference style), the number of time frames matches the length of the integrated token sequence, and after inputting the vocoder, the synthesized speech accurately expresses the text content of "weekend go hiking?" and reproduces the questioning intonation and speech rate of user A.

[0130] In step S5, an audio signal corresponding to the training text is generated according to each of the first spectrum representations, and all the audio signals are integrated to obtain a training data set.

[0131] The audio signal refers to an audible sound wave signal obtained by converting the mel spectrum diagram through the vocoder, which contains acoustic parameters such as fundamental frequency, amplitude, and duration, and is the final output form of speech synthesis.

[0132] Specifically, in the embodiment, the step of generating an audio signal corresponding to the training text according to each of the first spectrum representations and integrating all the audio signals to obtain a training data set comprises:

[0133] inputting the first spectrum representation into a preset vocoder, performing signal conversion on the first spectrum representation through the preset vocoder, and outputting the audio signal corresponding to the training text;

[0134] integrating all the audio signals into the training data set output.

[0135] The preset vocoder refers to a pre-trained speech waveform generation model, has the function of converting a mel spectrum into a high-quality time-domain audio signal, and is composed of a generator and a discriminator, and the waveform details are optimized through adversarial learning. In the embodiments of the present application, the preset vocoder can be a HiFiGAN (High-Fidelity Generative Adversarial Network), which is composed of a generator and a multi-scale discriminator. The generator is responsible for mapping the input mel spectrum to a time-domain waveform through inverse mel spectrum conversion, and its network structure usually includes residual blocks and up-sampling modules based on a convolutional neural network, which can capture detailed features (such as formant frequencies and spectral envelope changes) in the spectrum. The multi-scale discriminator optimizes the performance of the generator by comparing the differences between the generated waveform and the real speech, using an adversarial loss function (such as a multi-resolution STFT loss), to reduce noise and spectral distortion in the synthesized speech.

[0136] The training data set refers to a set of "text-audio" pairs generated by cloning data, and each sample is labeled with a real timbre label (such as a specific speaker's voiceprint feature vector) for model learning of the mapping relationship between personalized timbre and speech features.

[0137] Specifically, the vocoder performs signal conversion on the first spectral representation through its internal neural network structure to restore the first spectral representation to a time-domain signal, i.e., to convert the mel spectrum into the original audio waveform. Finally, all first spectral representations corresponding to the training texts are converted into corresponding audio signals, and a training data set containing about 1000 commonly used words is formed.

[0138] In step S6, the training data set is input into the model to be trained, and a lightweight speech model is trained.

[0139] The model to be trained refers to an initially constructed small speech synthesis model architecture; the timbre modeling parameters refer to a set of parameters in the model for characterizing the voice features of a specific speaker, including a voiceprint feature vector, a spectral envelope adjustment matrix, a time length prediction bias item, etc., which directly determine the timbre recognition and individual characteristics of the synthesized speech; and the lightweight speech model is a model obtained after training and parameter pruning, which only retains parameters directly related to timbre modeling (such as a voiceprint embedding matrix, a fundamental frequency prediction weight, and a formant adjustment parameter), removes redundant layers of a general language model, and realizes volume compression and efficiency optimization.

[0140] Specifically, the "text-audio" pairs in the training data set are input to the model to be trained, each input pair containing training text, corresponding audio signal and labeled real vocal color label. The training process adopts a supervised learning strategy: taking the real vocal color label as the supervision signal, calculating the loss value between the predicted vocal color label (voiceprint feature extracted from the generated audio) output by the model and the real label, iteratively optimizing the vocal modeling parameters through back propagation until the training iteration condition is met. After training, the model parameters are pruned to obtain a lightweight speech model.

[0141] More specifically, in the embodiments of the present application, the step of inputting the training data set to the model to be trained to obtain a lightweight speech model includes:

[0142] annotating the real vocal color label for the audio signal in the training data set;

[0143] inputting the annotated audio signal to the initial speech model to be trained, and outputting the predicted vocal color label of the current audio signal through the initial speech model;

[0144] Specifically, the voiceprint vector is extracted from each audio through a voiceprint recognition algorithm as the real vocal color label of the audio signal, and the annotated audio signal is then input to the initial speech model to be trained in sequence. The voiceprint feature extraction layer of the model extracts the spectral features of the audio signal, and finally forms a predicted vocal color label with the same dimension as the real vocal color label.

[0145] Subsequently, the model loss value is calculated according to the predicted vocal color and the corresponding real vocal color label, and the parameters of the initial speech model are updated based on the model loss value. The step of annotating the real vocal color label for one audio signal in the training data set is iteratively trained until the preset iteration condition is met;

[0146] Specifically, the mean square error of the predicted label and the real label is calculated as the model loss value, and the weight matrix of the vocal prediction layer is updated through the back propagation algorithm. After each round of training, the model performance is evaluated using the validation set to determine whether the iteration condition is met, otherwise the annotation step is returned for further iteration. The iteration conditions include but are not limited to the loss value not improving for a plurality of consecutive rounds, the loss value converging or the number of iterations reaching the upper limit.

[0147] Finally, the model parameter extraction is performed on the trained initial speech model, and the vocal modeling parameters are selected to construct the lightweight speech model based on the vocal modeling parameters.

[0148] Specifically, after the training is completed, parameter extraction is performed on the initial model, non-voice timbre related components such as text encoding layers and syntax analysis modules are removed, only voice timbre modeling parameters (for example, only the convolution kernel weights of the voiceprint feature extraction layer, the full connection matrix of the voice timbre prediction layer, and the output bias term) are retained, redundant information is removed, and finally a lightweight speech model is constructed based on the retained content.

[0149] In step S7, the lightweight speech model is updated to the end-side device, so that the end-side device completes end-side speech model deployment based on the lightweight speech model.

[0150] At this time, the end-side device returns to the step of obtaining the lightweight speech model fed back by the cloud-side server and performing end-side speech model deployment, wherein the lightweight speech model is a voice model that only retains voice timbre modeling parameters and is trained by the cloud-side server according to the reference audio.

[0151] Specifically, after the lightweight speech model completes the training, the model parameters of the lightweight speech model are first converted into a format supported by the end-side device, and then the processed model file is transmitted to the end-side device using OTA (Over-the-Air, wireless update technology). The end-side device loads the end-side speech model based on the received data, realizes the localization of the speech synthesis task, and does not need to connect to the cloud server in real time. Subsequent users can directly use the speech synthesis model deployed by the end-side device to perform real-time speech synthesis.

[0152] In a specific application scenario, for example, in an application scenario of an end-side device facing a storage-computing integrated chip. The end-side device of the storage-computing integrated chip hopes that the chips, modules, sensors, and other components deployed inside the device occupy less area, so that such an end-side device has a corresponding cloud-side server to realize an end-cloud combined transmission processing system. However, how to use the capabilities of the cloud-side server to output a speech model adapted to the end-side device to meet the lightweight demand of the end-side device is a problem to be solved.

[0153] By introducing the end-side speech model deployment method of the present application, the reference semantic token and embedding vector of the end-side device can be extracted on the cloud-side server to train a lightweight speech model. Finally, through the OTA update and upgrade mode, the end-side device is deployed and updated based on the lightweight speech model published or transmitted by the cloud-side server, reducing the storage pressure of the end-side device on different versions of speech models, thereby reducing the size of the components deployed on the end-side device, further reducing the occupied area of the components on the end-side device, and improving the inference speed of the end-side device.

[0154] Reference Figure 3 , Figure 3 is a virtual structural schematic diagram of an end-side speech model deployment device provided by the present application. The second aspect of the present application provides an end-side speech model deployment device, which comprises:

[0155] Reference audio acquisition module 100, configured to acquire reference audio sent by the end-side device;

[0156] Audio information extraction module 200, configured to extract reference semantic tokens and embedding vectors of the reference audio; the reference semantic tokens are discrete labels with semantic information in the reference audio, and the embedding vectors are used to represent target user voice characteristics;

[0157] Token sequence generation module 300, configured to acquire a training text set, integrate training text encoding corresponding to each training text in the training text set, the embedding vectors and the reference semantic tokens into a preset language model, and output a comprehensive token sequence of the corresponding training text according to the embedding vectors and the reference semantic tokens through the preset language model; the comprehensive token sequence includes voice characteristics of each semantic unit in the current training text;

[0158] Spectrum representation conversion module 400, configured to perform mel spectrum conversion on each of the comprehensive token sequences to obtain a first spectrum representation of the corresponding comprehensive token sequence;

[0159] Training data acquisition module 500, configured to generate an audio signal of the corresponding training text according to each of the first spectrum representations, and integrate all the audio signals to obtain a training data set;

[0160] Lightweight model generation module 600, configured to input the training data set into a model to be trained to obtain a lightweight speech model; the lightweight speech model is a speech model that only retains voice modeling parameters;

[0161] End-side model deployment module 700, configured to update the lightweight speech model to the end-side device to complete end-side speech model deployment.

[0162] The end-side speech model deployment device provided in the embodiments of the present application can execute the end-side speech model deployment method provided in the above embodiments, and has the corresponding functional steps and beneficial effects of the end-side speech model deployment method described in the above embodiments. For details, please refer to the embodiments of the end-side speech model deployment method. The embodiments of the present application will not be repeated here.

[0163] The embodiments of the present application also provide an electronic device, please refer to Figure 4 , Figure 4Fig. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device can include a processor and a memory, where the processor and the memory can be connected through a bus or other means. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions / modules corresponding to the end-side speech model deployment method in the embodiments of the present application. The processor performs various function applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the end-side speech model deployment method in the above method embodiments.

[0164] The memory can include a program storage area and a data storage area, where the program storage area can store an operating system and application programs required by at least one function; and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. The one or more modules are stored in the memory and, when executed by the processor, perform the end-side speech model deployment method in the above method embodiments. The above electronic device specific details can correspond to the relevant description and effects of the corresponding method embodiments, which will not be described here. Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0165] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0166] Similarly, it is to be understood that the various features of the present application can sometimes be used to advantage together, but each of the features can be used independently as provided by the foregoing description, and the independent use of individual features is specifically contemplated. In addition, many of the concepts are applicable in view of other, similar devices or methodologies. Accordingly, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the claims.

[0167] It is noted that the foregoing examples have been provided merely for the purposes of illustration and are not intended to limit the application of the application. Other examples and embodiments thereof will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. Therefore, reference should be made to the claims for an appropriate determination of the scope of the application thereof.

Claims

1. A method for deploying a device-side voice model, characterized in that: The method is applied to a cloud server, which is in communication with a terminal device, and includes: Obtaining reference audio sent by the end-side device; Extracting a reference semantic token and an embedding vector of the reference audio; the reference semantic token is a discrete tag with semantic information in the reference audio, and the embedding vector is used to represent the timbre characteristics of the target user; Obtaining a training text set, integrating the training text code, the embedding vector, and the reference semantic token corresponding to each training text in the training text set into a preset language model, and outputting a comprehensive token sequence corresponding to the training text through the preset language model based on the embedding vector and the reference semantic token; the comprehensive token sequence includes the timbre characteristics of each semantic unit in the current training text; Performing a Mel-spectrum conversion on each of the integrated token sequences to obtain a first spectrum representation of the corresponding integrated token sequence; generating an audio signal corresponding to a training text according to each of the first spectral representations, and integrating all of the audio signals to obtain a training data set; Inputting the training data set into the model to be trained to obtain a lightweight speech model; the lightweight speech model is a speech model that only retains timbre modeling parameters; The lightweight voice model is sent to the end-side device, so that the end-side device completes the end-side voice model deployment based on the lightweight voice model.

2. The method for deploying a client-side speech model according to claim 1, wherein: In the step of extracting the reference semantic token and embedding vector of the reference audio, the step of obtaining the reference semantic token includes: Performing Mel spectrum extraction on the reference audio to obtain a second spectrum representation, and inputting the second spectrum representation into a preset semantic token extraction module; the semantic token extraction module includes a start encoding unit, a feature encoding unit, and a decoding unit; Inputting the second spectral representation into the start encoding unit, the start encoding unit including a start position encoder and a start feature encoder, performing temporal encoding on the second spectral representation by the start position encoder to generate a context-aware representation; Inputting the context-aware representation into the starting feature encoder to perform encoding to obtain a first-level code, performing vector quantization processing on the first-level code to generate an initial code sequence; Performing code embedding on the initial code sequence, inputting the initial code sequence after code embedding into the feature encoding unit, the feature encoding unit including at least one feature encoder, and sequentially encoding the initial code sequence through all feature encoders in the feature encoding unit to obtain a final code sequence; The final encoded sequence is input into the decoding unit, so that the final encoded sequence is decoded by the decoding unit to obtain the reference semantic token.

3. The method for deploying a terminal-side speech model according to claim 2, wherein: The feature encoding unit includes a terminal encoding unit, and the terminal encoding unit includes a terminal position encoder and a terminal feature encoder; The step of sequentially encoding the initial encoding sequence through all feature encoders in the feature encoding unit to obtain a final encoding sequence includes: Determining an input coding sequence, and inputting the input coding sequence into the terminal coding unit, including, when the feature coding unit has an intermediate coding subunit, sequentially inputting the initial coding sequence into all intermediate feature encoders in the intermediate coding subunit to perform encoding, generating an intermediate coding sequence, and using the intermediate coding sequence as the input coding sequence; the intermediate coding subunit includes at least one feature encoder; When the feature encoding unit does not have an intermediate encoding subunit, using the initial encoding sequence as the input encoding sequence; The terminal encoding unit performs final encoding on the input encoding sequence, including performing context enhancement encoding on the input encoding sequence through the terminal position encoder, and inputting the enhanced sequence into the terminal feature encoder to perform encoding to obtain the final encoding sequence.

4. The method for deploying a terminal-side speech model according to claim 1, wherein: The step of integrating each training text in the training text set, the embedding vector and the reference semantic token into a preset language model comprises: Initialize the sequence to be constructed, and add a start identifier as the sequence start position of the sequence to be constructed; Adding the embedding vector, the training text code of the current training text, and the reference semantic token to the sequence to be constructed in a preset order, and adding a semantic separation identifier between the training text code and the reference semantic token; A termination identifier is added to the end of the sequence to be constructed to complete the construction of the input sequence, and the input sequence is input into the preset language model.

5. The method for deploying a terminal-side speech model according to claim 1, wherein: The step of performing Mel spectrum conversion on each of the integrated token sequences to obtain a first spectrum representation of the corresponding integrated token sequence comprises: Inputting the comprehensive token sequence and the reference semantic token into a pre-trained spectrum generation model, wherein the spectrum generation model learns the Mel spectrum distribution through pre-training; In the spectrum generation model, taking the reference semantic token as a condition, generating spectrum features corresponding to the comprehensive token sequence from the Mel spectrum distribution sampling; A first spectrum representation corresponding to the integrated token sequence is output according to the spectrum feature.

6. The method for deploying a terminal-side speech model according to claim 1, wherein: The step of inputting the training data set into the model to be trained to obtain a lightweight speech model comprises: Labeling the audio signals in the training data set with real timbre labels; Inputting the labeled audio signal into an initial speech model to be trained, and outputting a predicted timbre label of the current audio signal through the initial speech model; Calculating a model loss value according to the predicted timbre and the corresponding true timbre label, and updating the parameters of the initial speech model based on the model loss value, returning to the step of labeling an audio signal in the training data set with a true timbre label, and iterating the training until a preset iteration condition is met; Model parameter extraction is performed on the trained initial speech model to screen out the timbre modeling parameters, and the lightweight speech model is constructed based on the timbre modeling parameters.

7. A method for deploying a device-side voice model, characterized in that: The method is applied to a terminal-side device, the terminal-side device being communicatively connected to a cloud server, and the method includes: Sending the pre-recorded reference audio to the cloud-side server; Obtain the lightweight speech model fed back by the cloud-side server and perform end-side speech model deployment; the lightweight speech model is a speech model obtained by the cloud-side server based on the reference audio training and only retains the timbre modeling parameters.

8. A device-side voice model deployment device, characterized in that: The device is applied to a cloud server, the cloud server is communicatively connected to a terminal device, and the device includes: A reference audio acquisition module, configured to acquire the reference audio sent by the terminal-side device; An audio information extraction module, configured to extract reference semantic tokens and embedding vectors of the reference audio; the reference semantic tokens are discrete markers with semantic information in the reference audio, and the embedding vectors are used to characterize the timbre characteristics of the target user; A token sequence generation module is configured to obtain a training text set, integrate the training text code, the embedding vector, and the reference semantic token corresponding to each training text in the training text set into a preset language model, and output a comprehensive token sequence corresponding to the training text based on the embedding vector and the reference semantic token through the preset language model; the comprehensive token sequence includes the timbre characteristics of each semantic unit in the current training text; a spectrum representation conversion module, configured to perform a Mel spectrum conversion on each of the integrated token sequences to obtain a first spectrum representation of the corresponding integrated token sequence; a training data acquisition module, configured to generate an audio signal corresponding to a training text according to each of the first spectral representations, and integrate all of the audio signals to obtain a training data set; A lightweight model generation module, configured to input the training data set into a model to be trained to obtain a lightweight speech model; the lightweight speech model is a speech model that only retains timbre modeling parameters; The end-side model deployment module is used to update the lightweight voice model to the end-side device to complete the end-side voice model deployment.

9. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a program or instruction, and when the program or instruction is executed, the method according to any one of claims 1 to 7 is implemented.

Images