A cross-language speech generation method based on IPA phoneme mapping and vector quantization alignment

Abstract

The application discloses a cross-language speech generation method based on IPA phoneme mapping and vector quantization alignment, belongs to the technical field of speech signal processing and artificial intelligence, and aims at solving the problems of multi-language phoneme representation isolation, insufficient low-resource language acoustic alignment and OOV symbol influence on cross-language speech generation stability. The application is characterized in that: through standardization processing, IPA mapping, unified phoneme table construction, acoustic feature extraction and vector quantization processing, IPA embedded vector and discrete semantic Token alignment, OOV fallback processing and speech generation, target speech waveform and structured alignment information are output, and the application is suitable for the fields of multi-language speech synthesis, cross-language voiceprint cloning, low-resource language speech generation and multi-language intelligent speech interaction.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing and artificial intelligence technology, specifically involving cross-language speech generation technology based on IPA phoneme mapping and vector quantization alignment. Background Technology

[0002] With the development of multilingual speech synthesis, cross-lingual speaker cloning, and low-resource language speech services, speech generation systems are gradually shifting from monolingual modeling to multilingual shared modeling. In existing technologies, common solutions typically first convert the input text from character to phoneme, then combine this with acoustic models, vocoders, or end-to-end speech synthesis models to generate speech waveforms. Some solutions also incorporate multilingual training data, speaker vectors, or language tags to improve the synthesis adaptability between different languages.

[0003] Most existing solutions focus on speech synthesis model structure, speaker feature extraction, cross-language timbre transfer, or multilingual text front-end processing. Some solutions distinguish input text in different languages ​​using language tags, others configure independent phoneme sets or pronunciation rule bases for specific languages, and still others use pre-trained acoustic models to extract speech features and then generate target speech through neural networks. These solutions can improve the performance of multilingual speech synthesis to some extent, but their phoneme representations usually still rely on the rules within each language, lacking a unified pronunciation representation foundation across languages.

[0004] Under the above processing methods, feature isolation easily forms between phoneme sets of different languages. For low-resource languages, due to insufficient training speech samples, the model struggles to stably learn the correspondence between phonemes and acoustic features, easily leading to problems such as stiff pronunciation, omissions, repetitions, or abnormal silences. In cross-language voiceprint cloning scenarios, insufficiently trained phonemes in the target language also weaken voiceprint similarity and speech naturalness. Furthermore, when pronunciation symbols outside the unified vocabulary appear during the inference stage, existing solutions often lack a backoff mechanism directly connected to the speech generation process, making it difficult for unknown phonemes to call stable phoneme embedding representations.

[0005] Therefore, it is necessary to propose a cross-lingual speech generation method that can establish a unified pronunciation representation across multiple languages ​​and effectively align phoneme representations with acoustic features. This method constructs a unified IPA phoneme sequence and a unified phoneme vocabulary through IPA phoneme mapping, generates discrete semantic tokens by combining vector quantization processing, and establishes an alignment relationship between the IPA embedding vector and the discrete semantic tokens. Simultaneously, it performs backtracking processing on OOV symbols during the inference stage, thereby improving pronunciation stability, acoustic alignment capability, and target speech output quality in low-resource languages ​​and cross-lingual speech generation processes. Summary of the Invention

[0006] To address the problems of isolated multilingual phoneme representations, insufficient acoustic alignment of low-resource languages, and the impact of OOV symbols on the stability of cross-lingual speech generation in existing technologies, this invention proposes the following solutions: A cross-lingual speech generation method based on IPA phoneme mapping and vector quantization alignment, the method comprising: S1. Obtain multilingual text-speech pair data during the training phase and the text to be synthesized and target language labels during the inference phase, and perform standardization processing on the text in the multilingual text-speech pair data and the text to be synthesized. S2. Convert the standardized text into a phoneme sequence within the corresponding language, and call the preset IPA mapping function to map the phoneme sequence within the corresponding language into a unified IPA phoneme sequence. S3. Construct a unified phoneme vocabulary shared by all languages ​​based on the unified IPA phoneme sequence, and construct a phoneme embedding matrix based on the unified phoneme vocabulary; S4. Perform acoustic feature extraction and vector quantization on the speech waveforms in the multilingual text-speech pair data to generate discrete semantic tokens corresponding to the acoustic features. S5. Based on the phoneme embedding matrix and discrete semantic token, establish an alignment relationship between the IPA embedding vector and the discrete semantic token, and update the phoneme embedding matrix according to the alignment relationship, so that IPA symbols with the same or similar pronunciations are clustered in the embedding space to obtain a cross-language unified phoneme space for speech generation. S6. In the inference stage, based on the target language tag, the standardized text to be synthesized is converted into a unified IPA phoneme sequence to be synthesized. When there is an OOV symbol in the unified IPA phoneme sequence to be synthesized, OOV back-off processing is performed on the OOV symbol, the OOV symbol is mapped to the known phoneme index in the unified phoneme vocabulary, and the phoneme embedding vector corresponding to the known phoneme index is called. S7. Based on the cross-language unified phoneme space, perform speech generation on the unified IPA phoneme sequence to be synthesized after OOV back-off processing, and output the target speech waveform and structured alignment information.

[0007] Furthermore, the multilingual text-speech pair data in S1 includes text, speech waveforms, language tags, and speaker IDs; the standardization process includes: scanning the text and the text to be synthesized using a preset regular expression library, filtering out non-standard characters, and performing regular expression expansion on non-text entities such as numerical values, percentages, and times.

[0008] Furthermore, S2, which describes converting the standardized text into a phoneme sequence within the corresponding language, includes: converting the standardized text into a phoneme sequence within the corresponding language by using a multilingual G2P tool that integrates mapping rule bases for different language families, based on the language tag.

[0009] Furthermore, the preset IPA mapping function in S2 performs mapping based on the IPA mapping index table. The input of the IPA mapping index table is the original phoneme symbols of a specific language, and the output is the corresponding International Phonetic Alphabet symbols. The IPA mapping index table includes phoneme mapping rules and mapping types within the language. The mapping types include one-to-one mapping and many-to-one mapping.

[0010] Further, the acoustic feature extraction and vector quantization processing described in S4 includes: extracting continuous acoustic features from the speech waveform using a pre-trained acoustic encoder, performing nonlinear dimensionality reduction and resampling on the continuous acoustic features to obtain latent acoustic features, and projecting the latent acoustic features onto a discrete codebook through a vector quantization mechanism to generate discrete semantic tokens; wherein, the pre-trained acoustic encoder is a pre-trained self-supervised acoustic model, and the pre-trained self-supervised acoustic model includes a wav2vec 2.0 encoder or an XLS-R encoder, and the sampling rate of the speech waveform is 16kHz.

[0011] Furthermore, the nonlinear dimensionality reduction and resampling consists of three one-dimensional convolutional layers. The first layer has a kernel size of 3, a stride of 2, an input dimension of 512, an output dimension of 256, and uses ReLU activation. The second layer has a kernel size of 3, a stride of 2, an input dimension of 256, an output dimension of 128, and uses ReLU activation and residual connections. The third layer has a kernel size of 3, a stride of 1, an input dimension of 128, an output dimension of 64, and uses Tanh activation. The discrete codebook includes multiple code centers. The codebook size of the discrete codebook is 1024, and the code center dimension is 256. The code centers are initialized by k-means clustering of the acoustic features of the pre-trained acoustic encoder.

[0012] Furthermore, the generation of discrete semantic tokens includes: performing a nearest neighbor search on the latent acoustic features in the discrete codebook, and determining the corresponding discrete semantic token based on the code center with the smallest Euclidean distance.

[0013] Furthermore, the alignment relationship between the IPA embedding vector and the discrete semantic token described in S5 includes: establishing a probabilistic correspondence between the IPA embedding vector and the codebook index in the discrete codebook through a multilayer perceptron, wherein the input of the multilayer perceptron is the IPA embedding vector and the output is the probability distribution of the codebook index, and the probability value of the IPA embedding vector and the corresponding acoustic code is maximized by minimizing the cross-entropy loss.

[0014] Furthermore, the updating of the phoneme embedding matrix in S5 includes: synchronously optimizing the phoneme embedding matrix, speech generator, discriminator, and discrete codebook through a joint loss function, so that IPA symbols with the same or similar pronunciations are clustered in the embedding space; the joint loss function includes Mel spectrum reconstruction loss, adversarial loss, feature matching loss, and vector quantization loss, wherein the feature matching loss is used to minimize the feature distance of the same phoneme across languages, and the vector quantization loss includes codebook loss and commitment loss.

[0015] Based on the same inventive concept, the present invention also proposes a computer storage medium storing a computer program, which, when executed by a processor, implements the above-described method.

[0016] Compared with the prior art, the present invention has the following beneficial effects: By converting standardized text into phoneme sequences within the corresponding language and calling a preset IPA mapping function to map it into a unified IPA phoneme sequence, phonemes from different languages ​​can enter into a unified representation basis, thereby improving the problem of isolation in multilingual phoneme representation in existing technologies.

[0017] By constructing a unified phoneme vocabulary shared across all languages ​​based on the unified IPA phoneme sequence, and constructing a phoneme embedding matrix based on the unified phoneme vocabulary, IPA symbols with the same or similar pronunciations in different languages ​​can share phoneme embedding representations, thus providing a basis for low-resource languages ​​to call the pronunciation representations of high-resource languages.

[0018] By extracting acoustic features and quantizing vectors from speech waveforms, discrete semantic tokens corresponding to the acoustic features are generated, and an alignment relationship is established between the IPA embedding vector and the discrete semantic tokens, so that the phoneme representation can correspond to the acoustic features, thereby improving the pronunciation instability problem caused by insufficient acoustic alignment in low-resource languages.

[0019] By updating the phoneme embedding matrix according to the alignment relationship, IPA symbols with the same or similar pronunciations are clustered in the embedding space to obtain a cross-lingual unified phoneme space for speech generation. Compared with the prior art, this can reduce feature fragmentation caused by independent modeling of phonemes in different languages, thereby improving the pronunciation consistency of cross-lingual speech generation.

[0020] By performing OOV backoff processing on OOV symbols in the unified IPA phoneme sequence to be synthesized during the inference stage, the OOV symbols are mapped to known phoneme indices in the unified phoneme vocabulary, and the corresponding phoneme embedding vectors are called, so that unknown pronunciation symbols can be accessed by existing phoneme representations, thereby reducing the risk of abnormal speech generation or decreased stability caused by OOV symbols.

[0021] By generating speech from the unified IPA phoneme sequence to be synthesized after OOV backtracking based on a cross-lingual unified phoneme space, the target speech waveform and structured alignment information are output, giving the cross-lingual speech generation process a clear phoneme representation and acoustic alignment basis, thereby improving the stability of the target speech waveform output.

[0022] This invention has the ability to unify multilingual phoneme representation, improve acoustic alignment of low-resource languages, and reduce the impact of OOV symbols on speech generation stability. It can improve the stability of target speech waveform output during cross-language speech generation and is applicable to fields such as multilingual speech synthesis, cross-language voiceprint cloning, low-resource language speech generation, and multilingual intelligent voice interaction. Attached Figure Description

[0023] Figure 1 This is a flowchart of the cross-language speech generation method described in the implementation method; Figure 2 This is the overall system flowchart described in the implementation method; Figure 3 This is a flowchart of the IPA mapping and OOV fallback mechanism described in the implementation method; Figure 4 This is a design diagram of the vector quantization alignment architecture described in the implementation method; Figure 5 This is an example diagram of the G2P+ multilingual IPA phoneme mapping described in the implementation method; Figure 6 This is a schematic diagram of the dataset input format described in the implementation method; Figure 7 This is a schematic diagram of the IPA extraction output format of the dataset described in the implementation method; Figure 8 This is a schematic diagram illustrating the extraction of semantic tokens using the VITS model described in the implementation method. Figure 9 This is a schematic diagram of the symbol registry described in the implementation method; Figure 10 It is a visualization analysis diagram of the phoneme embedding matrix E described in the implementation method; Figure 11 This is a visualization analysis diagram of the phoneme embedding matrix (IPA) distribution described in the implementation method; Figure 12It is the discrete semantic token frequency distribution histogram described in the implementation method. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Implementation Method 1 like Figure 1 As shown, a cross-lingual speech generation method based on IPA phoneme mapping and vector quantization alignment is described, the method comprising: S1. Obtain multilingual text-speech pair data during the training phase and the text to be synthesized and target language labels during the inference phase, and perform standardization processing on the text in the multilingual text-speech pair data and the text to be synthesized. S2. Convert the standardized text into a phoneme sequence within the corresponding language, and call the preset IPA mapping function to map the phoneme sequence within the corresponding language into a unified IPA phoneme sequence. S3. Construct a unified phoneme vocabulary shared by all languages ​​based on the unified IPA phoneme sequence, and construct a phoneme embedding matrix based on the unified phoneme vocabulary; S4. Perform acoustic feature extraction and vector quantization on the speech waveforms in the multilingual text-speech pair data to generate discrete semantic tokens corresponding to the acoustic features. S5. Based on the phoneme embedding matrix and discrete semantic token, establish an alignment relationship between the IPA embedding vector and the discrete semantic token, and update the phoneme embedding matrix according to the alignment relationship, so that IPA symbols with the same or similar pronunciations are clustered in the embedding space to obtain a cross-language unified phoneme space for speech generation. S6. In the inference stage, based on the target language tag, the standardized text to be synthesized is converted into a unified IPA phoneme sequence to be synthesized. When there is an OOV symbol in the unified IPA phoneme sequence to be synthesized, OOV back-off processing is performed on the OOV symbol, the OOV symbol is mapped to the known phoneme index in the unified phoneme vocabulary, and the phoneme embedding vector corresponding to the known phoneme index is called. S7. Based on the cross-language unified phoneme space, perform speech generation on the unified IPA phoneme sequence to be synthesized after OOV back-off processing, and output the target speech waveform and structured alignment information.

[0026] Preferably, the structured alignment information includes a unified IPA phoneme sequence, discrete semantic tokens, and the correspondence between the two, which are used to characterize the alignment state between phoneme representation and acoustic representation in the speech generation process.

[0027] By using the above method, the phoneme representation of multilingual text, the acoustic features of speech waveforms, and the OOV backoff processing in the inference stage are incorporated into the same processing link, so that the cross-language speech generation process has a unified phoneme representation basis and acoustic alignment basis, thereby improving the stability of the target speech waveform output.

[0028] Furthermore, the multilingual text-speech pair data in S1 includes text, speech waveforms, language tags, and speaker IDs; the standardization process includes: scanning the text and the text to be synthesized using a preset regular expression library, filtering out non-standard characters, and performing regular expression expansion on non-text entities such as numerical values, percentages, and times.

[0029] Preferably, the same set of standardized processing rules is used in the training phase and the inference phase to ensure that the phoneme mapping relationship and acoustic feature representation formed in the training phase are consistent with the text processing results to be synthesized in the inference phase.

[0030] By acquiring multilingual text-speech pair data containing text, speech waveforms, language tags, and speaker IDs, and standardizing the text and the text to be synthesized, subsequent phoneme conversion and acoustic feature processing have standardized inputs, thereby reducing the interference of non-standard characters and non-text entities on the speech generation process.

[0031] Furthermore, S2, which describes converting the standardized text into a phoneme sequence within the corresponding language, includes: converting the standardized text into a phoneme sequence within the corresponding language by using a multilingual G2P tool that integrates mapping rule bases for different language families, based on the language tag.

[0032] Preferably, the multilingual G2P tool calls the mapping rule library of the corresponding language family according to the language tag, so that the text of different languages ​​is first converted into the phoneme sequence within the corresponding language, and then enters the mapping process of the unified IPA phoneme sequence.

[0033] By using language tags and combining multilingual G2P tools to perform intralingual phoneme conversion, the training text can be converted into the corresponding intralingual phoneme sequence, thus providing a conversion basis for the subsequent formation of a unified IPA phoneme sequence.

[0034] Furthermore, the preset IPA mapping function in S2 performs mapping based on the IPA mapping index table. The input of the IPA mapping index table is the original phoneme symbols of a specific language, and the output is the corresponding International Phonetic Alphabet symbols. The IPA mapping index table includes phoneme mapping rules and mapping types within the language. The mapping types include one-to-one mapping and many-to-one mapping.

[0035] Preferably, the IPA mapping index table distinguishes the original phoneme symbols of different languages ​​according to the language label, and retains the corresponding International Phonetic Alphabet symbols during the mapping process, so that phonemes within different languages ​​can be merged into a unified IPA phoneme sequence.

[0036] By performing mapping based on the IPA mapping index table, the original phoneme symbols of a specific language can be converted into the corresponding International Phonetic Alphabet symbols, thereby enabling phonemes from different languages ​​to be expressed in a unified IPA phoneme sequence.

[0037] Further, the acoustic feature extraction and vector quantization processing described in S4 includes: extracting continuous acoustic features from the speech waveform using a pre-trained acoustic encoder, performing nonlinear dimensionality reduction and resampling on the continuous acoustic features to obtain latent acoustic features, and projecting the latent acoustic features onto a discrete codebook through a vector quantization mechanism to generate discrete semantic tokens; wherein, the pre-trained acoustic encoder is a pre-trained self-supervised acoustic model, and the pre-trained self-supervised acoustic model includes a wav2vec 2.0 encoder or an XLS-R encoder, and the sampling rate of the speech waveform is 16kHz.

[0038] Preferably, the vector quantization mechanism projects continuous acoustic features onto a discrete codebook and uses discrete semantic tokens to represent acoustic articulation units in the speech waveform, so that the continuous acoustic representation and the discrete IPA phoneme representation are in an alignable representation form.

[0039] By extracting continuous acoustic features through a pre-trained acoustic encoder and generating discrete semantic tokens through nonlinear dimensionality reduction, resampling, and vector quantization mechanisms, the acoustic information in the speech waveform can be converted into a discrete representation that can be aligned with the IPA embedding vector.

[0040] Furthermore, the nonlinear dimensionality reduction and resampling consists of three one-dimensional convolutional layers. The first layer has a kernel size of 3, a stride of 2, an input dimension of 512, an output dimension of 256, and uses ReLU activation. The second layer has a kernel size of 3, a stride of 2, an input dimension of 256, an output dimension of 128, and uses ReLU activation and residual connections. The third layer has a kernel size of 3, a stride of 1, an input dimension of 128, an output dimension of 64, and uses Tanh activation. The discrete codebook includes multiple code centers. The codebook size of the discrete codebook is 1024, and the code center dimension is 256. The code centers are initialized by k-means clustering of the acoustic features of the pre-trained acoustic encoder.

[0041] Preferably, the three one-dimensional convolutional layers process the continuous acoustic features by compressing the dimensions step by step, and the potential acoustic features after dimensionality reduction are received by the code cores in the discrete codebook, so that the potential acoustic features can enter the subsequent nearest neighbor search process.

[0042] By performing nonlinear dimensionality reduction and resampling on continuous acoustic features through three one-dimensional convolutional layers and quantizing them through a discrete codebook with multiple code centers, continuous acoustic features can be transformed into structured latent acoustic features and discretized representations, thus providing a basis for the alignment between acoustic features and phoneme embeddings.

[0043] Furthermore, the generation of discrete semantic tokens includes: performing a nearest neighbor search on the latent acoustic features in the discrete codebook, and determining the corresponding discrete semantic token based on the code center with the smallest Euclidean distance.

[0044] Preferably, the nearest neighbor search uses the Euclidean distance between the latent acoustic features and each code core in the discrete codebook as the matching criterion, and uses the code core index with the smallest distance as the criterion for determining the discrete semantic token.

[0045] By performing nearest neighbor search on latent acoustic features in the discrete codebook and determining the discrete semantic token with the code center that has the smallest Euclidean distance, the latent acoustic features can be stably converted into discrete semantic tokens, thereby improving the determinism of subsequent alignment relationship establishment.

[0046] Furthermore, the alignment relationship between the IPA embedding vector and the discrete semantic token described in S5 includes: establishing a probabilistic correspondence between the IPA embedding vector and the codebook index in the discrete codebook through a multilayer perceptron, wherein the input of the multilayer perceptron is the IPA embedding vector and the output is the probability distribution of the codebook index, and the probability value of the IPA embedding vector and the corresponding acoustic code is maximized by minimizing the cross-entropy loss.

[0047] Preferably, the multilayer perceptron uses the IPA embedding vector as the input representation and the codebook index probability distribution in the discrete codebook as the output representation, so that the IPA embedding vector can establish a correspondence with the discrete semantic token through the probability distribution form.

[0048] By establishing a probabilistic correspondence between IPA embedding vectors and codebook indices in a discrete codebook using a multilayer perceptron, and by minimizing the cross-entropy loss to adjust this probabilistic correspondence, the IPA embedding vectors can form a more explicit correspondence with the corresponding acoustic codes, thereby supporting the alignment of phoneme representations with acoustic representations.

[0049] Furthermore, the updating of the phoneme embedding matrix in S5 includes: synchronously optimizing the phoneme embedding matrix, speech generator, discriminator, and discrete codebook through a joint loss function, so that IPA symbols with the same or similar pronunciations are clustered in the embedding space; the joint loss function includes Mel spectrum reconstruction loss, adversarial loss, feature matching loss, and vector quantization loss, wherein the feature matching loss is used to minimize the feature distance of the same phoneme across languages, and the vector quantization loss includes codebook loss and commitment loss.

[0050] Preferably, the speech generator takes the IPA embedding vector output by the phoneme embedding matrix as input and combines discrete semantic tokens to constrain the alignment relationship between acoustic features and phoneme embeddings; the discriminator generates the acoustic feature distribution of speech by constraining adversarial loss and feature matching loss.

[0051] By simultaneously optimizing the phoneme embedding matrix, speech generator, discriminator, and discrete codebook through a joint loss function, IPA symbols with the same or similar pronunciations are clustered in the embedding space, thereby forming a cross-linguistic unified phoneme space for speech generation.

[0052] The method described in this embodiment can be executed by a processor calling a computer program, which can be stored in a computer storage medium. When the computer program is executed by the processor, the above-described cross-language speech generation method based on IPA phoneme mapping and vector quantization alignment can be implemented.

[0053] Implementation Method 2 This implementation provides a specific process for a cross-lingual speech generation method based on IPA phoneme mapping and vector quantization alignment. This implementation focuses on two phases: training and inference. The training phase constructs a unified IPA phoneme sequence, a unified phoneme vocabulary, a phoneme embedding matrix, discrete semantic tokens, and a cross-lingual unified phoneme space based on multilingual text-speech pair data. The inference phase generates a unified IPA phoneme sequence to be synthesized based on the text to be synthesized, and performs OOV backoff processing when OOV symbols are present. Finally, it outputs the target speech waveform and structured alignment information.

[0054] The overall processing flow is as follows Figure 2 As shown, this implementation method forms a complete cross-language speech generation process, from text input, text preprocessing, multilingual G2P conversion, IPA mapping, OOV fallback processing, vector quantization alignment, construction of a unified phoneme space across languages, to target speech waveform output. During the training and inference phases, the system first enters the initialization process, executing text preprocessing and model loading in parallel. The system receives input text through a cleaning interface, performs standardized format conversion, including unifying to UTF-8 and processing Unicode characters; simultaneously, it loads a pre-trained multilingual acoustic model for extracting acoustic features. During text preprocessing, a pre-defined regular expression library is used. It scans the input multilingual text, filters out non-standard characters such as special emojis and unconventional escape characters, and performs regular expression expansion on non-text entities such as numbers, percentages, and times to prevent abnormal conversion by the multilingual G2P tool.

[0055] During the acoustic encoder loading process, the system loads a pre-trained self-supervised acoustic model using a deep learning framework. This pre-trained self-supervised acoustic model can employ a wav2vec 2.0 encoder or an XLS-R encoder to sample the input raw speech and generate acoustic context representations at 50Hz or finer granularity. This acoustic encoder is used to extract continuous acoustic features from speech waveforms in multilingual text-speech pair data, where the sampling rate of the speech waveforms is uniformly set to 16kHz.

[0056] During IPA mapping initialization, the system loads the IPA mapping index table. The input to the IPA mapping index table is the original phoneme symbols of a specific language, and the output is the corresponding International Phonetic Alphabet (IPA) symbols. The original phoneme symbols of the specific language include Chinese Pinyin, English Arpabet, and Uyghur phonemes. The IPA mapping index table includes intra-language phoneme mapping rules and mapping types. The intra-language phoneme mapping rules establish one-to-one or many-to-one mapping relationships for different language phoneme systems. The mapping type is primarily many-to-one, with some high-frequency monophones using one-to-one mapping. The initials "zh", "ch", and "sh" in Mandarin Chinese are mapped to [t] in the IPA. ]、[t h ]、[ [ɑ]; In English Arpabet, “AH” and “AA” both correspond to IPA [ɑ]; The Chinese pinyin “ü” has a one-to-one mapping with IPA [y]; In English Arpabet, “EH” and “AE” correspond to IPA [ɛ] and [æ] respectively, which is also a one-to-one mapping.

[0057] Multilingual G2P tools convert text from different languages ​​into phoneme sequences within the corresponding language, and then convert them into unified IPA phoneme sequences using an IPA mapping function. The specific mapping process is as follows: Figure 5 As shown. In the process of unified multilingual phoneme mapping, the system uses a multilingual G2P tool to convert the standardized text into phoneme sequences within the corresponding language. The multilingual G2P tool adapts to the pronunciation rules of different languages, handling the correspondence between pinyin and initials / finals / tones for Chinese, and handling pronunciation variations of letter combinations for English, thereby generating phoneme representations reflecting the pronunciation characteristics of the corresponding language. Subsequently, the system calls a preset IPA mapping function, mapping the phonemes within each language to a unified IPA phoneme sequence based on the IPA mapping index table, allowing the original phoneme symbols of different languages ​​to enter a unified International Phonetic Alphabet (IPA) symbol space.

[0058] In constructing the unified phoneme vocabulary and phoneme embedding matrix, the system performs deduplication and aggregation on IPA phonemes for all languages ​​to build a unified phoneme vocabulary shared by all languages, and initializes the phoneme embedding matrix on the unified phoneme vocabulary. The phoneme embedding matrix is ​​used to map IPA symbols in the unified phoneme vocabulary into continuous vector representations, so that discrete IPA phoneme symbols enter the embedding representation space that can participate in model training.

[0059] The IPA mapping and OOV fallback process is as follows: Figure 3 As shown, after detecting the OOV symbol, the system performs backtracking processing in the order of glyph backtracking and expert rule mapping, and performs known phoneme index substitution based on the matching results. During the OOV backtracking process, OOV refers to pronunciation symbols not included in the preset phonetic symbol list, distinguishing them from rare characters at the glyph level. The OOV backtracking process uses a multi-path backtracking strategy to map OOV symbols to known phonemes to maintain the continuity of the speech synthesis or recognition task. OOV backtracking is mainly performed during the inference phase; the training data does not contain OOV symbols, and OOV substitution is used to maintain compatibility during inference.

[0060] When the system detects that the unified IPA phoneme sequence to be synthesized contains OOV symbols, it first extracts the Unicode encoding features of the OOV symbols and their corresponding basic glyphs. Since the design of IPA characters is systematic, phonetic symbols with diacritical marks are highly correlated with the original phonetic symbols in terms of glyphs. The system can initiate subsequent fallback retrieval based on character-level glyph and semantic analysis. The fallback retrieval includes Path A and Path B. Path A is glyph fallback, which is triggered when the OOV symbol is a decomposable composite glyph and there is at least one subpart with a known phoneme mapping; the system extracts subpart phonemes through glyph decomposition and combines them into a complete phoneme using expert rules. Path B is expert rule mapping, which is triggered when the OOV symbol is an independent symbol without glyph rules; the system calls a predefined rule library, including a dialect-IPA对照表 and an ancient sound-modern sound mapping table, to directly match the OOV symbol to a known phoneme.

[0061] In terms of fallback priority, Path A takes precedence over Path B. If the subparts after Path A decomposition are still OOV, Path B is automatically triggered; if both Path A and Path B fail, the default placeholder "" is output and the OOV symbol and context log are recorded. The system uses the matched known phoneme index Ir to replace the original OOV position, and its index redirection can be expressed as:

[0062] where represents a retrieval function based on glyph similarity or rule tables. Through this character-level fallback processing, when the model encounters un-trained pronunciation symbols, it can obtain the corresponding initialization representation from the unified phoneme space, avoiding output anomalies or silence caused by unknown indices.

[0063] In the OOV fallback case, for the Chinese character "䶮", Path A is adopted and the phoneme [j ŋ] is matched. The reason for the match is that the glyph is decomposed into "dragon +厶", "dragon" corresponds to [lʊŋ], "厶" corresponds to [s], and they are combined into [j ŋ] using expert rules. For the English word "quz", Path B is adopted and the phoneme [kw z] is matched. The reason for the match is that in the expert rules, "qu" is fixedly mapped to [kw], "z" is mapped to [z], and a default vowel is inserted in the middle to complete the syllable. For the Uyghur word " ", Path B is adopted and the phoneme [ɑ] is matched. The reason for the match is that this symbol is a low vowel in Old Uyghur and is directly mapped to the basic vowel [ɑ] in Modern Uyghur.

[0064] The vector quantization alignment architecture is as shown in Figure 4As shown, continuous acoustic features are processed through nonlinear dimensionality reduction and resampling before being fed into a discrete codebook. Discrete semantic tokens are obtained through nearest neighbor search and used to establish alignment with the IPA embedding vectors. During acoustic feature extraction and vector quantization, the system receives high-dimensional continuous acoustic features Hw∈R^(T×D) extracted by a pre-trained acoustic encoder. Since the original acoustic features contain redundant information such as ambient background noise and minute phase fluctuations, the system uses a set of one-dimensional convolutional layers or residual connective blocks to perform temporal resampling and nonlinear transformation on the features, mapping them to latent acoustic features z(x). The nonlinear dimensionality reduction and resampling network consists of three one-dimensional convolutional layers: the first layer has a kernel size of 3, a stride of 2, an input dimension of 512, and an output dimension of 256, uses ReLU activation, and has no residual connections; the second layer has a kernel size of 3, a stride of 2, an input dimension of 256, and an output dimension of 128, uses ReLU activation, and has residual connections; the third layer has a kernel size of 3, a stride of 1, an input dimension of 128, and an output dimension of 64, uses Tanh activation, has no residual connections, and has an output dimension of 64, thus achieving dimensionality reduction from 512 to 64 dimensions.

[0065] In the process of discrete codebook matching and pronunciation unit quantization, the system introduces a predefined discrete codebook. This contains K d-dimensional center vectors, each representing a code center and a typical human vocal acoustic pattern. The system targets each continuous vector after dimensionality reduction. Perform a nearest neighbor search in the discrete codebook to find the codecenter with the smallest Euclidean distance. The nearest neighbor search relationship can be represented as:

[0066] In the specific implementation, the codebook size K=1024, the code center dimension d=256, and the initialization method is to use the k-means clustering algorithm to cluster the acoustic features of the pre-trained wav2vec 2.0 model and initialize the code center vector. Through this quantization process, continuous acoustic features are quantized into a series of discrete semantic tokens.

[0067] In the joint constraint optimization and spatial alignment process, the system introduces a multi-constraint loss function for optimization to reduce feature loss or spatial divergence during quantization. First, the system uses codebook loss to calculate the distance between the acoustic features output by the encoder and the selected codebook center, and applies gradient blocking to the acoustic features; codebook loss is used to update the center vector in the codebook, making the code center closer to the true acoustic distribution. Second, the system uses commitment loss to calculate the distance between the acoustic feature z(x) output by the encoder and the selected codebook center e, and applies gradient blocking to the code center to constrain the features output by the encoder to align to a preset discrete code center. The vector quantization loss is expressed as:

[0068] in, This is a gradient blocking operation. To adjust the balance coefficient of the constraint strength, Take 0.25, This represents the acoustic characteristics of the encoder output. This indicates that a codebook center has been selected.

[0069] In establishing the alignment between IPA embedding vectors and discrete semantic tokens, the system trains a multilayer perceptron (MLP) through a proxy task based on mask prediction. The MLP takes the IPA embedding vector as input (128 dimensions) and outputs the probability distribution of the codebook index, passing the probability values ​​through a softmax layer. The training objective is to minimize the cross-entropy loss, maximizing the probability values ​​of the IPA embedding vector and its corresponding acoustic code, thereby automatically learning the probabilistic correspondence between the IPA embedding vector and the discrete semantic token.

[0070] In the process of constructing a unified phoneme space across languages, the system, based on the implicit alignment of acoustic features and phonemes, forces IPA symbols with the same or similar pronunciations to cluster in the embedding space through feature matching loss, and simultaneously updates the phoneme embedding matrix and generator parameters. The total loss function is:

[0071] in, The Mel spectrum reconstruction loss is used to monitor spectral differences; To combat the loss, it is used to improve the naturalness of the speech; The feature matching loss is used to minimize the distance between identical phonemes across languages; The vector quantization loss includes codebook loss and commitment loss. During training in this implementation, , , , .

[0072] During the training phase, the system constructs a joint loss function. The phoneme embedding matrix E, the speech generator, and the discriminator are updated synchronously. The speech generator, discriminator, and unified phoneme space are connected through the following mechanism: At the input layer, the unified phoneme space maps discrete IPA symbols into high-dimensional continuous vectors through the phoneme embedding matrix. The speech generator uses these IPA embedding vectors as input, rather than the original text or language-specific phonemes. On the generator side, after receiving the phoneme embedding vector sequence, the generator converts it into a Mel spectrogram through the decoder, and then generates the final speech waveform through the vocoder. The discrete semantic tokens generated by vector quantization constrain the alignment relationship between acoustic features and phoneme embeddings. On the discriminator side, the discriminator distinguishes between real speech and generated speech, making the generator output acoustic features close to natural speech. It also extracts intermediate layer features between generated speech and real speech through feature matching loss, minimizing the difference between the two, so that the generated speech conforms to the clustering structure of the unified phoneme space at the feature level.

[0073] The training process employs a phased approach. In the first phase, the unified phoneme space embedding is pre-trained, along with the vector quantization mechanism, including codebook clustering and dimensionality reduction networks. All other modules are frozen during this phase. In the second phase, the generator, discriminator, and vector quantization module are jointly trained end-to-end, and all parameters are updated. Through this phased training process, the system optimizes the generator, discriminator, and phoneme embedding matrix: the Mel-spectral reconstruction loss (Lmel) constrains the generated speech's spectrum to match that of real speech, while the vector quantization loss (Lv) constrains the correspondence between discrete semantic tokens and phoneme embeddings, and correlates the generator's output features with the embedding distribution of the unified phoneme space.

[0074] During the cross-linguistic clustering feature verification process, the system periodically extracts the feature vectors of the phoneme embedding matrix E for manifold learning dimensionality reduction analysis during training. t-SNE is used to reduce the dimensionality of the high-dimensional embedding vectors, followed by KMeans clustering with a cluster size of k=10. The system quantifies the clustering quality using silhouette coefficients, which are expressed as:

[0075] in, Indicates sample The average distance between it and similar samples Indicates sample The average distance between the unit and the nearest other class samples. The average intra-class distance of similar vocal units increases with the number of training steps. Reducing the inter-class distance between different pronunciation categories Maintain stability. This analysis is used to observe the aggregation state of IPA phonemes with similar physical articulation characteristics in the embedding space.

[0076] In the process of multilingual IPA mapping and model training, the input languages include Chinese, English, Russian, Uyghur, etc. The system uses a multilingual G2P tool to uniformly map them into IPA phoneme sequences, construct a symbol vocabulary, and train the VITS model. The IPA phoneme mapping relationships among Chinese, English, and Uyghur are as follows: The k in the Chinese word "卡(kǎ)" is mapped to / k / , and the mapping description is common mapping: voiceless stop; The k in the English word "cat" is mapped to / k / , and the mapping description is common mapping: voiceless stop; The th in the English word "think" is mapped to / θ / , and the mapping description is specific mapping: interdental fricative; The Arabic letter in the Uyghur word "qet, firm" is mapped to / q / ~ / k / , and the mapping description is common mapping: uvular voiceless stop to velar voiceless stop; The Arabic letter in the Uyghur word "tel s, search" is mapped to / t / , and the mapping description is specific mapping: voiceless stop, and is associated with the phonetic change of / θ / .

[0077] The input format of the training dataset is as Figure 6 shown, the output format of IPA extraction from the dataset is as Figure 7 shown, the result of the VITS model extracting semantic tokens is as Figure 8 shown, and the symbol registration table is as Figure 9 shown. The training dataset is sourced from the CommonVoice open-source dataset. The format of each data contains four core fields: {text, speech waveform, language label, speaker ID}. The sampling rate of the text-to-speech paired data is uniformly 16 kHz, and the audio duration is controlled within the range of 3 seconds to 10 seconds. The speaker information covers 14 languages, with 50 people for each language, totaling 7,000 people; the male-female ratio is maintained at 1:1; each person provides 100 samples, covering different age groups from 18 to 60 years old and diverse accents. According to the training data duration, it is divided into two categories: high-resource languages are those with a data duration ≥ 100 hours, including Chinese and English; low-resource languages are those with a data duration < 10 hours, including Uyghur and Burmese.

[0078] Taking a Chinese sample as a complete data sample, the initial text is "我爱自然语言处理", the language label is zh-CN, the G2P output is wo3 ai4 zi4 ran2 yu3 yan2 chu3 li3, and the IPA sequence is [wɔ aɪ s an y jæn t h u li The corresponding audio is a 16kHz mono WAV file, 4.2 seconds long, with VQToken output as [128, 45, 203, 91, 342, 189, 501, 276, 103, 412]. The input for the inference phase includes text, target speaker audio, and target language tag, where the target speaker audio is a 3-5 second reference speech, and the target language tag can be en-US. The output for the inference phase is the target speaker's speech waveform, which is 16kHz and matches the duration of the input text.

[0079] The visualization analysis results of the phoneme embedding matrix E are as follows: Figure 10 As shown, the visualization analysis results of the phoneme embedding matrix (IPA) distribution are as follows: Figure 11 As shown. To analyze the clustering of the unified IPA phoneme embedding space, clustering and visualization analysis were performed on the phoneme embedding matrix E after model training on a small dataset. The small dataset includes 14 languages: Chinese, Japanese, Korean, Russian, Uyghur, Hungarian, Mongolian, Thai, Lao, Vietnamese, Indonesian, Bengali, English, Tamil, and Arabic, with 5000 data points for each language. The system uses t-SNE to reduce the dimensionality of the high-dimensional embedding vectors, and then uses KMeans for clustering, k=10. In the two-dimensional visualization results, each point represents a phoneme embedding vector, and different colors represent different clustering categories. Among the clustering evaluation metrics, the silhouette score reaches 0.004 on the small sample dataset of 5000 data points for each of the 14 languages, while the traditional language-independent model is -0.12. The silhouette score ranges from [-1, 1]. A semantic distance metric is constructed by using the physical pronunciation attributes of IPA symbols, which enables cross-lingual phonemes of the same type to automatically cluster into neighborhoods in the embedding space. The vector quantization mechanism discretizes continuous acoustic features into discrete semantic tokens that match the dimensions of IPA phonemes, and achieves bidirectional alignment between phoneme symbols and acoustic features through code center vector constraints.

[0080] Discrete semantic token frequency distribution histogram as follows Figure 12As shown, the horizontal axis represents Token ID, and the vertical axis represents frequency of occurrence, illustrating the distribution of discrete semantic tokens output by the quantizer. To further analyze the correspondence between semantic tokens and phoneme tokens, this implementation performs statistical distribution and visualization analysis on the token sequences output by the quantizer. In the histogram of discrete semantic token frequency distribution, the horizontal axis represents Token ID, ranging from 0 to 1024, and the vertical axis represents frequency of occurrence. The token distribution is generally relatively uniform, with local high-frequency peaks, and low-frequency tokens corresponding to specific or rare pronunciation features of the language.

[0081] In the process of semantic statistical analysis, let the discrete semantic token set be:

[0082] Where N=1024 is the total size of the codebook, assuming Indicates the first The probability of a token appearing in all speech segments is represented by the information entropy as:

[0083] Information entropy is used to measure the uncertainty of token distribution. If all tokens appear with equal probability, then...

[0084] The result of this implementation is H=9.14 bits.

[0085] Utilization rate is expressed as:

[0086] Utilization rate is used to characterize the activation ratio of the codebook. The results of this implementation are as follows: .

[0087] The frequency variance is expressed as:

[0088] Frequency variance is used to measure the smoothness of a distribution. The results of this implementation show... Smaller.

[0089] Using the above statistics, the discrete semantic token sequence output by the quantizer can be used to characterize common pronunciation structures and specific language pronunciation variations in multilingual speech.

[0090] In this implementation, during the inference phase, the system receives the standardized text to be synthesized and the target language tag. It then generates a unified IPA phoneme sequence to be synthesized using a multilingual G2P tool and a preset IPA mapping function. When the OOV symbol is absent from the unified IPA phoneme sequence, the system calls the corresponding IPA embedding vector based on the cross-language unified phoneme space and phoneme embedding matrix. When the OOV symbol is present, the system maps the OOV symbol to a known phoneme index in the unified phoneme vocabulary through OOV backtracking and calls the phoneme embedding vector corresponding to that known phoneme index. Subsequently, the system generates speech from the unified IPA phoneme sequence after OOV backtracking based on the cross-language unified phoneme space, outputting the target speech waveform and structured alignment information.

Claims

1. A cross-lingual speech generation method based on IPA phoneme mapping and vector quantization alignment, characterized by, The method includes: S1. Obtain multilingual text-speech pair data during the training phase and the text to be synthesized and target language labels during the inference phase, and perform standardization processing on the text in the multilingual text-speech pair data and the text to be synthesized. S2. Convert the standardized text into a phoneme sequence within the corresponding language, and call the preset IPA mapping function to map the phoneme sequence within the corresponding language into a unified IPA phoneme sequence. S3. Construct a unified phoneme vocabulary shared by all languages ​​based on the unified IPA phoneme sequence, and construct a phoneme embedding matrix based on the unified phoneme vocabulary; S4. Perform acoustic feature extraction and vector quantization on the speech waveforms in the multilingual text-speech pair data to generate discrete semantic tokens corresponding to the acoustic features. S5. Based on the phoneme embedding matrix and discrete semantic token, establish an alignment relationship between the IPA embedding vector and the discrete semantic token, and update the phoneme embedding matrix according to the alignment relationship, so that IPA symbols with the same or similar pronunciations are clustered in the embedding space to obtain a cross-language unified phoneme space for speech generation. S6. In the inference stage, based on the target language tag, the standardized text to be synthesized is converted into a unified IPA phoneme sequence to be synthesized. When there is an OOV symbol in the unified IPA phoneme sequence to be synthesized, OOV back-off processing is performed on the OOV symbol, the OOV symbol is mapped to the known phoneme index in the unified phoneme vocabulary, and the phoneme embedding vector corresponding to the known phoneme index is called. S7. Based on the cross-language unified phoneme space, perform speech generation on the unified IPA phoneme sequence to be synthesized after OOV back-off processing, and output the target speech waveform and structured alignment information.

2. The method according to claim 1, characterized in that, The multilingual text-speech pair data in S1 includes text, speech waveforms, language tags, and speaker IDs; the standardization process includes: scanning the text and the text to be synthesized using a preset regular expression library, filtering out non-standard characters, and performing regular expression expansion on non-text entities such as numerical values, percentages, and times.

3. The method according to claim 2, characterized in that, S2, which describes converting the standardized text into a phoneme sequence within the corresponding language, includes: converting the standardized text into a phoneme sequence within the corresponding language by using a multilingual G2P tool that integrates mapping rule bases for different language families, based on the language tag.

4. The method according to claim 1, characterized in that, The preset IPA mapping function in S2 performs mapping based on the IPA mapping index table. The input of the IPA mapping index table is the original phoneme symbols of a specific language, and the output is the corresponding International Phonetic Alphabet symbols. The IPA mapping index table includes phoneme mapping rules and mapping types within the language. The mapping types include one-to-one mapping and many-to-one mapping.

5. The method according to claim 1, characterized in that, The acoustic feature extraction and vector quantization processing described in S4 includes: extracting continuous acoustic features from the speech waveform using a pre-trained acoustic encoder, performing nonlinear dimensionality reduction and resampling on the continuous acoustic features to obtain latent acoustic features, and projecting the latent acoustic features onto a discrete codebook through a vector quantization mechanism to generate discrete semantic tokens; wherein the pre-trained acoustic encoder is a pre-trained self-supervised acoustic model, and the pre-trained self-supervised acoustic model includes a wav2vec 2.0 encoder or an XLS-R encoder, and the sampling rate of the speech waveform is 16kHz.

6. The method according to claim 5, characterized in that, The nonlinear dimensionality reduction and resampling consists of three one-dimensional convolutional layers. The first layer has a kernel size of 3, a stride of 2, an input dimension of 512, an output dimension of 256, and uses ReLU activation. The second layer has a kernel size of 3, a stride of 2, an input dimension of 256, an output dimension of 128, and uses ReLU activation and residual connections. The third layer has a kernel size of 3, a stride of 1, an input dimension of 128, an output dimension of 64, and uses Tanh activation. The discrete codebook includes multiple code centers. The codebook size is 1024, the code center dimension is 256, and the code centers are initialized by k-means clustering of the acoustic features of the pre-trained acoustic encoder.

7. The method according to claim 5, characterized in that, The generation of discrete semantic tokens includes: performing a nearest neighbor search on the latent acoustic features in the discrete codebook, and determining the corresponding discrete semantic token based on the code center with the smallest Euclidean distance.

8. The method according to claim 5, characterized in that, S5 establishes the alignment relationship between the IPA embedding vector and the discrete semantic token, including: establishing a probabilistic correspondence between the IPA embedding vector and the codebook index in the discrete codebook through a multilayer perceptron, wherein the input of the multilayer perceptron is the IPA embedding vector and the output is the probability distribution of the codebook index, and the probability value of the IPA embedding vector and the corresponding acoustic code is maximized by minimizing the cross-entropy loss.

9. The method according to claim 1, characterized in that, S5 describes updating the phoneme embedding matrix, which includes: simultaneously optimizing the phoneme embedding matrix, speech generator, discriminator, and discrete codebook through a joint loss function, so that IPA symbols with the same or similar pronunciations are clustered in the embedding space; the joint loss function includes Mel spectrum reconstruction loss, adversarial loss, feature matching loss, and vector quantization loss, wherein the feature matching loss is used to minimize the feature distance of the same phoneme across languages, and the vector quantization loss includes codebook loss and commitment loss.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 9.

Images