Dialect speech synthesis method based on phoneme contrast energy learning, terminal and medium

Abstract

The present application relates to the technical field of speech synthesis, and discloses a dialect speech synthesis method based on phoneme contrast energy learning, a terminal and a medium. The speech synthesis method first acquires a standard language text and converts it into a standard language phoneme sequence; a dialect phoneme set is constructed based on audio samples of the dialect; then, using a trained phoneme contrast energy model, the energy difference between each standard language phoneme and each dialect phoneme is calculated, and the dialect phoneme with the smallest energy gap is selected as the mapping result for each standard language phoneme; then, a dialect phoneme sequence with the same length as the standard language phoneme sequence is generated according to the mapping result; finally, the dialect phoneme sequence is converted into a target dialect speech waveform. The energy difference is used to represent the similarity between phoneme features, and is obtained by joint calculation based on the distance of phonemes in the feature space, the pronunciation manner feature and the pronunciation position feature. The present application can realize efficient and accurate dialect speech synthesis under low resource conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a dialect speech synthesis method based on phoneme contrast energy learning, and a computer terminal and a computer readable storage medium applying the method. BACKGROUND

[0002] Dialect speech synthesis technology is an important branch of natural language processing and speech synthesis, and has important value for protecting language and cultural diversity, improving human-computer interaction experience, and serving dialect area users. With the popularity of intelligent devices, dialect speech synthesis has wide application needs in the fields of intelligent customer service, dialect translation, cultural protection, and localized content production. Compared with standard language synthesis, dialect speech synthesis not only needs to present the pronunciation, tone and rhythm characteristics unique to dialects, but also needs to accurately express the language structure and expression habits unique to dialects, which puts higher requirements on speech synthesis technology. At present, mainstream speech synthesis systems are mainly optimized for standard languages, and the quality and diversity of dialect speech synthesis still need to be improved, especially for small languages and endangered dialects, the application of speech synthesis technology is particularly urgent.

[0003] However, the main challenge of dialect speech synthesis is the extreme scarcity of training data. Unlike standard languages, which usually have tens to hundreds of hours of high-quality recording data, most dialects only have a small amount of labeled data, and some even have only a few minutes or sentences of samples. The current mainstream methods to solve the problem of low-resource dialect synthesis mainly include: (1) transfer learning method, which fine-tunes the model on dialects after pre-training on standard languages; (2) multi-language shared representation method, which utilizes the common characteristics of multiple languages or dialects; (3) speech conversion method, which converts standard language to dialect speech; (4) few-shot learning method, which tries to extract dialect features from limited samples. Although these methods have reduced the data requirements to some extent, it is still difficult to synthesize high-quality speech with accurate dialect characteristics in the extreme sample scenario (such as only 1-10 sentences of dialect samples), especially in terms of phoneme pronunciation and rhythm characteristics. SUMMARY

[0004] To solve the technical problems existing in the prior art, the present application provides a dialect speech synthesis method based on phoneme contrast energy learning, terminal and medium, which quantitatively characterizes the correspondence between standard language phonemes and dialect phonemes through a phoneme contrast energy model, and can capture the subtle differences between standard language and dialect phonemes from limited samples, thereby realizing efficient and accurate dialect speech synthesis under low-resource conditions.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] The application discloses a dialect speech synthesis method based on phoneme contrast energy learning, comprising the following steps:

[0007] standard language text is acquired and converted into a standard language phoneme sequence, which comprises features of all standard language phonemes in the text;

[0008] a dialect phoneme set is constructed based on a dialect audio sample, which comprises features of all dialect phonemes;

[0009] an energy difference between each standard language phoneme and each dialect phoneme is calculated by using a trained phoneme contrast energy model, and a dialect phoneme with the smallest energy difference is selected as a mapping result for each standard language phoneme;

[0010] a dialect phoneme sequence with the same length as the standard language phoneme sequence is generated according to the mapping result;

[0011] the dialect phoneme sequence is converted into a target dialect speech waveform;

[0012] wherein the energy difference is used to represent the similarity between phoneme features, and is obtained by jointly calculating the distance between phonemes in a feature space, pronunciation manner features and pronunciation position features.

[0013] As a further improvement of the above-mentioned scheme, the phoneme contrast energy model is trained by a triplet contrast learning strategy.

[0014] wherein the training set comprises several triplets, each triplet is composed of an anchor point, a positive sample and a negative sample; the anchor point is a standard language phoneme feature; for each anchor point, the positive sample is a dialect phoneme feature corresponding to the semantic, and the negative sample is a dialect phoneme feature not corresponding to the semantic and belonging to a different dialect category from the positive sample.

[0015] As a further improvement of the above-mentioned scheme, the following loss function is used in the process of training the phoneme contrast energy model.

[0016] ;

[0017] wherein, is the total loss; is the contrast loss; is the reconstruction loss, is the regularization loss, and are corresponding loss weights;

[0018] wherein, the expression of the contrast loss is as follows:

[0019] ;

[0020] wherein, maxdenotes taking the maximum value; a phoneme feature of an anchor point, a phoneme feature of a positive sample, a phoneme feature of a negative sample; a phoneme feature and an energy difference between; a phoneme feature and an energy difference between; a boundary parameter for controlling the energy difference between a positive sample and a negative sample pair;

[0021] The expression of the reconstruction loss is as follows:

[0022] ;

[0023] In the formula, |·|2 is an L2 norm; is an acoustic decoder; is a feature of any one phoneme in a dialect phoneme set i ; is a corresponding audio segment;

[0024] The expression of the regularization loss is as follows:

[0025] ;

[0026] As a further improvement of the above scheme, in the process of training the phoneme contrast energy model, an adaptive learning rate gradient descent algorithm is used to optimize the model parameters, and a meta-learning strategy is introduced to accelerate the convergence.

[0027] As a further improvement of the above scheme, a hard example mining strategy is used when training the phoneme contrast energy model, and negative samples with the smallest energy difference with the anchor point phoneme feature are preferentially selected to construct a triple, and the expression is as follows:

[0028] ;

[0029] In the formula, is a feature of a dialect phoneme ; is an energy difference between a phoneme feature and .

[0030] As a further improvement of the above scheme, the function definition of the phoneme contrast energy model is as follows:

[0031] ;

[0032] In the formula, characteristics of any phoneme i characteristics of any phoneme characteristics of any phoneme j characteristics of any phoneme ; is the Euclidean distance term, which measures the distance between the phoneme characteristics and in the feature space, |·|2 is the L2 norm; and are regularization coefficients; is the pronunciation manner regularization term, is the pronunciation position regularization term, and the expression is as follows:

[0033] ;

[0034] ;

[0035] In the formula, K 1 is the total number of pronunciation manner characteristics; is the weight parameter of the k th pronunciation manner characteristic; is the k th pronunciation manner characteristic extraction function; |·| is the absolute value symbol; K 2 is the total number of pronunciation position characteristics; is the weight parameter of the th pronunciation position characteristic; is the th pronunciation position characteristic extraction function.

[0036] As a further improvement of the above scheme, by performing word segmentation and phoneme decomposition on the standard language text, and by performing noise reduction, segmentation, and phoneme alignment on the audio samples of the dialect, the corresponding phoneme characteristics are extracted using a pre-trained general speech model;

[0037] Among them, all the phoneme characteristics of the standard language constitute the standard language phoneme sequence, denoted as , denotes the t th standard language phoneme characteristic, , T is the total number of phoneme characteristics in the standard language phoneme sequence ; all the audio samples of the dialect constitute the dialect phoneme set, denoted as , denotes the m th dialect phoneme characteristic, , M is the total number of phoneme characteristics in the dialect phoneme set ;

[0038] The standard language phoneme is selected as the mapping result, and the expression is as follows:

[0039] ;

[0040] In the formula, The first dialect phoneme feature in the dialect phoneme sequence is represented as ; t The first dialect phoneme feature in the dialect phoneme sequence is represented as The energy difference between the phoneme features and .

[0041] As a further improvement of the above scheme, after the dialect phoneme sequence is generated, a dynamic programming algorithm is further introduced to optimize the sequence, and the optimized dialect phoneme sequence is converted into the target dialect speech waveform; wherein the expression of the optimization process is as follows:

[0042] ;

[0043] In the formula, The energy difference between the phoneme features and ; The transition cost between adjacent phoneme features.

[0044] The application further discloses a computer terminal, including a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program, the steps of the dialect speech synthesis method based on phoneme contrast energy learning are realized.

[0045] The application further discloses a computer readable storage medium, which stores a computer program, characterized in that when the program is executed by a processor, the steps of the dialect speech synthesis method based on phoneme contrast energy learning are realized.

[0046] Compared with the prior art, the application has the following beneficial effects:

[0047] 1、The application greatly reduces the amount of training data required for dialect synthesis by introducing a phoneme contrast energy learning mechanism, so that high-quality dialect speech synthesis can be realized under the condition of a small number of dialect samples, overcoming the shortcomings of strong data dependence of the prior art in a low-resource scenario. The phoneme contrast energy model constructed can accurately quantify and learn the relationship between different dialect phonemes, greatly improving the accuracy of dialect phoneme mapping compared with the traditional method of relying only on direct acoustic feature imitation, so that the synthesized speech has more real and accurate dialect feature performance.

[0048] Through the innovative technology of phoneme contrast energy learning, the data efficiency, phoneme accuracy, computing performance, generalization ability and practicality are comprehensively improved, and the key technical problems in the field of low-resource dialect synthesis are effectively solved, providing more efficient and practical technical support for dialect protection and speech application.

[0049] 2、The energy difference minimization optimization strategy adopted by the present application simplifies the model structure and training process, has higher computing efficiency and lower resource consumption compared with existing deep learning methods, reduces the complexity and deployment difficulty of the system. Based on the innovative design of the phoneme contrast energy learning framework, the present application establishes a unified energy space representation between different dialects, enhances the cross-dialect generalization ability of the system, makes the adaptation process of new dialects more efficient, and even can produce speech output with certain dialect characteristics under zero sample condition. At the same time, the present application combines the constraint regularization method of energy prior, introduces the physical characteristics of phoneme pronunciation as the constraint condition of energy function, and through adjusting the energy function parameters, the continuous control of dialect feature intensity can be realized, which enhances the flexibility and adaptability of the system.

[0050] 3、The present application trains through the triple contrast learning strategy, establishes an energy learning mechanism based on contrast sample pairs, and guides the model to learn the similarity and difference between phonemes through the construction of positive and negative sample pairs, so as to realize effective modeling of phoneme feature space. By taking different pronunciation samples of the same phoneme category as positive sample pairs and taking samples of different phoneme categories as negative sample pairs, the energy difference between positive sample pairs is minimized and the energy difference between negative sample pairs is maximized, so that the model can accurately capture the essential features of phonemes and improve the accuracy of cross-language phoneme mapping.

[0051] 4、The present application designs a contrast loss function, combines with multiple optimization objectives such as reconstruction loss, and forms a comprehensive training strategy. The contrast loss function is used to narrow the energy representation of similar phonemes and push away the energy representation of different phonemes, so that the phonemes form a clear clustering structure in the feature space; the reconstruction loss ensures the integrity and restorability of the phoneme features, and ensures that the learned phoneme representation contains enough acoustic information for subsequent speech synthesis. This multi-loss joint optimization method can effectively balance the phoneme distinguishability and the integrity of the phoneme representation, and improve the accuracy and stability of the model in the process of mapping standard language phonemes to dialect phonemes.

[0052] 5. This invention constructs a multi-dimensional energy difference calculation method by jointly considering the distance of phonemes in the feature space, articulation features, and articulation place features. The distance in the feature space reflects the acoustic similarity of phonemes, the articulation features reflect the similarity of the phoneme generation mechanism, and the articulation place features characterize the similarity of the articulation organ configuration of the phonemes. This allows for a more comprehensive measurement of the similarity between standard language phonemes and dialect phonemes, improving the accuracy of phoneme mapping. This multi-dimensional joint calculation method enables the model to effectively utilize the linguistic knowledge and acoustic characteristics of phonemes, even with limited dialect training data, to achieve stable and reliable dialect speech synthesis. Attached Figure Description

[0053] Figure 1 This is a flowchart of the dialect speech synthesis method based on phoneme contrastive energy learning in Embodiment 1 of the present invention.

[0054] Figure 2 This is a diagram showing the flow of example data in the model in Embodiment 1 of the present invention.

[0055] Figure 3 This is a schematic diagram of the structure of the computer terminal in Embodiment 2 of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1

[0058] Please see Figure 1 and Figure 2 This embodiment provides a dialect speech synthesis method based on phoneme contrast energy learning, including the following steps, namely S1~S5.

[0059] S1. Obtain standard language text and convert it into a standard language phoneme sequence, which includes the features of all standard language phonemes in the text.

[0060] S2. Construct a dialect phoneme set based on dialect audio samples, which includes features of all dialect phonemes.

[0061] In steps S1 and S2, the standard language text is segmented and phoneme decomposed, and the dialect audio samples are denoised, segmented, and phoneme aligned, thereby employing a pre-trained general speech model. Extract the corresponding phoneme features, and represent them as ,in phonetic features corresponding audio segment. In order to enhance the feature extraction ability in a low-resource environment, an attention mechanism at the phoneme level can be used to extract key features from limited samples.

[0062] Each phoneme is represented as a D dimensional feature vector , R is a real set. For a standard language phoneme sequence composed of all phonetic features of a standard language, it is represented as , represents the t th standard language phonetic feature, , T is the total number of phonetic features in the standard language phoneme sequence . For a dialect phoneme set composed of all audio samples of a dialect, it is represented as , represents the m th dialect phonetic feature, , M is the total number of phonetic features in the dialect phoneme set .

[0063] S3. Using the trained phoneme contrast energy model, the energy difference between each standard language phoneme and each dialect phoneme is calculated, and the dialect phoneme with the smallest energy difference is selected as the mapping result for each standard language phoneme.

[0064] wherein the energy difference is used to represent the similarity between phonetic features, and is obtained based on the joint calculation of the distance of phonemes in the feature space, the pronunciation manner feature and the pronunciation site feature.

[0065] The function definition of the phoneme contrast energy model is as follows:

[0066] ;

[0067] In the formula, is the energy difference between the feature i of any phoneme and the feature j of any phoneme ; is the Euclidean distance term, used to measure the distance of phonetic features and in the feature space, |·|2 is the L2 norm; and are regularization coefficients; is the pronunciation manner regularization term, is the pronunciation site regularization term, and the expression is as follows:

[0068] ;

[0069] ;

[0070] wherein, K 1 is the total number of pronunciation manner features; is the weight parameter of the k th pronunciation manner feature; is the extraction function of the k th pronunciation manner feature; |·| is an absolute value symbol; K 2 is the total number of pronunciation position features; is the weight parameter of the th pronunciation position feature; is the extraction function of the th pronunciation position feature. The phoneme feature is the input voice, and the voice content can be stripped out of the different pronunciation manners of each person (such as some male tenors and female altos) through a specific function. The pronunciation position can also be found through a specific function (the sound is produced by the joint action of 3 large areas and 10 muscles).

[0071] Further, the phoneme contrast energy model is trained by adopting a triplet contrast learning strategy.

[0072] The training set includes a plurality of triplets, each triplet is composed of an anchor point, a positive sample and a negative sample; the anchor point is a standard language phoneme feature; for each anchor point, the positive sample is a dialect phoneme feature corresponding to the semantics, and the negative sample is a dialect phoneme feature not corresponding to the semantics and belonging to a different dialect category from the positive sample. Based on this, a contrast loss function is designed:

[0073] ;

[0074] wherein, max {·,·} represents taking the maximum value; is the phoneme feature of the anchor point, is the phoneme feature of the positive sample, is the phoneme feature of the negative sample; is the energy difference between the phoneme features and ; is the energy difference between the phoneme features and ; is a boundary parameter for controlling the energy difference between the positive and negative sample pairs.

[0075] In order to improve the learning efficiency, a difficult example mining strategy is adopted to preferentially select the negative samples with small energy gaps:

[0076] ;

[0077] wherein, is a dialectal phoneme feature; is a phoneme feature and energy difference between them. argmin represents the value of the independent variable when the objective function takes the minimum value.

[0078] When the energy difference value is smaller, it indicates that the two phonemes are more similar in the feature space and linguistic properties, which can be used as the basis for pairing anchor samples with positive samples. This design not only guarantees the geometric similarity of the feature space, but also incorporates linguistic prior knowledge, improving the accuracy of phoneme matching.

[0079] In the process of training the phoneme contrast energy model, the following loss function is used:

[0080] ;

[0081] wherein, is the total loss; is the reconstruction loss, is the regularization loss, and are the corresponding loss weights;

[0082] The reconstruction loss is used to ensure that the mapped phonemes can reconstruct the original acoustic features, and the expression is as follows:

[0083] ;

[0084] wherein, |·|2 is the L2 norm; is an acoustic decoder; is the feature of any one phoneme i in the dialectal phoneme set, is the corresponding audio segment;

[0085] The regularization loss is used to prevent overfitting, and the expression is as follows:

[0086] ;

[0087] Further, the optimization process uses a gradient descent algorithm with an adaptive learning rate:

[0088] ;

[0089] wherein, is the model parameter, is the learning rate, is the total loss function on the parameter The gradient.

[0090] In the case of very few samples, a meta-learning strategy is introduced to accelerate convergence:

[0091] ;

[0092] ;

[0093] In the formula, and These are the loss functions for the support set and the query set, respectively.

[0094] At this point, the complete dialect speech synthesis process is as follows: given the input text, it is first converted into a standard language phoneme sequence through text analysis. Then, based on the learned phoneme contrast energy model, the dialect phoneme with the smallest energy difference for each standard language phoneme is selected as the mapping result, represented as:

[0095] ;

[0096] In the formula, Represents the first phoneme in the dialect phoneme sequence t Dialectal phoneme characteristics; Phoneme characteristics and The energy difference between them.

[0097] S4. Generate a dialect phoneme sequence with the same length as the standard language phoneme sequence based on the mapping result. .

[0098] In generating the dialect phoneme sequence Subsequently, considering the influence of context, a dynamic programming algorithm was introduced to optimize the sequence, resulting in the optimized dialect phoneme sequence. Convert to the target dialect speech waveform; the expression for the optimization process is:

[0099] ;

[0100] In the formula, Phoneme characteristics and The energy difference between them; This represents the conversion cost between adjacent phoneme features.

[0101] S5. Convert the dialect phoneme sequence into the target dialect speech waveform. An acoustic model can be used to convert the optimized dialect phoneme sequence into the final dialect speech waveform.

[0102] In this embodiment, taking Mandarin to Sichuan dialect conversion as an example, the specific process of implementing the method of the application is as follows.

[0103] First, 5 Mandarin-Sichuan dialect parallel speech samples are collected, each sample is about 10 seconds long, and contains basic speech coverage. The audio is preprocessed, including noise reduction, segmentation and phoneme alignment. The initial feature representation of each phoneme is extracted using a pre-trained general speech model, with a dimension setting of .

[0104] Then, the phoneme contrast energy function is configured, with parameter settings of , , the pronunciation manner features include 8 dimensions such as clear and dull sound, friction degree, airflow direction, etc., and the pronunciation site features include 6 dimensions such as lip shape, tongue position, glottis state, etc. Based on the parallel samples, a triple set training set is constructed, each Mandarin phoneme is used as an anchor point, the corresponding Sichuan dialect phoneme is used as a positive sample, and 5 non-corresponding dialect phonemes are randomly selected as candidate negative samples.

[0105] In the training stage, the meta-learning strategy is adopted, and the 5 samples are divided into 3 support samples and 2 query samples. The model initialization adopts the Xavier method, the learning rate is set to , the boundary parameter , the weighting coefficient , . During the training process, the hard example pool is updated once every 10 rounds, and the negative sample with the smallest energy gap is preferentially selected. After 100 rounds of training, the model converges, and the phoneme mapping accuracy reaches more than 85%.

[0106] In the inference stage, the input Mandarin text "today the weather is really good" is obtained through word segmentation and phoneme decomposition to get the Mandarin phoneme sequence. Using the trained phoneme contrast energy model, the energy value difference of each Mandarin phoneme and all Sichuan dialect phonemes is calculated, and the dialect phoneme with the smallest energy difference is selected as the mapping result. Considering the influence of context, the dynamic programming algorithm is applied to optimize the overall sequence to obtain the final Sichuan dialect phoneme sequence.

[0107] Finally, the phoneme sequence is converted into Sichuan dialect speech waveform through the acoustic decoder to generate synthesized speech with obvious Sichuan dialect characteristics.

[0108] In summary, the low-resource dialect synthesis method based on phoneme contrast energy learning proposed in the present application aims to solve the key problems of existing dialect synthesis techniques in a low-resource environment. The core innovation of the present application lies in the introduction of a phoneme contrast energy learning mechanism, which quantitatively characterizes the correspondence between standard language phonemes and dialect phonemes by constructing an energy difference function, and trains the model to capture the subtle differences between phonemes based on the contrast learning principle. This method first establishes a phoneme energy representation space, which maps phonemes to a unified vector space; then designs a special contrast energy function to measure the similarity between different phonemes; then constructs a contrast sample pair to guide the model to learn the mapping relationship between phonemes; further optimizes the model under low-resource conditions through an energy minimization strategy; finally, based on the learned phoneme contrast energy model, high-quality conversion from standard language to target dialect is achieved. The present application breaks through the limitation of traditional methods which rely on a large amount of data, and can realize accurate dialect phoneme mapping and natural speech synthesis under the condition of only a small amount of dialect samples, providing effective technical support for the protection and application of low-resource dialects.

[0109] To comprehensively evaluate the performance of the low-resource dialect synthesis method based on phoneme contrast energy learning of the present application, multiple dialect speech data sets are selected for experimental verification in this embodiment. The experimental data includes: Chinese dialect speech database (CSDSD), which contains 8 major dialects such as Mandarin and Sichuan dialect, Cantonese, Shanghai dialect, and Minnan dialect, each dialect containing 100-500 speakers of speech data; AISHELL-Dialect data set, covering dialects in 10 provinces of China; self-built extremely low-resource dialect data set, containing only 5-10 sentences of Miao, Dong, Tujia and other minority languages. To simulate the real low-resource scenario, this embodiment randomly selects 1, 3, 5, and 10 sentences from each dialect as training samples, and the rest as the test set.

[0110] This embodiment selects the current mainstream low-resource speech synthesis method as the contrast baseline, including:

[0111] 1. AdaSpeech: A self-adaptive TTS method proposed by Microsoft, which realizes fast speech cloning through conditional layer normalization.

[0112] 2. Meta-TTS: A few-shot speech synthesis method based on meta-learning, which uses the model-agnostic meta-learning (MAML) framework.

[0113] 3. TransferTTS: A dialect synthesis method based on transfer learning, which is fine-tuned after pre-training on large-scale Mandarin data.

[0114] 4. YourTTS: An end-to-end system supporting zero-shot multilingual speech synthesis, based on the VITS architecture.

[0115] 5、VALL-E: A neural codec language model proposed by Microsoft, supporting zero-shot speech synthesis.

[0116] The experimental environment is configured as 4xNVIDIA V100 (32GB) GPUs, and is implemented based on PyTorch 1.13. The batch size is 16, the initial value of the learning rate is 1e-3, and the Adam optimizer is used. All experiments are repeated 5 times to take the average value, eliminating the influence of randomness.

[0117] This embodiment first evaluates the synthesis performance of each method under different training sample quantities. Taking Sichuan dialect as an example, comprehensive evaluation is performed using objective evaluation indicators and subjective evaluation indicators, and the results are shown in Table 1.

[0118] Table 1: Performance comparison of each method on Sichuan dialect synthesis under different training sample quantities

[0119] ;

[0120] Note: In the table, the downward arrow "↓" of MCD (Mel-Cepstral Distortion) and F0 RMSE indicates that the lower the result is, the better, and the upward arrow "↑" of MOS (Mean Opinion Score) and dialect similarity (5-point scale) indicates that the higher the result is, the better.

[0121] The experimental results show that the method of the present application has significant advantages under extremely low resource conditions. In particular, when there is only one training sample, the MCD of the present method is reduced by 18.6% compared with AdaSpeech, the MOS score is increased by 35.1%, and the dialect similarity is increased by 57.7%.

[0122] To verify the generalization ability of the present method, cross-validation experiments are performed on multiple dialects in this embodiment. Using 5 training samples, the synthesis performance of each method on different dialects is evaluated, and the results are shown in Table 2.

[0123] Table 2: Comparison of synthesis performance of each method on different dialects (5 training samples)

[0124] ;

[0125] The results show that the method of the present application achieves the best performance on various dialects, with an average MCD reduction of 16.4% compared with AdaSpeech, and an MOS score increase of 23.7%. This indicates that the unified energy space established by the phoneme contrast energy learning framework effectively promotes cross-dialect knowledge transfer.

[0126] To verify the contribution of each core component, detailed ablation experiments were conducted. Using five training samples in Sichuan dialect, the results are shown in Table 3.

[0127] Table 3: Results of core component ablation experiments

[0128] ;

[0129] The ablation experiment results show that:

[0130] 1. Triplet contrastive learning is the most critical component, and the MCD increases by 24.2% after removal, verifying the core role of the contrastive learning mechanism.

[0131] 2. The pronunciation manner and pronunciation site regularization terms contribute 13.5% and 15.9% performance improvement respectively, proving the importance of phonetic prior knowledge.

[0132] 3. Although the meta-learning strategy contributes relatively small performance improvement (9.8%), it significantly reduces the training time (33.2%).

[0133] 4. Difficult negative sample mining and dynamic programming optimization contribute 6.6% and 4.8% performance improvement respectively.

[0134] This embodiment compares the computational resource consumption of each method, and the results are shown in Table 4.

[0135] Table 4: Comparison of computational resource consumption (5 training samples)

[0136] ;

[0137] The method of the present application exhibits significant advantages in computational efficiency: the training time is reduced by 67.7% compared to the fastest baseline method (Meta-TTS), the GPU memory usage is reduced by 55.6%, the inference speed is improved by 40.0%, and the model parameter quantity is reduced by 35.0%. This is mainly due to the phoneme-level energy optimization that avoids large-scale parameter updates of end-to-end models.

[0138] Finally, this embodiment invited 30 native speakers to conduct subjective evaluation of the synthesized speech. The evaluation dimensions include speech quality, dialect authenticity and overall satisfaction, and the results are shown in Table 5.

[0139] Table 5: Results of user subjective evaluation (5-point scale)

[0140] ;

[0141] The subjective evaluation result is consistent with the objective index, the method has obvious advantages over the baseline method in each subjective index, and in particular, the dialect authenticity is improved by 34.1% compared with the best baseline, which indicates that the synthesized dialect speech is highly recognized by native speakers.

[0142] Based on the above experimental results, the low-resource dialect synthesis method based on phoneme contrast energy learning proposed in the application has significant performance advantages under the condition of few samples, not only leads existing methods by a large margin in objective indicators, but also obtains high recognition of users in subjective evaluation, and has obvious advantages in computing efficiency, thereby providing a feasible technical solution for protection and application of low-resource dialects.

[0143] Embodiment 2

[0144] The embodiment provides a computer terminal, including a memory, a processor and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the steps of the dialect speech synthesis method based on phoneme contrast energy learning are realized.

[0145] As shown in Figure 3 , the computer terminal provided by the embodiment includes at least one processor 101 and a memory 102 connected with the at least one processor 101, and the specific connection medium between the processor 101 and the memory 102 is not limited in the embodiment, Figure 3 and the connection between the processor 101 and the memory 102 through the bus 100 is taken as an example. The bus 100 is represented by a thick line in Figure 3 , and the connection mode between other components is only illustrative and is not limited. The bus 100 can be divided into an address bus, a data bus, a control bus and the like, and for convenience, only one thick line is represented in Figure 3 , but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 101 can also be called a controller, and the name is not limited.

[0146] In the embodiment, the memory 102 stores instructions executable by the at least one processor 101, and the at least one processor 101 can execute the foregoing method by executing the instructions stored in the memory 102.

[0147] The processor 101 is the control center of the device, can connect each part of the whole control equipment through various interfaces and lines, and through running or executing the instructions stored in the memory 102 and calling the data stored in the memory 102, the device has various functions and processes data, thereby overall monitoring the device.

[0148] In one possible design, processor 101 may include one or more processing units. Processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 101. In some embodiments, processor 101 and memory 102 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0149] Processor 101 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the dialect speech synthesis method based on phoneme contrast energy learning disclosed in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in processor 101.

[0150] Memory 102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 102 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), and electrically erasable programmable read-only memory (EPR). Only Memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 102 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. In this embodiment, memory 102 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0151] By designing and programming the processor 101, the code corresponding to the dialect speech synthesis method described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the code during operation.Figure 1 The steps of the dialect speech synthesis method based on phoneme pair contrast energy learning are shown. How to design and program the processor 101 is known to those skilled in the art, and will not be described here.

[0152] Embodiment 3

[0153] The embodiment provides a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the steps of the dialect speech synthesis method based on phoneme pair contrast energy learning.

[0154] The computer readable storage medium can include a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage medium can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the storage medium can also include both the internal storage unit and the external storage device of the computer device. In the embodiment, the memory is usually used to store an operating system and various application software installed in the computer device. In addition, the memory can also be used to temporarily store various data that have been output or will be output.

[0155] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent replacements or changes to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered by the protection scope of the present application.

Claims

1. A dialect speech synthesis method based on phoneme contrastive energy learning, characterized in that, The method comprises the following steps: obtaining a standard language text and converting it into a standard language phoneme sequence, which comprises the features of all standard language phonemes in the text; constructing a dialect phoneme set based on dialect audio samples, which comprises the features of all dialect phonemes; using a trained phoneme contrast energy model to calculate the energy difference between each standard language phoneme and each dialect phoneme, and selecting the dialect phoneme with the smallest energy gap for each standard language phoneme as the mapping result; generating a dialect phoneme sequence consistent with the length of the standard language phoneme sequence according to the mapping result; converting the dialect phoneme sequence into a target dialect speech waveform; wherein the energy difference is used to represent the similarity between phoneme features, and is obtained by jointly calculating the distance between phonemes in the feature space, the pronunciation manner feature and the pronunciation position feature; the phoneme contrast energy model is trained by a triplet contrast learning strategy; wherein the training set comprises several triplets, each triplet consists of an anchor, a positive sample and a negative sample; the anchor is a standard language phoneme feature; for each anchor, the positive sample is a dialect phoneme feature corresponding to the semantics, and the negative sample is a dialect phoneme feature that does not correspond to the semantics and belongs to a different dialect category from the positive sample; in the process of training the phoneme contrast energy model, the following loss function is used: wherein is the total loss; is the contrastive loss; is the reconstruction loss, is the regularization loss, and are the respective loss weights. wherein the expression of the contrast loss is as follows: wherein, max {•,•} denotes taking the maximum value; is a phoneme feature of an anchor point, is a phoneme feature of a positive sample, is a phoneme feature of a negative sample; is a phoneme feature and the energy difference between; is a phoneme feature and the energy difference between; is a boundary parameter for controlling the energy difference between a positive and negative sample pair; the expression of the reconstruction loss is as follows: where | · |2is the L2norm; is an acoustic decoder; is any one of the phonemes in the dialect phoneme set i features of the'100, is corresponding audio segment; the expression of the regularization loss is as follows: 。 2. The dialectal TTS method based on phoneme-pair contrastive energy learning according to claim 1, wherein, in the process of training the phoneme contrast energy model, an adaptive learning rate gradient descent algorithm is used to optimize the model parameters, and a meta-learning strategy is introduced to accelerate convergence.

3. The dialectal TTS method based on phoneme-pair contrastive energy learning according to claim 1, wherein, In training the phoneme contrast energy model, a hard example mining strategy is used, which is to select the negative sample with the smallest energy difference from the anchor phoneme feature to construct a triplet, and the expression is as follows: wherein is a dialectal phoneme characteristic; is a phoneme characteristic and energy difference between 4. The dialectal TTS method based on phoneme-pair contrastive energy learning of claim 1, wherein, the function definition of the phoneme contrast energy model is as follows: wherein is a feature of an arbitrary phoneme i is a feature of an arbitrary phoneme is a feature of an arbitrary phoneme j is a feature of an arbitrary phoneme is an energy difference between the features of the phonemes; is an Euclidean distance term for measuring the distance between the features of the phonemes and in the feature space, |·|2 is an L2 norm; and are regularization coefficients; is a regularizing term for the manner of articulation, is a regularizing term for the place of articulation, expressed as follows: wherein K 1 is the total number of manner features; is the weight parameter of the k th manner feature; is the weight parameter of the k th manner feature extraction function; |·| is the absolute value symbol; K 2 is the total number of place features; is the weight parameter of the th place feature; is the weight parameter of the th place feature extraction function.

5. The dialectal TTS method based on phoneme-pair contrastive energy learning according to claim 4, characterized in that, by performing word segmentation and phoneme decomposition on the standard language text, and performing noise reduction, segmentation and phoneme alignment on the dialect audio samples, the corresponding phoneme features are extracted using a pre-trained general speech model; wherein all the phoneme features of the standard language constitute the standard language phoneme sequence, denoted as , denotes the t th standard language phoneme feature, , T is the total number of phoneme features in the standard language phoneme sequence ; all the audio samples of the dialect constitute the dialect phoneme set, denoted as , denotes the m th dialect phoneme feature, , M is the total number of phoneme features in the dialect phoneme set . selecting the dialect phoneme with the smallest energy gap for each standard language phoneme as the mapping result is represented as: wherein represents a dialect phoneme sequence the first t dialect phoneme feature in is an energy difference between phoneme features and ​ 6. The dialectal TTS method based on phoneme-pair contrastive energy learning of claim 5, wherein, In generating the dialect phoneme sequence Further, a dynamic programming algorithm is introduced to optimize the sequence, and the optimized dialect phoneme sequence is converted into a target dialect speech waveform; wherein the expression of the optimization process is: wherein is a phoneme feature and is an energy difference between the phoneme features; is a transition cost between adjacent phoneme features.

7. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the dialect speech synthesis method based on phoneme contrast energy learning as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, when the program is executed by the processor, it implements the steps of the dialect speech synthesis method based on phoneme contrast energy learning as claimed in any one of claims 1 to 6.

Images