A method for pronunciation enhancement of Mongolian unregistered words based on separable contrastive learning

Abstract

The present invention discloses a method for enhancing the pronunciation of unregistered Mongolian words based on separation contrastive learning, and relates to the technical field of natural language processing. The method comprises: S1. acquiring data; S2. dividing data; S3. establishing a model; S4. training a model; and S5. recognizing and evaluating. Based on a speech spectrogram, the present invention enhances the pronunciation from three aspects: speech content, speech rhythm, and speech emotion, so that the noise robustness of the contrastive pronunciation enhancement model is effectively improved, thereby ensuring the qualified rate of the augmented speech; the coverage of the pronunciation of unregistered words is achieved, and through fine-grained rhythm control, it is beneficial to improve the rhythmic performance and emotional expression of the generated pronunciation, and improve the quality of the augmented pronunciation data.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a Mongolian unregistered word pronunciation enhancement method based on separation contrast learning. Background Art

[0002] The augmented model of out-of-domain (OOV) pronunciation data based on text-to-speech (TTS) can achieve coverage of the pronunciation of unregistered words. For the problem of low naturalness of generated pronunciation, fine-grained rhythm control can improve the rhythmic performance and emotional expression of generated pronunciation. However, in the process of controlling the rhythmic features, external speech needs to be introduced, and the augmented model will also fit the noise data in the out-of-domain speech, such as reverberation and environmental noise. In addition, the phenomena such as vowels, vowel vibrato, and vowel splicing in Mongolian pronunciation aggravate the problem of fitting the noise in the out-of-domain speech by the pronunciation data augmented model, resulting in a decrease in the naturalness of the augmented pronunciation, poor comprehensibility, and even ambiguity.

[0003] In the prior art, speech enhancement methods have gone through a process from signal processing to in-depth optimization of specific scenarios. Li et al. proposed a speech enhancement framework that combines dynamic attention mechanism and recursive learning. By introducing dynamic attention mechanism, the dynamic characteristics of speech are effectively captured and the performance of speech enhancement is improved. Subsequently, Leng et al. further explored the trade-off between attenuating interfering sounds and maintaining sound quality in speech enhancement. The proposed method not only performed well in maintaining sound quality, but also provided rich data for subsequent research. Huang et al. achieved a breakthrough in separating speech from contrasting mixtures through self-supervised learning methods, further reducing model parameters and dependence on clean speech. The wav2vec method introduced by Wang et al. encodes noise robustness into speech context representation. This method not only improves the performance of speech enhancement, but also provides tools for real-time speech processing. On this basis, Chen et al. further emphasized the effectiveness of scale-invariant signal-to-noise ratio as training data. In terms of enhancement processing of specific scenarios, the CDPAM metric method proposed by Manocha et al. provides an effective tool for performance evaluation of speech enhancement tasks. This method enables a more objective evaluation of the advantages and disadvantages of different methods, providing a clearer direction for subsequent research. At the same time, Vanderreydt et al. improved noise robustness through speech reconstruction technology, providing strong support for noisy speech recognition. Develasco et al. transformed the speech task paradigm from regression to classification, providing new ideas and methods for speech processing.

[0004] The Mongolian OOV pronunciation data augmentation based on TTS needs to rely on out-of-domain speech to improve the rhythmic representation and emotional expression of the augmented OOV pronunciation. During the augmentation process, the model will simultaneously fit the noise in the out-of-domain speech, resulting in a decrease in the pass rate and naturalness of the augmented pronunciation. How to eliminate the noise in the augmented OOV pronunciation becomes a problem. In the denoising problem of augmented OOV pronunciation, the difficulty lies in how to construct a pronunciation enhancement network and retain the rhythmic and emotional features in the augmented data while eliminating the noise of the augmented pronunciation; due to the diversity of noise types and degrees, how to make the pronunciation enhancement network robust and generalizable, so that the generated OOV pronunciation data is clear and natural becomes a challenge.

[0005] Therefore, it is an urgent problem for those skilled in the art to propose a method for enhancing the pronunciation of unregistered Mongolian words based on separation contrastive learning to solve the difficulties existing in the prior art. Summary of the invention

[0006] In view of this, the present invention provides a Mongolian unregistered word pronunciation enhancement method based on separation contrastive learning, which is beneficial to improving the rhythmic performance and emotional expression of generated pronunciation and improving the quality of augmented pronunciation data.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A method for enhancing the pronunciation of Mongolian unregistered words based on separation contrastive learning comprises the following steps:

[0009] S1. Get data: Get Mongolian text sequence and generate Mel spectrum M corresponding to pronunciation data 2 ;

[0010] S2. Data division: The Mel spectrum graph M corresponding to the original speech data 1 Mel spectrum M corresponding to the pronunciation data generated according to the Mongolian text sequence 2 Divide into training set and test set;

[0011] S3. Model building: Building a separation and contrast pronunciation enhancement model based on the feature separation module and the contrast enhancement module;

[0012] S4. Model training: input the training set into the separation contrast pronunciation enhancement model for training, update the model weight parameters through the loss function, and obtain the trained separation contrast pronunciation enhancement model after several trainings;

[0013] S5. Recognition evaluation: Input the test set into the trained separation contrast pronunciation enhancement model to evaluate the Mongolian pronunciation recognition results and achieve Mongolian pronunciation enhancement.

[0014] In the above method, optionally, the Mel spectrum graph M corresponding to the original speech data in S2 1 Mel spectrum M corresponding to the pronunciation data generated according to the Mongolian text sequence 2 The data is divided into training and testing sets in pairs.

[0015] In the above method, optionally, the feature separation module in S3 includes a convolution layer, a normalization layer, a linear layer, a fully connected layer, and a Dropout layer; wherein,

[0016] The convolution layer first extracts local features from the input training set data, then accelerates training and stabilizes feature distribution through the normalization layer. The linear layer performs linear transformation on the normalized features, and the ReLU activation function introduces nonlinear factors. The fully connected layer further integrates features and prevents overfitting through the Dropout layer. Finally, the original speech feature F is output. 1 and generate pronunciation features F 2 .

[0017] In the above method, optionally, the loss function of the feature separation module is as follows:

[0018] Loss = α·Lossrhythm(Frhythm dropout ,Frhythm target )+β·Lossemotion(Femotion dropout ,Femotion target )

[0019] Among them, Frhythm represents the rhythm feature, Femotion represents the emotional feature, Lossrhythm represents the loss function of the rhythm feature, Lossemotion represents the loss function of the emotional feature, Frhythm target Represents the target value of the prosodic feature, Femotion target Represents the target value of the emotional feature, Frhythm dropout Represents the target value of the rhythmic feature after the dropout layer, Femotion dropout represents the target value of the emotional feature after the dropout layer, α represents the weight of the rhythmic feature loss, and β represents the weight of the emotional feature loss.

[0020] In the above method, optionally, the contrast enhancement module in S3 receives the original speech feature F obtained by the feature separation module 1 and generate pronunciation features F 2 As input, the convolutional neural network and ReLU activation function are used to extract and transform the original speech features F 1 and generate pronunciation features F 2, the original speech feature F is embedded in the 1 and generate pronunciation features F 2 Mapped to a low-dimensional embedding space to obtain the learned enhanced feature F'.

[0021] In the above method, optionally, the contrast enhancement module uses the InfoNCE loss function to calculate the original speech feature F 1 and generate pronunciation features F 2 The similarity between them is measured and the contrast loss is minimized by the optimizer.

[0022] In the above method, optionally, the InfoNCE loss function of the contrast enhancement module is as follows:

[0023]

[0024] Among them, f i The feature vector representing the original speech data, Represents the enhanced features of the original speech data, f j represents the feature vector of the generated pronunciation data, sim(·,·) represents the similarity function, τ represents the concentration of the control distribution, and N represents the number of samples.

[0025] In the above method, optionally, in S5, the test set is input into the trained separation contrast pronunciation enhancement model to evaluate the Mongolian pronunciation recognition result. The specific contents of implementing Mongolian pronunciation enhancement are as follows:

[0026] The test set is input into the feature separation module, and the feature separation module performs original speech feature F 1 and generate pronunciation features F 2 Extraction of the original speech feature F 1 and generate pronunciation features F 2 To the contrast enhancement module, the contrast enhancement module further enhances and optimizes the original speech features F 1 and generate pronunciation features F 2 , output the further enhanced and optimized feature F', F' is converted into an enhanced reading audio spectrogram M through the Mel encoder, and the Waveglow vocoder technology is used to convert the enhanced reading audio spectrogram M into enhanced reading data Speech_Out.

[0027] It can be seen from the above technical solution that, compared with the prior art, the present invention provides a method for enhancing the pronunciation of Mongolian unregistered words based on separation contrast learning, which has the following beneficial effects:

[0028] (1) Based on the speech spectrogram, the present invention enhances the pronunciation from three aspects: speech content, speech rhythm and speech emotion, so that the noise robustness of the comparative pronunciation enhancement model is effectively improved, thereby ensuring the pass rate of the augmented speech;

[0029] (2) The present invention achieves coverage of the pronunciation of unregistered words, and through fine-grained rhythm control, it is beneficial to improve the rhythmic performance and emotional expression of the generated pronunciation, and improve the quality of the augmented pronunciation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A flow chart of a method for enhancing the pronunciation of unregistered Mongolian words based on separation contrast learning provided by the present invention;

[0032] Figure 2 A schematic diagram of the architecture of the separation contrast pronunciation enhancement model provided by the present invention;

[0033] Figure 3 A schematic diagram of the structure of a feature separation module provided by the present invention;

[0034] Figure 4 This is a schematic diagram of the contrast enhancement module structure provided by the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Reference Figure 1 As shown, the present invention discloses a method for enhancing the pronunciation of Mongolian unregistered words based on separation contrast learning, comprising the following steps:

[0037] S1. Get data: Get Mongolian text sequence and generate Mel spectrum M corresponding to pronunciation data 2 ;

[0038] S2. Data division: The Mel spectrum graph M corresponding to the original speech data 1Mel spectrum M corresponding to the pronunciation data generated according to the Mongolian text sequence 2 Divide into training set and test set;

[0039] S3. Model building: Building a separation and contrast pronunciation enhancement model based on the feature separation module and the contrast enhancement module;

[0040] S4. Model training: input the training set into the separation contrast pronunciation enhancement model for training, update the model weight parameters through the loss function, and obtain the trained separation contrast pronunciation enhancement model after several trainings;

[0041] S5. Recognition evaluation: Input the test set into the trained separation contrast pronunciation enhancement model to evaluate the Mongolian pronunciation recognition results and achieve Mongolian pronunciation enhancement.

[0042] Furthermore, the Mel spectrum M corresponding to the original speech data in S2 1 Mel spectrum M corresponding to the pronunciation data generated according to the Mongolian text sequence 2 The data is divided into training and testing sets in pairs.

[0043] Furthermore, the feature separation module in S3 includes a convolutional layer, a normalization layer, a linear layer, a fully connected layer, and a Dropout layer;

[0044] The convolution layer first extracts local features from the input training set data, then accelerates training and stabilizes feature distribution through the normalization layer. The linear layer performs linear transformation on the normalized features, and the ReLU activation function introduces nonlinear factors. The fully connected layer further integrates features and prevents overfitting through the Dropout layer. Finally, the original speech feature F is output. 1 and generate pronunciation features F 2 .

[0045] Specifically, the feature separation module first uses the convolution layer to extract local features of the input training set data, then uses the normalization layer to accelerate the training process and stabilize the feature distribution, and uses the pooling layer to reduce the dimension of the feature map while retaining important information. Next, the linear layer performs a linear transformation on the normalized features, and the ReLU activation function introduces nonlinear factors, allowing the model to learn more complex feature representations. The fully connected layer further integrates key features and prevents overfitting through the Dropout layer. Finally, the rhythm features and emotional features are output separately. Its structure is as follows: Figure 3 shown.

[0046] The input training set data is Where N represents the number of samples and D represents the data dimension. Perform a convolution operation on the input data X, where K represents the convolution kernel. The specific convolution operation is shown in the following formula:

[0047]

[0048] Among them, Y c Represents the output of the convolution.

[0049] Then the normalization layer is used to normalize the output Y of the convolutional layer. c Perform batch normalization as shown below:

[0050] Y n =BatchNorm(Y c )

[0051] Among them, Y n Represents the normalized output.

[0052] The linear layer performs a linear transformation on the normalized features and applies the ReLU activation function, as shown in the following formula:

[0053] Z = ReLU(W l Y n +b l )

[0054] Among them, W l represents the weight matrix of the linear layer, b l represents the bias term, and Z represents the output after the linear layer and the ReLU activation function.

[0055] Then the fully connected layer integrates the output of the ReLU layer and outputs the final rhythmic features and emotional features, as shown in the following formula:

[0056] Frhythm,Femotion=W f Z+b f

[0057] Among them, W f represents the weight matrix of the fully connected layer, b f represents the bias term, Frhythm and Femotion represent the rhythmic feature and emotional feature respectively.

[0058] Furthermore, the loss function of the feature separation module is as follows:

[0059] Loss = α·Lossrhythm(Frhythm darget ),Frhythm target )+β·Lossemotion(Femotion dropout ,Femotion target )

[0060] Among them, Frhythm represents the rhythm feature, Femotion represents the emotional feature, Lossrhythm represents the loss function of the rhythm feature, Lossemotion represents the loss function of the emotional feature, Frhythm target Represents the target value of the prosodic feature, Femotion target Represents the target value of the emotional feature, Frhythm dropout Represents the target value of the rhythmic feature after the dropout layer, Femotion dropout represents the target value of the emotional feature after the dropout layer, α represents the weight of the rhythmic feature loss, and β represents the weight of the emotional feature loss.

[0061] Further, such as Figure 4 As shown, the contrast enhancement module in S3 receives the original speech feature F obtained by the feature separation module 1 and generate pronunciation features F 2 As input, the convolutional neural network and ReLU activation function are used to extract and transform the original speech features F 1 and generate pronunciation features F 2 , the original speech feature F is embedded in the 1 and generate pronunciation features F 2 Mapped to a low-dimensional embedding space to obtain the learned enhanced feature F'.

[0062] Furthermore, the contrast enhancement module uses the InfoNCE loss function to calculate the original speech feature F 1 and generate pronunciation features F 2 The similarity between them is measured and the contrast loss is minimized by the optimizer.

[0063] Furthermore, the InfoNCE loss function of the contrast enhancement module is as follows:

[0064]

[0065] Among them, f i The feature vector representing the original speech data, Represents the enhanced features of the original speech data, f j represents the feature vector of the generated pronunciation data, sim(·,·) represents the similarity function, τ represents the concentration of the control distribution, and N represents the number of samples.

[0066] Specifically, for the input generated speech data X=x 1 ,x 2 ,…,x N , where N represents the number of samples. For each sample x i , the feature separation module extracts its feature vector as shown in the following formula:

[0067] f i =FeatureSeparationModule(x i )

[0068] The contrast enhancement module enhances the input sample x by data enhancement. i Enhance and generate enhanced features And use the feature separation module to extract its feature vector, as shown in the following formula:

[0069]

[0070] Then construct positive and negative sample pairs, where the positive sample is composed of the feature vector f of the original sample i and the enhanced feature vector The negative samples are composed of f i And the generated pronunciation feature vector f j Composition, where j≠i.

[0071] Furthermore, in S5, the test set is input into the trained separation contrast pronunciation enhancement model to evaluate the Mongolian pronunciation recognition result. The specific contents of Mongolian pronunciation enhancement are as follows:

[0072] The test set is input into the feature separation module, and the feature separation module performs original speech feature F 1 and generate pronunciation features F 2 Extraction of the original speech feature F 1 and generate pronunciation features F 2 To the contrast enhancement module, the contrast enhancement module further enhances and optimizes the original speech features F 1 and generate pronunciation features F 2 , output the further enhanced and optimized feature F', F' is converted into an enhanced reading audio spectrogram M through the Mel encoder, and the Waveglow vocoder technology is used to convert the enhanced reading audio spectrogram M into enhanced reading data Speech_Out.

[0073] Specifically, the separation contrast pronunciation enhancement model is mainly composed of a feature separation module and a contrast enhancement module. The feature separation module extracts key rhythmic and emotional features from the input pronunciation signal, while suppressing non-target features. That is, the Mel spectrum M corresponding to the input original speech data 1 Mel spectrum M corresponding to the pronunciation data generated according to the Mongolian text sequence 2 After passing through the feature separation module, the output is the original speech feature F 1 and generate pronunciation features F 2. This process can be expressed as F=FeatureSeparation(M), where FeatureSeparation(·) represents the operation of the feature separation module. The function of the contrast enhancement module is to further enhance and optimize the rhythmic and emotional features extracted from the feature separation module. The contrast enhancement module highlights the target features by comparing the similarities and differences between the original speech data containing noise and the generated pronunciation data, thereby further improving the accuracy and clarity of the pronunciation and obtaining a further enhanced and optimized feature representation F'. This process can be expressed as F'=ContrastiveEnhancement(F), where ContrastiveEnhancement(·) represents the operation of the contrast enhancement module. F' is further converted into an enhanced reading audio spectrogram M through a Mel encoder. Finally, the enhanced reading audio spectrogram M is converted into enhanced pronunciation data Speech_Out using vocoder technologies such as Waveglow. Its overall architecture is as follows Figure 2 shown.

[0074] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0075] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for enhancing the pronunciation of unregistered Mongolian words based on separation contrastive learning, characterized in that: The following steps are involved: S1. Obtain data: obtain a Mongolian text sequence and generate a Mel spectrum M2 corresponding to the pronunciation data; S2. Data division: The Mel spectrogram M1 corresponding to the original speech data and the Mel spectrogram M2 corresponding to the pronunciation data generated according to the Mongolian text sequence are divided into a training set and a test set; S3. Model building: Building a separation and contrast pronunciation enhancement model based on the feature separation module and the contrast enhancement module; S4. Model training: input the training set into the separation contrast pronunciation enhancement model for training, update the model weight parameters through the loss function, and obtain the trained separation contrast pronunciation enhancement model after several trainings; S5. Recognition evaluation: Input the test set into the trained separation contrast pronunciation enhancement model to evaluate the Mongolian pronunciation recognition results and achieve Mongolian pronunciation enhancement; The feature separation module in S3 includes convolution layer, normalization layer, linear layer, fully connected layer, and Dropout layer; among them, The convolution layer first extracts local features from the input training set data, then accelerates training and stabilizes feature distribution through the normalization layer. The linear layer performs linear transformation on the normalized features, and the ReLU activation function introduces nonlinear factors. The fully connected layer further integrates features and prevents overfitting through the Dropout layer. Finally, the original speech feature F1 and the generated pronunciation feature F2 are output; The loss function of the feature separation module is shown as follows: Loss=ɑ·Lossrhythm(Frhythm dropout ,Frhythm target )+β·Lossemotion(Femotion dropout ,Femotion target ) Among them, Frhythm represents the rhythm feature, Femotion represents the emotional feature, Lossrhythm represents the loss function of the rhythm feature, Lossemotion represents the loss function of the emotional feature, Frhythm target Represents the target value of the prosodic feature, Femotion target Represents the target value of the emotional feature, Frhythm dropout Represents the target value of the rhythmic feature after the dropout layer, Femtion dropout represents the target value of the emotional feature after the dropout layer, α represents the weight of the rhythmic feature loss, and β represents the weight of the emotional feature loss; In S5, the test set is input into the trained separation contrast pronunciation enhancement model to evaluate the Mongolian pronunciation recognition results. The specific contents of Mongolian pronunciation enhancement are as follows: The test set is input into the feature separation module, which extracts the original speech features F1 and the generated pronunciation features F2, and outputs the original speech features F1 and the generated pronunciation features F2 to the contrast enhancement module. The contrast enhancement module further enhances and optimizes the original speech features F1 and the generated pronunciation features F2, and outputs the further enhanced and optimized features F'. F' is converted into an enhanced reading audio spectrogram M through a Mel encoder, and the enhanced reading audio spectrogram M is converted into enhanced pronunciation data Speech_Out using the Waveglow vocoder technology.

2. The method for enhancing the pronunciation of Mongolian unregistered words based on separation contrastive learning according to claim 1, characterized in that: The Mel spectrogram M1 corresponding to the original speech data in S2 and the Mel spectrogram M2 corresponding to the pronunciation data generated according to the Mongolian text sequence are divided into a training set and a test set in pairs.

3. The method for enhancing the pronunciation of Mongolian unregistered words based on separation contrastive learning according to claim 1, characterized in that: The contrast enhancement module in S3 receives the original speech feature F1 and the generated pronunciation feature F2 obtained by the feature separation module as input, uses the convolutional neural network and the ReLU activation function to extract and transform the original speech feature F1 and the generated pronunciation feature F2, and maps the original speech feature F1 and the generated pronunciation feature F2 to a low-dimensional embedding space through the embedding layer, thereby obtaining the feature F' after learning enhancement.

4. The method for enhancing the pronunciation of Mongolian unregistered words based on separation contrast learning according to claim 3, characterized in that: The contrast enhancement module uses the InfoNCE loss function to calculate the similarity between the original speech feature F1 and the generated pronunciation feature F2, and minimizes the contrast loss through the optimizer.

5. The method for enhancing the pronunciation of Mongolian unregistered words based on separation contrastive learning according to claim 4, characterized in that: The InfoNCE loss function of the contrast enhancement module is as follows: Among them, f i The feature vector representing the original speech data, Represents the enhanced features of the original speech data, f j represents the feature vector of the generated pronunciation data, sim(·,·) represents the similarity function, τ represents the concentration of the control distribution, and N represents the number of samples.

Images