An unsupervised speech recognition modeling method based on phoneme segment level representation discretization

Abstract

The present application relates to an unsupervised speech recognition modeling method based on phoneme segment level representation discretization, belonging to the field of speech recognition. The present application uses an IFMF model to extract phoneme features from the original audio through a speech feature discretization module, then trains a K-Means clustering algorithm to cluster the audio representation, and obtains the clustering index as a discrete label; through an adversarial learning module, a generator network and a discriminator are trained; through a speech discrete representation decoder module, a language model is used to decode the phoneme discretization-based model output obtained by unsupervised training. The present application fuses multiple speech features, adopts a phoneme segment level representation discretization method, considers the influence of tone information on the Lao language, and as much as possible reduces the influence of information redundancy caused by fine-grained features on cross-modal modeling. The method of the present application achieves competitive results compared with traditional speech recognition methods.

Description

TECHNICAL FIELD

[0001] The present application relates to an unsupervised speech recognition modeling method based on phoneme segment-level representation discretization, belonging to the field of speech recognition. TECHNICAL BACKGROUND

[0002] Speech recognition technology, as an important branch of artificial intelligence, has made great progress in recent years. However, for languages with scarce resources, the development of speech recognition technology still faces many challenges. Traditional speech recognition methods often require a large amount of labeled data for model training, and for languages such as Lao, the acquisition and organization of labeled data is often a time-consuming and costly task. Therefore, exploring an unsupervised learning-based speech recognition modeling method is of great significance for promoting the development of speech recognition technology for languages with scarce resources such as Lao.

[0003] In recent years, speech recognition modeling methods based on phoneme segment-level representation have gradually attracted the attention of researchers. This method can better capture key information in speech by converting continuous speech signals into discrete representations, improving recognition accuracy. At the same time, the discrete representation method also helps to reduce the complexity of the model and improve computational efficiency. However, existing speech recognition modeling methods based on discrete representation mostly rely on supervised learning, i.e., a large amount of labeled data is needed for model training. This is undoubtedly a huge obstacle for languages with scarce resources such as Lao. Therefore, researching an unsupervised speech recognition modeling method based on phoneme segment-level representation discretization has important theoretical significance and practical value.

[0004] In the process of exploring unsupervised speech recognition modeling methods, researchers have proposed a variety of different ideas and methods. Some researchers try to use autoencoders and other technologies to convert continuous speech signals into discrete phoneme segment-level representations, and then use clustering methods for unsupervised learning. Other researchers use generative adversarial networks and other deep learning models to gradually learn the internal rules and structures of speech signals through the adversarial training of generators and discriminators. These methods provide new ideas for unsupervised speech recognition modeling.

[0005] However, the application of these methods in resource-poor languages such as Lao is still relatively limited. Therefore, this study aims to explore an unsupervised speech recognition modeling method based on phoneme segment-level representation discretization and apply it to the Lao speech recognition task. It is hoped that through this method, the problem of insufficient labeled data for Lao can be overcome, and the accuracy and robustness of Lao speech recognition can be improved. Specifically, the IFMF model is used to segment the speech at the phoneme level, and the K-Means model is used to obtain discrete features. Then, using the strategy of adversarial learning, a large amount of unlabeled audio data and text data are used to learn the mapping relationship between audio representation and phonemes by establishing the alignment relationship between audio and text. Finally, the self-training method is used to build an unsupervised learning-based Lao speech recognition model, and experimental verification and performance evaluation are carried out. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a phoneme segment-level representation discretization-based unsupervised speech recognition modeling method, which can overcome the problem of insufficient labeled data for Lao and improve the accuracy and robustness of Lao speech recognition.

[0007] The method of the present application uses discrete labels as input to replace the traditional raw waveform, acoustic features or self-supervised feature input. These labels are obtained by K-Means clustering calculation of the phoneme-level representation segmented by the IFMF model. Through the strategy of adversarial training, a large amount of unlabeled data is used to learn the mapping relationship between Lao audio and text. Compared with previous research on unsupervised speech recognition, the method proposed in the present application fuses multiple speech features and uses phoneme segment-level representation discretization, which not only considers the influence of tone information on Lao, but also reduces the information redundancy caused by fine-grained features to the influence of cross-modal modeling. Experiments on the Lao dataset show that the proposed method achieves competitive results compared with traditional speech recognition methods.

[0008] The technical solution of the present application is: an unsupervised speech recognition method based on phoneme segment-level representation discretization, comprising:

[0009] Step 1: The speech feature discretization module uses the IFMF model to extract phoneme features from the original audio, and then trains a K-Means clustering algorithm to cluster the audio representation and obtain the clustering index as a discrete label. Before extracting the phoneme features from the original audio, data selection and preprocessing are performed. Using the constructed dataset, the strategy of unsupervised pre-training with large-scale unlabeled data and validation testing with a small amount of labeled data is used to construct a Lao speech recognition dataset, which has a total amount of 1042.3 hours.

[0010] Step 2, training a generator network G and a discriminator network C through the adversarial learning module, wherein the generator network generates samples, and then the discriminator network judges, the discriminator network is trained to distinguish whether the sample is from the generator network or from the real data distribution, and the target of the generator network is to generate samples that cannot be distinguished by the discriminator network;

[0011] Step 3, using a language model to decode the phoneme discretization-based model output obtained through unsupervised training through a speech discrete representation decoder module.

[0012] As a further scheme of the present application, in step 1, an IFMF model is used to extract phoneme features from the original audio; the IFMF model fully utilizes the advantages of self-supervised features and spectral features through multi-feature interaction fusion to perform phoneme segmentation on the Lao audio; then a K-Means clustering algorithm is trained to cluster the audio representation, and a clustering index is obtained as a discrete label, which is used as the input of the speech recognition model.

[0013] As a further scheme of the present application, step 2 includes:

[0014] G takes a series of T segments S = [s1,...,s T ] as input, and then maps it to a series of M phoneme sequences G(S) = [p1,...,p M ]; the generator network predicts the distribution on the phoneme set O for each segment and outputs the phoneme with the highest probability; if the argmax prediction results of consecutive segments are the same phoneme, then one of the segments will be sampled, M≤T;

[0015] The phoneme set O contains a silence label SIL so as to mark the silence part in the speech audio as silence; during the back propagation process, the sampling point output by the generator network is back propagated, and during the unsupervised training, the segment representation S is not modified, and the generator network is parameterized as a single-layer convolutional neural network;

[0016] The input of the discriminator network is a one-hot vector sequence p r ~ P r representing phonetized text obtained from the real data distribution P r , or the output distribution sequence of the generator G(S); each input vector has |O| dimensions, representing the distribution of each phoneme segment; the discriminator network is a CNN, which outputs a probability representing the possibility that the sample comes from the data distribution;

[0017] In the experiment, the original GAN objective with gradient penalty, a segment smoothing penalty and a phoneme diversity penalty are used, as shown in the following formula:

[0018]

[0019] where p r ~ P r denotes phonetized unlabelled text, G(S) denotes the transcription output generated by the generator for the input speech segment representation S; for some unlabelled speech audio, the first term trains the discriminator to assign high probability to real transcriptions, and the second term encourages the discriminator to assign low probability to generator outputs; L gp is a gradient penalty term, l pd is a phoneme diversity loss; during training, the discriminator and generator are updated alternately, and the predicted transcriptions from the generator and phonetized unlabelled text are also used alternately as batches;

[0020] Gradient penalty: the experiments penalize the gradient norm of the input in the discriminator; the penalty is computed for arbitrary samples , which are linear combinations of the activations of real and fake samples; this is shown in the following equation:

[0021]

[0022] Phoneme diversity loss: maximize the entropy of the average softmax distribution H G (G(S)) over the phonemes vocabulary of the generator, which is computed over a batch B of utterances, as shown in the following equation:

[0023]

[0024] The goal of the experiments was to build a speech recognition model without any supervision, for which cross-validation metrics that do not require labelled data were used for early stopping, random seed selection, and hyperparameter selection (λ, η); in this metric, two quantities are considered: the LM negative log-likelihood and the vocabulary usage; this is shown in the following equation:

[0025]

[0026] NLL LM is a measure of fluency of a given transcription, which is measured by using a language model p LM trained on phonetized text data, where p LM (p t ) is a short-hand for p LM (p t | p t-1 ,..., p1); the vocabulary usage U(P) is the proportion of phoneme vocabulary that the model outputs through Viterbi decoding; the model configuration chosen this way produces phoneme sequences that score high under the language model but are not long.

[0027] As a further scheme of the present application, in the step 3, the phoneme-based model output obtained through unsupervised training is decoded using a language model; the model uses PyKaldi to construct WFSTs; the WFSTs take the model output as input and use the same phonetizer as the preprocessed unlabelled text data to construct a mapping from phonemes to text; the WFSTs are combined with a 4-grams language model, and only 4-grams with an occurrence frequency of more than 3 are retained.

[0028] As a further scheme of the present application, in the step 3, during decoding, an acoustic scale is provided as a parameter to the Kaldi decoder, and a scalar v is added to the emission of the blank label; the optimal weights of the two parameters are adjusted by minimizing a quantity that measures the fluency of the output and the realness of the model output; the following quantity is minimized on an unlabelled development set, assuming that the phoneme-based model is decoded into syllables, as shown in the following formula:

[0029]

[0030] wherein, is the output of the Viterbi model, is the conversion of the syllable-based output of the WFSTs into phonemes, H LM (p j ) is the entropy of the language model, ED is the edit distance, and μ = 0.03.

[0031] The present application has the following beneficial effects:

[0032] The present application provides an unsupervised speech recognition modeling method based on phoneme segment-level representation discretization. The method uses discrete labels as input to replace the traditional raw waveform, acoustic features or self-supervised feature input. These labels are obtained by K-Means clustering calculation of the phoneme granularity representation after IFMF model segmentation. Through the strategy of adversarial training, the mapping relationship between Lao audio and text is learned by using a large amount of unlabeled data. Compared with existing ASR models, the proposed unsupervised speech recognition method based on representation discretization outperforms other methods in terms of performance on the development set and the test set. Compared with the optimal result of the benchmark model, the proposed method reduces the word error rate by 1.61% on the development set and by 2.63% on the test set. Experiments on the Lao language data set show that the proposed method achieves competitive results compared with traditional speech recognition methods, and the present application can overcome the problem of insufficient labeled data for Lao language and improve the accuracy and robustness of Lao language speech recognition. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a general design block diagram of an unsupervised speech recognition modeling method based on phoneme segment-level representation discretization proposed by the present application;

[0034] Figure 2 is a recognition example of an unsupervised speech recognition method based on phoneme segment-level representation discretization proposed by the present application; DETAILED DESCRIPTION

[0035] The unsupervised speech recognition modeling method based on phoneme segment-level representation discretization proposed by the present application will be further described below in combination with the drawings and specific embodiments. It should be noted that the drawings are very simplified and are only used to facilitate and clarify the purpose of assisting in the description of the embodiments of the present application.

[0036] Embodiment 1: As shown in the following table, an unsupervised speech recognition modeling method based on phoneme segment-level representation discretization includes: Figure 1

[0037] Step 1: The IFMF model is used by the speech feature discretization module to extract phoneme features from the original audio, and then a K-Means clustering algorithm is trained to cluster the audio representation and obtain the clustering index as a discrete label; in the speech recognition model of the present application, instead of using original continuous speech or acoustic features, the discrete label is used as input. The IFMF model is used to extract phoneme features from the original audio; the IFMF model fully utilizes the advantages of self-supervised features and spectral features through multi-feature interaction fusion to perform phoneme segmentation on Lao audio; then a K-Means clustering algorithm is trained to cluster the audio representation and obtain the clustering index as a discrete label, which is used as input of the speech recognition model.

[0038] Before extracting phoneme features, the present application first selects and preprocesses data,

[0039] The speech data set used for training and verification in the experiment is obtained by first obtaining a written expression data set from several Internet websites, and then forming a small batch of small-scale carefully prepared labeled data set by applying speech synthesis technology and manual annotation. Specifically, in the speech synthesis link, the advanced speech synthesis API provided by the Microsoft platform is used to generate Lao language standard labeled data with high-quality audio characteristics; in the aspect of manual annotation, the Lao professional linguists cooperating with the laboratory closely cooperate with the laboratory, and these experts use the professional platform provided by the laboratory to accurately annotate the selected unlabeled data and strictly check the synthesized audio data to ensure the high accuracy and reliability of the data set.

[0040] The unsupervised pre-training of large-scale unlabeled data and the verification test strategy of small amount of labeled data are successfully combined to build a Lao language speech recognition data set, and the total amount of the data set reaches 1042.3 hours, and the detailed information is shown in Table 1.

[0041] ​Table 1: Experimental data sets

[0042]

[0043] The present invention compares the data size of different data formats, including raw waveforms, traditional acoustic features, features based on self-supervised learning, multi-feature interactive fusion speech features, and discrete labels. Taking T seconds of single-channel speech pronunciation as an example. The size of the raw waveform data depends on the sampling rate and audio sample encoding. Here, the commonly used settings in speech recognition are used, i.e. 16 kHz wav format audio. For acoustic features, a D-dimensional floating point vector is used, with a frame shift of 10 ms, which is for a rate of 100 frames per second, where an example value of D is the conventional 80. For self-supervised learning features, the single-layer hidden embedding in the XLS-R model is used as an example of self-supervised learning features. For multi-feature interactive fusion, the last layer of hidden embedding of the IFMF model is used, and for discrete labels, the K-Means model is used to cluster the audio features segmented by the IFMF model.

[0044] Step 2, train a generator network G and a discriminator network C through an adversarial learning module, wherein the generator network generates samples, which are then judged by the discriminator network, and the discriminator network is trained to distinguish whether the samples are from the generator network or from the real data distribution, and the goal of the generator network is to produce samples that the discriminator network cannot distinguish; said step 2 comprises:

[0045] G takes a series of T segments S = [s1,...,s T ] as input, and then maps it to a series of M phoneme sequences G(S) = [p1,...,p M ]; the generator network predicts the distribution on the set of phonemes O for each segment and outputs the phoneme with the highest probability; if the argmax prediction results of consecutive segments are the same phoneme, then one of these segments will be sampled, M ≤ T;

[0046] The set of phonemes O contains a silence label SIL, so as to mark the silence part in the speech audio as silence; if there is no silence label, the model will reuse a specific phoneme to mark silence, which greatly reduces the performance, because it will interfere with the subsequent language model decoding. During backpropagation, the sampling point output by the generator network is backpropagated, and during unsupervised training, the segment representation S is not modified, and the generator network is parameterized as a single-layer convolutional neural network;

[0047] The input of the discriminator network is a sequence of one-hot vectors p r ~ P r representing phonetized text obtained from the real data distribution P r, or the sequence of output distributions of the generator G(S); each input vector has |O| dimensions, representing the distribution of each phoneme segment; the discriminator network is a CNN that outputs a probability indicating the likelihood that the sample comes from the data distribution;

[0048] In the experiments, the original GAN objective with gradient penalty, a segment smoothness penalty, and a phoneme diversity penalty were used, as follows:

[0049]

[0050] where p r ~ P r represents the phonetized unlabelled text, and G(S) represents the transcription output generated by the generator for the input speech segment representation S; for some unlabelled speech audio, the first term trains the discriminator to assign high probabilities to real transcriptions, and the second term encourages the discriminator to assign low probabilities to the generator output; L gp is a gradient penalty term, and L pd is a phoneme diversity loss; during training, the discriminator and the generator are updated alternately, and the predicted transcriptions from the generator and the phonetized unlabelled text are alternately used as batches;

[0051] Gradient penalty: the experiments penalize the gradient norm of the input to the discriminator; the penalty is computed for arbitrary samples , which are linear combinations of activations of real samples and fake samples; the penalty is computed as follows:

[0052]

[0053] Phoneme diversity loss: maximize the entropy of the average softmax distribution H G (G(S)) over the phoneme vocabulary, which is computed over the pronunciations in a batch B, as follows:

[0054]

[0055] The experimental objective was to build a speech recognition model without any supervision, for which cross-validation metrics that do not require labelled data were used for early stopping, random seed selection, and hyperparameter selection (λ, η); in this metric, two quantities were considered: the LM negative log-likelihood (NLL) and the vocabulary usage; the metric is computed as follows:

[0056]

[0057] NLL LM is a measure of the fluency of a given transcription, which is measured by using a language model p LM trained on phonetized text data, where pLM (p t ) is a short form of p LM (p t |p t-1 ,...,p1); the vocabulary usage U(P) is the proportion of phoneme vocabulary output by the model through Viterbi decoding; the model configuration thus selected produces phoneme sequences that score high under the language model but are not long.

[0058] Step 3, the phoneme-discretized model output obtained through unsupervised training is decoded using a language model through a speech discrete representation decoder module.

[0059] In step 3, the phoneme-discretized model output obtained through unsupervised training is decoded using a language model; the model uses PyKaldi to construct WFSTs; the WFSTs use the same phonetizer as the pre-processed unlabelled text data to construct a phoneme-to-text mapping using the model output as input; the WFSTs are combined with a 4-grams language model, and only 4-grams that occur more than 3 times are kept.

[0060] During decoding, the acoustic scale is provided as a parameter to the Kaldi decoder, and a scalar v is added to the emission of the blank token; the optimal weight of these two parameters is adjusted by minimizing a quantity that measures the fluency of the output and the realism of the model output; the following quantity is minimized on an unlabelled development set assuming that the phoneme-based model is decoded into syllables as shown in the following formula:

[0061]

[0062] where, is the output of the Viterbi model, is the WFST-based syllable-to-phoneme conversion, H LM (p j ) is the entropy of the language model, ED is the edit distance, and μ = 0.03.

[0063] The present application designs a comparative experiment with other schemes to prove the effectiveness of the proposed method. The evaluation results of the character error rate (CER) of the proposed method and other common speech recognition models (Wenet, Speechbrain, DeepSpeech, SpecAugment, NST, Vq-Wav2vec, Wav2vec2.0, Wav2vec-U, Wav2vec-U+ST) are given in Table 2. The present application has achieved the lowest word error rate on the development set and the test set, proving the effectiveness of the present application in the ASR error correction task.

[0064] Table 2: Comparison experiments of Lao language speech recognition based on existing data

[0065]

[0066] From the analysis of the results in Table 2, it can be seen that the proposed unsupervised speech recognition method based on representation discretization outperforms other methods on both the development set and the test set. The four methods, Wenet, Speechbrain, DeepSpeech2 and SpecAugment, have relatively poor performance due to the lack of training data, which cannot fully exploit the advantages of deep learning. NST, VqWav2vec, Wav2vec2.0 and Wav2vec-U all experimental configurations are consistent with the proposed method, compared with the optimal results of the benchmark model, the proposed method reduces the word error rate by 1.61% on the development set and 2.63% on the test set. These results show the excellent performance of the method in this paper in the task of Lao speech recognition.

[0067] In order to verify the influence of different K values on Lao speech recognition, experiments were conducted on the basis of the benchmark model with cluster numbers of 25, 35, 50, 75 and 100.

[0068] Table 3: Influence of different K values on speech recognition performance

[0069]

[0070]

[0071] It can be seen from the analysis of Table 3 that the greater the K value, the better the performance of speech recognition is not necessarily. First, a larger K value means more cluster centers, which will make the algorithm more detailed in dividing audio features. There are 104 phonemes in IPA, and some phonemes are too similar in pronunciation, such as plosive, pause and laryngeal. These phonemes are relatively weak, usually accompanied by other sounds, and it is difficult to distinguish them only by sound signals without considering surrounding phonemes. When K = 75 or K = 100, almost every phoneme forms a class, in this case, the algorithm will pay too much attention to the noise and outliers in the data, making the model too sensitive to the details of the phoneme features, resulting in overfitting. When K = 25 or K = 35, the algorithm cannot fully capture the structural differences in the audio data, and will incorrectly classify different audio features into the same cluster, the clustering is too rough, and cannot fully capture the details of the audio data, resulting in underfitting, which reduces the performance of speech recognition. Since the number of commonly used phonemes is about 60, when K = 50, this division makes the audio features in each cluster more similar, and the feature difference between different clusters more obvious. The selection of this K value can reduce noise data or outliers into fewer clusters, which can capture enough details of the audio data, and will not cause overfitting. According to the experimental results, the K value used in all experiments of this study is 50.

[0072] In order to verify the effectiveness of the method, Table 4 gives the comparison of the data preprocessing method after the discretization processing of the proposed method and four different features (FBank, FBankD, XLS-R, XLS-RD).

[0073] Table 4: Influence of different feature discretization on speech recognition

[0074]

[0075] Analysis of Table 4 shows that the method proposed in this study performs best compared to these methods. Comparing FBank and FBankD, XLS-R and XLS-RD reveals that the discretized features perform better. This is because the quality of audio data is often affected by various noises and interferences. Discretized speech features transform continuous speech signals into discrete ones, reducing the impact of noise and interference on speech feature representation. This ensures that even with small fluctuations or changes in the speech signal, the discretized features remain stable and consistent, allowing the model to better capture the semantic information inherent in the speech signal itself. Comparing FBankD, XLS-RD, and audio representations extracted using the IFMF model shows that neither traditional acoustic features nor self-supervised learning representations can fully extract information from Lao, a tonal language. Traditional acoustic features tend to focus on vowel recognition because vowels are longer and more continuous than consonants. Self-supervised learning features can compensate for the performance of traditional acoustic features in consonant recognition. The IFMF model, by incorporating pitch features, enhances the tonal characteristics of Lao audio. Therefore, the method proposed in this invention addresses the complexity and pronunciation diversity of Lao tones by fusing pitch features and then discretizing them into phoneme-level representations, which is more conducive to the model's recognition performance. This method can more fully consider the tonal characteristics of Lao and achieve better overall performance.

[0076] This invention is in Figure 2 The text demonstrates how using speech feature discretization can improve model recognition performance; a Lao speech recognition model that did not perform audio feature discretization made incorrect recognitions. and The main reason is and and They are high consonants and low consonants respectively. and These are single consonants and compound consonants, whose pronunciations are similar, leading to speech recognition's inability to distinguish them. The method proposed in this invention, in addition to incorporating the fundamental frequency features of the audio signal during feature extraction, also discretizes the audio features. By comparing their speech features, they are discretized into different clusters. During recognition, the model prioritizes candidate texts within the same cluster as the input phonemes, thereby improving the accuracy and robustness of the speech recognition model when handling similar phonemes. Experimental results show that, during the training phase, the Lao speech recognition model after discretizing the audio features is better able to distinguish texts with similar pronunciations, thus achieving better recognition results.

[0077] The specific embodiments of the present application are described above in detail with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. An unsupervised speech recognition modeling method based on phoneme segment level representation discretization, characterized in that: Comprising: Step 1: phoneme features are extracted from the original audio using an IFMF model by a speech feature discretization module, and then a K-Means clustering algorithm is trained to cluster the audio representation, and the clustering index is obtained as a discrete label; Step 2: a generator network G and a discriminator network C are trained by an adversarial learning module, wherein the generator network generates samples, which are then judged by the discriminator network, and the discriminator network is trained to distinguish whether the samples are from the generator network or from the real data distribution, and the goal of the generator network is to generate samples that the discriminator network cannot distinguish; Step 3: the phoneme discretization-based model output obtained by unsupervised training is decoded using a language model by a speech discrete representation decoder module; The step 2 comprises: Representing a sequence of T segments as input, and then map it to a sequence of M phonemes The generator network predicts a distribution over the set of phonemes O for each segment, and outputs the most probable phoneme; if the argmax predictions for consecutive segments are the same phoneme, then sample one of these segments, ​ The phoneme set O contains a silence label SIL, so as to mark the silence part in the speech audio as silence; during back propagation, the sampling points output by the generator network are back propagated, and during unsupervised training, the segment representation S is not modified, and the generator network is parameterized as a single-layer convolutional neural network; The input to the discriminator network is a sequence of one-hot vectors representing phonetized text taken from the real data distribution , or a sequence of output distributions from the generator G(S); each input vector has dimensions, representing the distribution of each phoneme segment; the discriminator network is a CNN that outputs a probability representing the likelihood that the sample came from the data distribution;​ In the experiment, the original GAN objective with gradient penalty, a segment smoothing penalty and a phoneme diversity penalty are used; Gradient penalty: the experiment penalizes the norm of the gradient of the input to the discriminator; this penalty is applied to arbitrary samples computed, these samples are linear combinations of the activation values of the real and fake samples; Phoneme diversity loss: maximizing the entropy of the average softmax distribution of the generator over the phoneme vocabulary, this distribution is computed over the pronunciations of a batch B ; The goal of the experiment was to build a speech recognition model without any supervision, for which a cross-validation metric that does not require labeled data was used for early stopping, random seed selection, and hyperparameter selection In this metric, two quantities are considered: the LM negative log-likelihood and the amount of vocabulary used. 2.The unsupervised speech recognition modeling method based on phoneme segment level representation discretization according to claim 1, characterized in that: In the step 1, the IFMF model is used to extract phoneme features from the original audio; The IFMF model fully utilizes the advantages of self-supervised features and spectral features through multi-feature interaction fusion to perform phoneme segmentation on the Lao speech audio; then a K-Means clustering algorithm is trained to cluster the audio representation, and the clustering index is obtained as a discrete label, which is used as the input of the speech recognition model. 3.The unsupervised speech recognition modeling method based on phoneme segment level representation discretization according to claim 1, characterized in that: In the step 3, the phoneme discretization-based model output obtained by unsupervised training is decoded using a language model; The model uses PyKaldi to construct WFST; WFST uses the same phonemizer as the preprocessed unlabeled text data to construct a phoneme-to-text mapping using the model output as input; WFST is combined with a 4-grams language model, and only 4-grams with a frequency of more than 3 times are retained.

Images