Unsupervised speech recognition modeling method based on phoneme segment-level characterization discretization

Abstract

The invention relates to an unsupervised speech recognition modeling method based on phoneme segment-level characterization discretization, and belongs to the field of speech recognition. According to the method, phoneme features of original audio are extracted by using an IFMF model through a voice feature discretization module, then a K-Means clustering algorithm is trained to cluster audio representation, and a clustering index is acquired as a discrete mark; training a generator network and a discriminator through an adversarial learning module; and decoding the phoneme discretization-based model output obtained by unsupervised training by using a language model through a voice discrete representation decoder module. According to the method, multiple speech features are fused, and phoneme segment-level characterization discretization is adopted, so that the influence of tone information on Lao is considered, and the influence of information redundancy caused by fine-grained features on cross-modal modeling is reduced as much as possible. Compared with a traditional speech recognition method, the method provided by the invention obtains a competitive result.

Description

Technical Field

[0001] The invention relates to an unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation, and belongs to the field of speech recognition. Technical Background

[0002] As an important branch of artificial intelligence, speech recognition technology has made great progress in recent years. However, the development of speech recognition technology for resource-poor languages ​​still faces many challenges. Traditional speech recognition methods often require a large amount of labeled data for model training, and for data such as Lao, the acquisition and organization of labeled data is often a time-consuming and costly task. Therefore, exploring a speech recognition modeling method based on unsupervised learning is of great significance to promoting the development of speech recognition technology for resource-poor languages ​​such as Lao.

[0003] In recent years, speech recognition modeling methods based on phoneme segment-level representation have gradually attracted the attention of researchers. This method can better capture the key information in speech and improve the accuracy of recognition by converting continuous speech signals into discrete representations. At the same time, the discretized representation method also helps to reduce the complexity of the model and improve computational efficiency. However, most of the existing speech recognition modeling methods based on discrete representations rely on supervised learning, that is, a large amount of labeled data is required for model training. This is undoubtedly a huge obstacle for resource-poor languages ​​such as Lao. Therefore, it is of great theoretical significance and practical value to study an unsupervised speech recognition modeling method based on the discretization of phoneme segment-level representation.

[0004] In the process of exploring unsupervised speech recognition modeling methods, researchers have proposed a variety of different ideas and methods. Some researchers have tried to use technologies such as autoencoders to convert continuous speech signals into discrete phoneme segment-level representations, and then perform unsupervised learning through clustering methods. Other researchers use deep learning models such as adversarial generative networks to gradually learn the inherent laws and structures of speech signals through adversarial training of generators and discriminators. These methods have provided new ideas for unsupervised speech recognition modeling.

[0005] However, these methods are still relatively rarely used in resource-poor languages ​​such as Lao. Therefore, this study aims to explore an unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation and apply it to the Lao speech recognition task. It is hoped that this method can overcome the problem of insufficient Lao annotated data and improve the accuracy and robustness of Lao speech recognition. Specifically, the IFMF model is combined to perform phoneme granularity segmentation on speech, and the K-Means model is used to obtain discrete features. Then, an adversarial learning strategy is adopted to use a large amount of unlabeled audio data and text data, and the mapping relationship between audio representation and phonemes is learned by establishing an alignment relationship between audio and text. Finally, a self-training method will be used to construct a Lao speech recognition model based on unsupervised learning, and experimental verification and performance evaluation will be carried out. Summary of the invention

[0006] The technical problem to be solved by the present invention is: the present invention provides an unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation, which can overcome the problem of insufficient Laotian annotated data and improve the accuracy and robustness of Laotian speech recognition.

[0007] The method of the present invention uses discrete tags as input to replace traditional raw waveforms, acoustic features or self-supervised feature inputs. These tags are calculated by K-Means clustering of the phoneme granularity representation after segmentation by the IFMF model. Through the strategy of adversarial training, a large amount of unlabeled data is used to learn the mapping relationship between Lao audio and text. Compared with previous studies on unsupervised speech recognition, the method proposed in the present invention integrates multiple speech features and adopts a discretization method of phoneme segment-level representation. It not only takes into account the influence of tone information on Lao, but also minimizes the influence of information redundancy caused by fine-grained features on cross-modal modeling. Experiments on Lao datasets show that the proposed method achieves competitive results compared with traditional speech recognition methods.

[0008] The technical solution of the present invention is: an unsupervised speech recognition method based on discretization of phoneme segment-level representation, comprising:

[0009] Step 1: The IFMF model is used to extract phoneme features from the original audio through the speech feature discretization module, and then a K-Means clustering algorithm is trained to cluster the audio representation and obtain the cluster index as a discrete label; before extracting phoneme features from the original audio, data selection and preprocessing are performed. Using the constructed dataset, a Lao speech recognition dataset is constructed by combining unsupervised pre-training of large-scale unlabeled data and a verification and testing strategy of a small amount of labeled data. The total amount of the dataset is 1042.3 hours.

[0010] Step 2: Train a generator network G and a discriminator network C through the adversarial learning module, where the generator network generates samples, which are then judged by the discriminator network. The discriminator network is trained to distinguish whether the samples are from the generator network or from the real data distribution. The goal of the generator network is to produce samples that the discriminator network cannot distinguish.

[0011] Step 3: Use the language model to decode the model output based on phoneme discretization obtained by unsupervised training through the speech discrete representation decoder module.

[0012] As a further solution of the present invention, in step 1, an IFMF model is used to extract phoneme features from the original audio; the IFMF model fully utilizes the advantages of self-supervisory features and spectral features by interactively fusing multiple features to perform phoneme segmentation on Lao audio; then a K-Means clustering algorithm is trained to cluster the audio representation, and the clustering index is obtained as a discrete label as the input of the speech recognition model.

[0013] As a further solution of the present invention, step 2 comprises:

[0014] G represents a series of T segments S = [s1, ..., s T ] as input, and then mapped to a series of M phoneme sequences G(S) = [p1, ..., p M ]; The generator network predicts a distribution over the phoneme set O for each segment and outputs the phoneme with the highest probability; if the argmax predictions of consecutive segments result in the same phoneme, then one of these segments is sampled, M ≤ T;

[0015] The phoneme set O contains a silence label SIL so that the silent part in the speech audio is marked as silence; in the back-propagation process, the sampling points output by the generator network are back-propagated, and during the unsupervised training, the segment representation S is not modified, and the generator network is parameterized as a single-layer convolutional neural network;

[0016] The input of the discriminator network is from the real data distribution P r The one-hot vector sequence p representing the phoneme text obtained in r ~P r , or the output distribution sequence of the generator G(S); each input vector has |O| dimensions, representing the distribution of each phoneme segment; the discriminator network is a CNN that outputs a probability, indicating the likelihood that the sample comes from the data distribution;

[0017] In the experiment, we use the original GAN ​​objective with a gradient penalty, a segment smoothness penalty, and a phoneme diversity penalty, as shown in the following formula:

[0018]

[0019] Among them, p r ~P r represents the phonemic unlabeled text, G(S) represents the transcription output generated by the generator for the input speech segment S; for some unlabeled speech audio, the first term trains the discriminator to assign high probability to the true transcription, and the second term encourages the discriminator to assign low probability to the generator output; L gp is a gradient penalty term, l pd is a phoneme diversity loss; during training, the discriminator and the generator are updated alternately, and the predicted transcription from the generator and the phonemed unlabeled text are used alternately as batches;

[0020] Gradient penalty: The experiment penalizes the gradient norm of the input to the discriminator; this penalty is for random samples Calculated, these samples are linear combinations of the activation values ​​of real samples and fake samples; as shown in the following formula:

[0021]

[0022] Phone diversity loss: maximize the average softmax distribution H of the generator over the phoneme vocabulary G The entropy of (G(S)), which is calculated over a batch of B pronunciations, is as follows:

[0023]

[0024] The goal of the experiment is to build a speech recognition model that does not require any supervision. To this end, a cross-validation indicator that does not require labeled data is used, which is used for early stopping, random seed selection, and hyperparameter selection (λ, η); in this indicator, two quantities are considered: LM negative log-likelihood and vocabulary usage; as shown in the following formula:

[0025]

[0026] NLL LM is an indicator of the fluency of a given transcription, which is obtained by using a language model p trained on phonemic text data LM To measure, where p LM (p t ) is p LM (p t |p t-1 ,...,p1); vocabulary usage U(P) is the proportion of the phoneme vocabulary output by the model through Viterbi decoding; the model configuration chosen in this way produces phoneme sequences that score high under the language model but are not long.

[0027] As a further solution of the present invention, in step 3, a language model is used to decode the model output based on phoneme discretization obtained by unsupervised training; the model uses PyKaldi to construct WFSTs; WFST uses the model output as input and uses the same phonemizer as the preprocessing unlabeled text data to construct a mapping of phonemes to text; WFST is combined with a 4-grams language model to retain only 4-grams that appear more than 3 times.

[0028] As a further solution of the present invention, in step 3, during decoding, the acoustic ratio is provided as a parameter to the Kaldi decoder, and a scalar v is added to the blank token emission; the optimal weights of the two parameters are adjusted by minimizing the amount of measuring the output fluency and the realism of the model output; the following amount is minimized on an unlabeled development set - assuming that the phoneme-based model is decoded into syllables, as shown in the following formula:

[0029]

[0030] in, is the output of the Viterbi model, The output of WFST based on syllables is converted into phonemes, H LM (p j ) is the entropy of the language model, ED is the edit distance, and μ=0.03.

[0031] The beneficial effects of the present invention are:

[0032] The present invention provides an unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation. The method uses discrete tags as input to replace traditional original waveforms, acoustic features or self-supervised feature inputs. These tags are calculated by K-Means clustering of phoneme granularity representation after segmentation by the IFMF model. Through the strategy of adversarial training, a large amount of unlabeled data is used to learn the mapping relationship between Lao audio and text. Compared with the existing ASR model, the proposed unsupervised speech recognition method based on representation discretization performs better than other methods on both the development set and the test set. Compared with the optimal result of the benchmark model, the proposed method reduces the word error rate by 1.61% on the development set and by 2.63% on the test set. Experiments on Lao data sets show that the proposed method achieves competitive results compared with traditional speech recognition methods. The present invention can overcome the problem of insufficient Lao labeled data and improve the accuracy and robustness of Lao speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a general design block diagram of an unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation proposed by the present invention;

[0034] Figure 2 This is a recognition example of an unsupervised speech recognition method based on discretization of phoneme segment-level representation proposed by the present invention; DETAILED DESCRIPTION

[0035] The following is a further description of an unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are in a very simplified form and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0036] Example 1: Figure 1 As shown, an unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation includes:

[0037] Step 1: The IFMF model is used to extract phoneme features from the original audio through the speech feature discretization module, and then a K-Means clustering algorithm is trained to cluster the audio representation and obtain the cluster index as a discrete tag; in the invented speech recognition model, instead of using the original continuous speech or acoustic features, discrete tags are used as input. The IFMF model is used to extract phoneme features from the original audio; the IFMF model fully utilizes the advantages of self-supervised features and spectral features to perform phoneme segmentation on Lao audio through multi-feature interactive fusion; then a K-Means clustering algorithm is trained to cluster the audio representation and obtain the cluster index as a discrete tag as the input of the speech recognition model.

[0038] Before extracting phoneme features, the present invention first performs data selection and preprocessing.

[0039] The speech dataset used for training and verification in the experiment was first obtained from a number of Internet websites, and then a small batch of carefully crafted annotated datasets was formed by applying speech synthesis technology and manual annotation. Specifically, in the speech synthesis link, the advanced speech synthesis API provided by the Microsoft platform was used to generate standard Laotian label data with high-quality audio characteristics; in terms of manual annotation, this study worked closely with Lao professional linguists who cooperated with the laboratory. These experts used the professional platform provided by the laboratory to accurately annotate the selected unlabeled data and conduct strict manual inspections on the synthesized audio data to ensure the high accuracy and reliability of the dataset.

[0040] By integrating the unsupervised pre-training of large-scale unlabeled data and the verification and testing strategy of a small amount of labeled data, a Lao speech recognition dataset was successfully constructed. The total amount of the dataset is 1042.3 hours. Its detailed information is shown in Table 1.

[0041] Table 1: Experimental dataset

[0042]

[0043] The present invention experimentally compares the data size of different data formats, including original waveforms, traditional acoustic features, features based on self-supervised learning, speech features of multi-feature interactive fusion, and discrete tags. Take T seconds of single-channel speech pronunciation as an example. The size of the original waveform data depends on the sampling rate and the audio sample encoding. The settings commonly used in speech recognition, that is, 16kHz wav format audio, are adopted here. For acoustic features, D-dimensional floating-point vectors are used, and the frame shift is 10ms, which is for a rate of 100 frames per second, where an example value of D is the conventional 80. For self-supervised learning features, a single-layer hidden embedding in the XLS-R model is used as an example of self-supervised learning features. For multi-feature interactive fusion, the last layer of hidden embedding of the IFMF model is used, and for discrete tags, the K-Means model is used to cluster the audio features after segmentation by the IFMF model.

[0044] Step 2: train a generator network G and a discriminator network C through an adversarial learning module, wherein the generator network generates samples, which are then judged by the discriminator network. The discriminator network is trained to distinguish whether the samples are from the generator network or from the real data distribution. The goal of the generator network is to generate samples that the discriminator network cannot distinguish. Step 2 includes:

[0045] G represents a series of T segments S = [s1, ..., s T ] as input, and then mapped to a series of M phoneme sequences G(S) = [p1, ..., p M ]; The generator network predicts a distribution over the phoneme set O for each segment and outputs the phoneme with the highest probability; if the argmax predictions of consecutive segments result in the same phoneme, then one of these segments is sampled, M ≤ T;

[0046] The phoneme set O contains a silence label SIL to mark the silent part of the speech audio as silence; if there is no silence label, the model will reuse the specific phoneme to mark silence, which leads to a significant reduction in performance because it interferes with the subsequent language model decoding. During the back-propagation process, back-propagation is performed through the sample points output by the generator network. During unsupervised training, the segment representation S is not modified. The generator network is parameterized as a single-layer convolutional neural network;

[0047] The input of the discriminator network is from the real data distribution P r The one-hot vector sequence p representing the phoneme text obtained in r ~P r, or the output distribution sequence of the generator G(S); each input vector has |O| dimensions, representing the distribution of each phoneme segment; the discriminator network is a CNN that outputs a probability, indicating the likelihood that the sample comes from the data distribution;

[0048] In the experiment, we use the original GAN ​​objective with a gradient penalty, a segment smoothness penalty, and a phoneme diversity penalty, as shown in the following formula:

[0049]

[0050] Among them, p r ~P r represents the phonemic unlabeled text, G(S) represents the transcription output generated by the generator for the input speech segment S; for some unlabeled speech audio, the first term trains the discriminator to assign high probability to the true transcription, and the second term encourages the discriminator to assign low probability to the generator output; L gp is a gradient penalty term, L pd is a phoneme diversity loss; during training, the discriminator and the generator are updated alternately, and the predicted transcription from the generator and the phonemed unlabeled text are used alternately as batches;

[0051] Gradient penalty: The experiment penalizes the gradient norm of the input to the discriminator; this penalty is for random samples Calculated, these samples are linear combinations of the activation values ​​of real samples and fake samples; as shown in the following formula:

[0052]

[0053] Phone diversity loss: maximize the average softmax distribution H of the generator over the phoneme vocabulary G The entropy of (G(S)), which is calculated over a batch of B pronunciations, is as follows:

[0054]

[0055] The goal of the experiment is to build a speech recognition model that does not require any supervision. To this end, a cross-validation indicator that does not require labeled data is used, and this indicator is used for early stopping, random seed selection, and hyperparameter selection (λ, η); in this indicator, two quantities are considered: LM negative log likelihood (NLL) and vocabulary usage; as shown in the following formula:

[0056]

[0057] NLL LM is an indicator of the fluency of a given transcription, which is obtained by using a language model p trained on phonemic text data LM To measure, where pLM (p t ) is p LM (p t |p t-1 ,...,p1); vocabulary usage U(P) is the proportion of the phoneme vocabulary output by the model through Viterbi decoding; the model configuration chosen in this way produces phoneme sequences that score high under the language model but are not long.

[0058] Step 3: Use the language model to decode the model output based on phoneme discretization obtained by unsupervised training through the speech discrete representation decoder module.

[0059] In step 3, a language model is used to decode the model output based on phoneme discretization obtained by unsupervised training; the model uses PyKaldi to construct WFSTs; WFST uses the model output as input and uses the same phonemizer as the preprocessing unlabeled text data to construct a mapping of phonemes to text; WFST is combined with a 4-grams language model to retain only 4-grams that appear more than 3 times.

[0060] During decoding, the acoustic scale is fed as a parameter to the Kaldi decoder, and a scalar v is added to the empty token emission; the optimal weights of these two parameters are adjusted by minimizing a measure of output fluency and how realistic the model output is; the following quantity is minimized on an unlabeled development set - assuming that the phoneme-based model decodes into syllables, as shown in the following equation:

[0061]

[0062] in, is the output of the Viterbi model, The output of WFST based on syllables is converted into phonemes, H LM (p j ) is the entropy of the language model, ED is the edit distance, and μ=0.03.

[0063] The present invention designs a comparative experiment with other schemes to prove the effectiveness of the proposed method. Table 2 shows the evaluation results of the character error rate (CER) of the proposed method and other common speech recognition models (Wenet, Speechbrain, DeepSpeech, SpecAugment, NST, Vq-Wav2vec, Wav2vec2.0, Wav2vec-U, Wav2vec-U+ST). The present invention achieves the lowest word error rate on both the development set and the test set, proving the effectiveness of the invention in the ASR error correction task.

[0064] Table 2: Comparative experiment on Lao speech recognition based on existing data

[0065]

[0066] From the analysis of the results in Table 2, it can be seen that the proposed unsupervised speech recognition method based on representation discretization performs better than other methods on both the development set and the test set. The performance of Wenet, Speechbrain, DeepSpeech2 and SpecAugment is relatively poor due to the small amount of training data and the inability to give full play to the advantages of deep learning. All experimental configurations of NST, VqWav2vec, Wav2vec2.0 and Wav2vec-U are consistent with the proposed method. Compared with the optimal results of the baseline model, the proposed method reduces the word error rate by 1.61% on the development set and 2.63% on the test set. These results show the excellent performance of the proposed method in the Lao speech recognition task.

[0067] In order to verify the influence of different K values ​​on Lao speech recognition, the present invention conducts experiments on the basis of the benchmark model with cluster numbers of 25, 35, 50, 75 and 100 respectively.

[0068] Table 3: Effect of different K values ​​on speech recognition performance

[0069]

[0070]

[0071] From the analysis of the results in Table 3, it can be seen that the larger or better the K value is, the better the performance of speech recognition is. First, a larger K value means more cluster centers, which will make the algorithm divide the audio features more finely. There are 104 phonemes in IPA, and some phonemes are pronounced too similarly, such as occlusive sounds, pauses, and glottal stops. These phonemes are relatively weak, usually accompanied by other sounds, and are difficult to distinguish based on sound signals without considering surrounding phonemes. When K = 75 or K = 100, almost every phoneme forms a class. In this case, the algorithm will pay too much attention to the noise and outliers in the data, making the model too sensitive to the details of the phoneme features, resulting in overfitting problems. When K = 25 or K = 35, the algorithm cannot fully capture the structural differences in the audio data, and will mistakenly classify different audio features into the same cluster. The clustering is too coarse and cannot fully capture the details of the audio data, resulting in underfitting problems and reducing the performance of speech recognition. Since there are about 60 commonly used phonemes, when K = 50, this division makes the audio features within each cluster more similar, while the feature differences between different clusters are more obvious. The selection of this K value can classify noise data or outliers into fewer clusters, which can capture enough details of the audio data without causing overfitting. According to the experimental results, the K value used in all experiments in this study is 50.

[0072] In order to verify the effectiveness of the proposed method, Table 4 compares the proposed method with four different features (FBank, FBankD, XLS-R, XLS-RD) after discretization of data preprocessing.

[0073] Table 4: Effects of different feature discretization on speech recognition

[0074]

[0075] Analysis of the results in Table 4 shows that compared with these methods, the method proposed in this study performs best. Comparing FBank and FBankD, XLS-R and XLS-RD, it can be found that the discretized features have better performance. This is because the quality of audio data is often affected by various noises and interferences. The discretized speech features convert continuous speech signals into discrete speech signals, reducing the impact of noise and interference on speech feature representation, so that even if there are some small fluctuations or changes in the speech signal, the discretized features can still remain stable and consistent, allowing the model to better understand the semantic information brought by the speech signal itself. Comparing FBankD, XLS-RD and the audio representation extracted based on the IFMF model, it can be seen that only traditional acoustic features or only self-supervised learning representations cannot fully extract the information of Lao, a tonal language. Traditional acoustic features are more inclined to the recognition of vowels, because compared with consonants, vowels are long and coherent, and self-supervised learning features can make up for the performance of traditional acoustic features in consonant recognition. On this basis, the IFMF model integrates the Pitch feature to enhance the tonal characteristics of Lao audio. Therefore, the method proposed in the present invention aims at solving the problems of Lao tone complexity and pronunciation diversity. After integrating the Pitch feature, it is divided into a representation of phoneme granularity and then discretized, which is more conducive to the recognition effect of the model. This method can more fully consider the tone characteristics of Lao and achieve better results in overall performance.

[0076] The present invention Figure 2 The example of how discretization of speech features can improve model recognition performance is shown in the text. The Lao speech recognition model without audio feature discretization recognizes incorrectly and The main reason is and and They are high consonants and low consonants respectively. and They are single consonants and compound consonants respectively, and their pronunciations are similar, which leads to the inability of speech recognition to distinguish them. The method proposed in the present invention, in addition to integrating the fundamental frequency characteristics of the audio signal in the feature extraction stage, also discretizes the audio features. By comparing their speech features, they are discretized into different clusters. During recognition, the model will give priority to candidate texts in the same cluster as the input phonemes, thereby improving the accuracy and robustness of the speech recognition model when processing similar phonemes. The experimental results show that in the training stage, the Lao speech recognition model after discretizing the audio features can better distinguish texts with similar pronunciations, thereby achieving better recognition effects.

[0077] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation, characterized in that: include: Step 1: The IFMF model is used to extract phoneme features from the original audio through the speech feature discretization module, and then a K-Means clustering algorithm is trained to cluster the audio representations and obtain the cluster index as a discrete label; Step 2: Train a generator network G and a discriminator network C through the adversarial learning module, where the generator network generates samples, which are then judged by the discriminator network. The discriminator network is trained to distinguish whether the samples are from the generator network or from the real data distribution. The goal of the generator network is to produce samples that the discriminator network cannot distinguish. Step 3: Use the language model to decode the model output based on phoneme discretization obtained by unsupervised training through the speech discrete representation decoder module.

2. The unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation according to claim 1 is characterized in that: In the step 1, the IFMF model is used to extract phoneme features from the original audio; The IFMF model fully utilizes the advantages of self-supervised features and spectral features to perform phoneme segmentation on Lao audio by interactively fusing multiple features. It then trains a K-Means clustering algorithm to cluster audio representations and obtain cluster indexes as discrete tags as input to the speech recognition model.

3. The unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation according to claim 1 is characterized in that: The step 2 comprises: G represents a series of T segments S = [s1, ..., s T ] as input, and then mapped to a series of M phoneme sequences G(S) = [p1, ..., p M ]; The generator network predicts a distribution over the phoneme set O for each segment and outputs the phoneme with the highest probability; if the argmax predictions of consecutive segments result in the same phoneme, then one of these segments is sampled, M ≤ T; The phoneme set O contains a silence label SIL so that the silent part in the speech audio is marked as silence; in the back-propagation process, the sampling points output by the generator network are back-propagated, and during the unsupervised training, the segment representation S is not modified, and the generator network is parameterized as a single-layer convolutional neural network; The input of the discriminator network is from the real data distribution P r The one-hot vector sequence p representing the phoneme text obtained in r ~P r , or the output distribution sequence of the generator G(S); each input vector has |O| dimensions, representing the distribution of each phoneme segment; the discriminator network is a CNN that outputs a probability, indicating the likelihood that the sample comes from the data distribution; In the experiment, we use the original GAN ​​objective with a gradient penalty, a segment smoothness penalty, and a phoneme diversity penalty, as shown in the following formula: Among them, p r ~P r represents the phonemic unlabeled text, G(S) represents the transcription output generated by the generator for the input speech segment S; for some unlabeled speech audio, the first term trains the discriminator to assign high probability to the true transcription, and the second term encourages the discriminator to assign low probability to the generator output; L gp is a gradient penalty term, L pd is a phoneme diversity loss; during training, the discriminator and the generator are updated alternately, and the predicted transcription from the generator and the phonemed unlabeled text are used alternately as batches; Gradient penalty: The experiment penalizes the gradient norm of the input to the discriminator; this penalty is for random samples Calculated, these samples are linear combinations of the activation values ​​of real samples and fake samples; as shown in the following formula: Phone diversity loss: maximize the average softmax distribution H of the generator over the phoneme vocabulary G The entropy of (G(S)), which is calculated over a batch of B pronunciations, is as follows: The goal of the experiment is to build a speech recognition model that does not require any supervision. To this end, a cross-validation indicator that does not require labeled data is used, which is used for early stopping, random seed selection, and hyperparameter selection (λ, η); in this indicator, two quantities are considered: LM negative log-likelihood and vocabulary usage; as shown in the following formula: NLL LM is an indicator of the fluency of a given transcription, which is obtained by using a language model p trained on phonemic text data LM To measure, where p LM (p t ) is p LM (p t |p t-1 ,...,p1); vocabulary usage U(P) is the proportion of the phoneme vocabulary output by the model through Viterbi decoding; the model configuration chosen in this way produces phoneme sequences that score high under the language model but are not long.

4. The unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation according to claim 1, characterized in that: In the step 3, a language model is used to decode the model output based on phoneme discretization obtained through unsupervised training; The model uses PyKaldi to build WFSTs; WFST takes the model output as input and uses the same phonemeizer as used to preprocess unlabeled text data to build a mapping of phonemes to text; WFST is combined with a 4-grams language model to only retain 4-grams that appear more than 3 times.

5. The unsupervised speech recognition modeling method based on discretization of phoneme segment-level representation according to claim 1, characterized in that: In step 3, during decoding, the acoustic ratio is provided as a parameter to the Kaldi decoder, and a scalar v is added to the blank token emission; the optimal weights of these two parameters are adjusted by minimizing the amount of measuring the fluency of the output and the realism of the model output; the following amount is minimized on an unlabeled development set - assuming that the phoneme-based model is decoded into syllables, as shown in the following formula: in, is the output of the Viterbi model, The output of WFST based on syllables is converted into phonemes, H LM (p j ) is the entropy of the language model, ED is the edit distance, and μ=0.03.

Images