Voice pre-training method and device, storage medium and electronic equipment
Through the assisted clustering of phonomic phoneme information, clustering tags are generated and speech pre-trained models are trained, which solves the problem of insufficient performance of the existing self-supervised learning model in downstream speech tasks, and achieves better speech feature clustering and task performance.
Patent Information
- Application Number
- CN202411184191.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-26
AI Technical Summary
The existing self-supervised learning model lacks performance in downstream voice tasks, and a method that can improve the controllability and interpretability of speech feature clustering results is urgently needed to obtain better speech pre-training effects.
By using phoneme information assisted clustering, we obtain speech samples and phoneme data, extract the speech characteristics of speech frames, divide speech segments, determine the target characteristics of phonemes, perform clustering, generate clustering tags, and use these tags to train preset networks to train Model, obtain the pre-trained speech model.
It improves the controllability and interpretability of the clustering results of speech characteristics, helps voice self-supervised training, and significantly improves the performance of downstream voice tasks.
Smart Images

Figure CN119028327A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech processing technology, and in particular to a speech pre-training method, device, storage medium and electronic equipment. Background Art
[0002] In recent years, pre-training based on self-supervised learning has greatly promoted the research progress of speech processing and shown great potential in a wide range of downstream speech tasks.
[0003] Existing self-supervised learning models are usually pre-trained on a large amount of unlabeled data in an application-independent manner, and then fine-tuned on downstream speech tasks by updating the entire network or only updating a small number of parameters. The learned representations have been found to be useful for a range of speech tasks, such as automatic speech recognition, text-to-speech, speaker verification, speech enhancement, etc. Currently, there is an urgent need for a solution that enables self-supervised learning models to achieve better performance in downstream speech tasks. Summary of the invention
[0004] The embodiment of this specification provides a speech pre-training method, which can make the clustering results of speech features more controllable and more interpretable by using speech phoneme information to assist clustering, which is helpful for speech self-supervision training, and thus enables the model to achieve better performance in downstream speech tasks. The method includes:
[0005] Acquire a voice sample and phoneme data corresponding to the voice sample;
[0006] Extracting speech features of each speech frame in the speech sample, and dividing the speech sample into at least one speech segment according to the speech features and the phoneme data, wherein one speech segment corresponds to one phoneme;
[0007] Determine the target feature of each phoneme according to the speech features of each speech frame in the speech segment corresponding to the same phoneme;
[0008] The target feature of each of the phonemes is used as an initial cluster center, and based on each of the initial cluster centers, the speech features of each speech frame in the speech sample are clustered to obtain a corresponding cluster label;
[0009] The preset network model is trained using the clustering labels to obtain a speech pre-training model.
[0010] Furthermore, in some implementations, dividing the speech sample into at least one speech segment according to the speech feature and the phoneme data includes:
[0011] The speech features of each speech frame in the speech sample are aligned with the phoneme data to determine the speech segment corresponding to each phoneme in the time dimension.
[0012] Furthermore, in some implementations, determining the target feature of each phoneme according to the speech feature of each speech frame in the speech segment corresponding to the same phoneme includes:
[0013] The speech features of each speech frame in the speech segment corresponding to the same phoneme are averaged to obtain the target feature of each of the phonemes.
[0014] Furthermore, in some implementations, clustering the speech features of each speech frame in the speech sample based on each of the initial cluster centers to obtain a corresponding cluster label includes:
[0015] Calculating the distance between each of the initial cluster centers and the speech features of each speech frame in the speech sample;
[0016] Determine the initial cluster center closest to each speech frame in the speech sample according to the distance, so as to assign each speech frame in the speech sample to the speech feature cluster where the initial cluster center is located;
[0017] The cluster center of each speech feature cluster is updated, and each speech frame in the speech sample is clustered based on the updated cluster centers until a clustering termination condition is met, thereby obtaining a cluster label corresponding to each speech frame in the speech sample.
[0018] Furthermore, in some implementations, updating the cluster center of each speech feature cluster includes:
[0019] Averaging the speech features of all speech frames in each speech feature cluster to obtain corresponding average features;
[0020] The average features are used as the updated cluster centers of the speech feature clusters.
[0021] Furthermore, in some implementations, the using the clustering labels to train a preset network model to obtain a speech pre-training model includes:
[0022] Randomly masking the speech features of each speech frame in the speech sample to obtain corresponding masked features;
[0023] Inputting each of the mask features into the preset network model to obtain a predicted label for each of the speech frames;
[0024] The loss value between the predicted label and the cluster label is calculated, and the model parameters of the preset network model are iteratively trained according to the loss value. When the iteration termination condition is met, the speech pre-training model is obtained.
[0025] Further, in some embodiments, the preset network model includes an encoder and a prediction head;
[0026] The step of inputting each of the mask features into the preset network model to obtain a predicted label of each of the speech frames includes:
[0027] Inputting each of the mask features into the encoder to obtain a corresponding hidden state;
[0028] Each hidden state is mapped to a label space through the prediction head to obtain a predicted label for each speech frame.
[0029] Further, in some implementations, the speech feature includes a first Mel-spectrogram feature;
[0030] The extracting speech features of each speech frame in the speech sample includes:
[0031] The speech samples are divided into frames, and the first Mel-spectrogram feature of each speech frame is extracted.
[0032] Further, in some implementations, the speech feature includes a second Mel-spectrogram feature;
[0033] The extracting speech features of each speech frame in the speech sample includes:
[0034] The text sequence corresponding to the speech sample is input into the trained feature extraction model to obtain the second Mel-spectrogram feature of each speech frame in the speech sample.
[0035] Further, in some embodiments, the feature extraction model includes at least a first encoder, a second encoder and a decoder;
[0036] The step of inputting the text sequence corresponding to the speech sample into a trained feature extraction model to obtain a second Mel-spectrogram feature of each speech frame in the speech sample includes:
[0037] Using the first encoder to extract features from the text sequence to obtain a first latent variable feature;
[0038] Using the second encoder to extract features from the linear spectrum of the speech sample to obtain a second latent variable feature;
[0039] The decoder is used to decode the second latent variable feature to obtain a second Mel-spectrogram feature of each speech frame in the speech sample.
[0040] Furthermore, in some implementations, the acquiring of the voice sample and the phoneme data corresponding to the voice sample includes:
[0041] Acquire the voice sample and the natural text corresponding to the voice sample;
[0042] The natural text is subjected to phoneme processing to obtain the phoneme data.
[0043] The embodiment of this specification also proposes a speech pre-training device, the device comprising:
[0044] A voice data acquisition module, used to acquire a voice sample and phoneme data corresponding to the voice sample;
[0045] A speech segment division module, used for extracting speech features of each speech frame in the speech sample, and dividing the speech sample into at least one speech segment according to the speech features and the phoneme data, wherein one speech segment corresponds to one phoneme;
[0046] A target feature determination module, used to determine the target feature of each phoneme according to the speech features of each speech frame in the speech segment corresponding to the same phoneme;
[0047] A clustering label generation module, used to take the target feature of each of the phonemes as an initial clustering center, and cluster the speech features of each speech frame in the speech sample based on each of the initial clustering centers to obtain a corresponding clustering label;
[0048] The model pre-training module is used to train the preset network model using the clustering labels to obtain a speech pre-training model.
[0049] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.
[0050] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.
[0051] The embodiments of the present specification also provide a computer program product, wherein the computer program product stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the above method steps.
[0052] In an embodiment of the present specification, by obtaining a speech sample and the phoneme data corresponding to the speech sample, the speech features of each speech frame in the speech sample are extracted, and the speech sample is divided into at least one speech segment according to the speech features and the phoneme data, wherein one speech segment corresponds to one phoneme. Then, the target features of each phoneme are determined according to the speech features of each speech frame in the speech segment corresponding to the same phoneme. Furthermore, the target features of each phoneme are used as the initial clustering center, and the speech features of each speech frame in the speech sample are clustered based on each initial clustering center to obtain a corresponding clustering label. Finally, the clustering label is used to train the preset network model to obtain a speech pre-training model. By adopting the speech pre-training method proposed in the embodiment of the present specification, by using speech phoneme information to assist clustering, the clustering results of speech features can be made more controllable and more interpretable, which is helpful for speech self-supervision training, and thus enables the obtained speech pre-training model to obtain better performance in downstream speech tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 A schematic diagram of a system architecture for applying the embodiments of this specification;
[0054] Figure 2 A flowchart of a speech pre-training method provided in an embodiment of this specification;
[0055] Figure 3 A flowchart of speech feature extraction provided in an embodiment of this specification;
[0056] Figure 4 A flowchart of a self-supervised pre-training method provided in an embodiment of this specification;
[0057] Figure 5 A schematic diagram of another speech pre-training method provided in an embodiment of this specification;
[0058] Figure 6 A schematic diagram of the structure of a speech pre-training device provided in an embodiment of this specification;
[0059] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0061] Figure 1 A schematic diagram of an architecture to which the embodiments of this specification can be applied is shown.
[0062] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices such as a smart phone 101, a portable computer 102, a desktop computer 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The terminal device may be various electronic devices with data processing functions, and the electronic device may have a display screen, which is used to display the speech features of each speech segment in the speech sample, the target features of each phoneme, the clustering labels corresponding to each speech segment, and the obtained speech pre-training model, etc.
[0063] It is understandable that the terminal device can also be various electronic devices with sound collection components and sound playback components. For example, the voice samples used in the voice pre-training process can be collected in real time by the sound collection component. Of course, the voice samples can also be pre-recorded by other voice recording devices and stored in the terminal device or server 105, and then called when the pre-training model is used. This specification does not limit this. When the obtained voice pre-training model is used for downstream tasks such as speech synthesis, the synthesized speech can be displayed to the tester through the sound playback component.
[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.
[0065] Speech pre-training refers to the process of using a large amount of unlabeled or labeled data to perform preliminary training on the model before a specific speech task. This allows the model to learn general speech representations, which can be transferred to different downstream tasks such as speech recognition, speaker verification, sentiment analysis, etc.
[0066] See also Figure 2 , which is a flow chart of a speech pre-training method provided in the embodiment of this specification. In the embodiment of this specification, the speech pre-training method is applied to a speech pre-training device or an electronic device equipped with a speech pre-training device. Figure 2 The process shown is described in detail, and the speech pre-training method may specifically include the following steps:
[0067] S202, obtaining a voice sample and phoneme data corresponding to the voice sample;
[0068] In one or more embodiments of the present specification, a speech sample refers to real speech data used for speech self-supervision training, such as a collected original speech signal, or a speech signal that has been pre-processed by denoising, pre-emphasis, etc.
[0069] The phoneme data corresponding to the voice sample is obtained by performing phoneme processing on the natural text corresponding to the voice sample. The natural text refers to the text content corresponding to the voice sample, that is, the natural text can present the content in the voice sample in a text format.
[0070] For example, the natural text may be standardized, such as removing punctuation marks and word segmentation, and the decomposed words may be converted into corresponding pinyin representations, and then the pinyin may be converted into phonemes using a pre-built phoneme dictionary or phoneme mapping table. In order to better simulate the rhythmic features of natural language, intonation and stress marks may be added to the phonemes, which is not limited in this specification.
[0071] Of course, you can also use a trained neural network model to convert natural text into a phoneme sequence. The neural network model can be a recurrent neural network, a long short-term memory network, etc. This specification does not limit the type of neural network model, as long as it can achieve the conversion of text to phonemes. For example, the natural text corresponding to the speech sample is used as the input of the phoneme embedding layer, and the corresponding phoneme sequence is output.
[0072] S204, extracting speech features of each speech frame in the speech sample, and dividing the speech sample into at least one speech segment according to the speech features and the phoneme data, wherein one speech segment corresponds to one phoneme;
[0073] In one or more embodiments of the present specification, the speech features may adopt Mel-spectrogram features. Of course, the speech features may also be extracted from the speech samples using a trained neural network model such as a convolutional network model, and the embodiments of the present specification do not limit this.
[0074] Taking the speech feature as a Mel spectrum feature as an example, it may include a first Mel spectrum feature and / or a second Mel spectrum feature. The first Mel spectrum feature may be obtained by performing signal processing and conversion on the speech sample, and the second Mel spectrum feature may be obtained by converting the text information corresponding to the speech sample using a trained neural network model.
[0075] Optionally, the speech samples are framed and the first Mel spectrum feature of each speech frame is extracted. For example, the speech frames after pre-emphasis, framing and windowing can be short-time Fourier transformed to obtain the amplitude spectrum of the speech frame, and then the amplitude spectrum after logarithmic transformation is output through a Mel filter bank to output the corresponding first Mel spectrum feature.
[0076] Optionally, the text sequence corresponding to the speech sample is input into a trained feature extraction model to obtain a second Mel-spectrogram feature of each speech frame in the speech sample, wherein the feature extraction model is used to convert the input text information into corresponding Mel-spectrogram information.
[0077] See also Figure 3 , is a flow chart of a speech feature extraction process provided by an embodiment of this specification. The feature extraction model at least includes a first encoder, a second encoder and a decoder. Figure 3 As shown, the method comprises the following steps:
[0078] S302, using the first encoder to perform feature extraction on the text sequence to obtain a first latent variable feature;
[0079] Optionally, the first encoder includes a text encoder and a projection layer. Specifically, the text encoder maps the text sequence to a text feature sequence, and the projection layer linearly projects the text feature sequence to obtain the first latent variable feature.
[0080] Among them, the text encoder can be a transformer encoder based on relative position encoding, which linearly projects the text sequence through the projection layer to generate the mean and variance of the prior distribution, and then determines the first latent variable feature according to the mean and variance.
[0081] S304, using the second encoder to extract features from the linear spectrum of the speech sample to obtain a second latent variable feature;
[0082] Optionally, the second encoder and decoder can both be constructed based on the WaveNet module, and are used to convert the spectral features of the speech sample into corresponding latent variable features. The second encoder takes the linear spectrum of the speech sample as input and outputs the second latent variable features. For example, a linear spectrum can be obtained from the speech sample by short-time Fourier transform as input to the second encoder.
[0083] S306: Use the decoder to decode the second latent variable feature to obtain a second Mel-spectrogram feature of each speech frame in the speech sample.
[0084] The decoder is used to convert the second latent variable feature into a Mel spectrum. Specifically, the second latent variable feature output by the second encoder is used as the input of the decoder, and the second latent variable feature is mapped into a latent variable feature representation for inputting the WaveNet module using a conversion layer in the decoder. The conversion layer may include a fully connected layer, a convolutional layer, a self-attention mechanism layer, etc. The second latent variable feature is nonlinearly transformed through the conversion layer to better meet the needs of the subsequent WaveNet module.
[0085] The Mel spectrum frame corresponding to the latent variable feature representation is generated through the WaveNet module. Specifically, the WaveNet module can generate the corresponding Mel spectrum frame frame by frame through its own complex conditional probability modeling capabilities. The WaveNet module can optimize the Mel spectrum of the current frame based not only on the latent variable features of the previous moment, but also on the Mel spectrum of the previous frames and other contextual information, so that the generated second Mel spectrum feature is more temporally coherent and natural.
[0086] In one or more embodiments of the present specification, after obtaining the speech features of each speech frame in the speech sample, the speech features of each speech frame in the speech sample and the phoneme data corresponding to the speech sample can be aligned to determine the speech segment corresponding to each phoneme in the time dimension, thereby dividing the speech sample into at least one speech segment, and one speech segment corresponds to one phoneme.
[0087] Optionally, the MFA (Monolingual Forced Alignment) method can be used to determine the position of each phoneme on the time axis, and then find all speech segments containing specific phonemes. For example, the forced alignment process is performed using the trained acoustic model and language model, combined with the speech features of each speech frame and the phoneme data corresponding to the speech sample. Among them, the Viterbi algorithm (a dynamic programming algorithm) can be used to find the phoneme corresponding to each speech frame and its time position in the speech sample. The output result is an estimate of the time boundary of each phoneme, that is, the start time and end time of each phoneme.
[0088] Among them, the acoustic model can map the speech features of each speech frame to a phoneme sequence, and the acoustic model can be GMM-HMM (Gaussian Mixture Model-Hidden Markov Model), DNN-HMM (Deep Neural Network-Hidden Markov Model), etc. The language model is used to predict a reasonable phoneme sequence, which can be an N-gram (statistical language) model or a language model based on a neural network, and this specification embodiment does not limit this.
[0089] In this embodiment, by obtaining the correspondence between phonemes and speech features in the time dimension, an accurate boundary at the phoneme level can be provided, which is helpful for clustering in subsequent steps.
[0090] S206, determining a target feature of each phoneme according to the speech features of each speech frame in the speech segment corresponding to the same phoneme;
[0091] In one or more embodiments of the present specification, for speech segments corresponding to the same phoneme, the speech features in these speech segments are similar, so the speech features of each speech frame in the speech segment corresponding to the same phoneme can be averaged, such as calculating the average value of all speech features corresponding to the same phoneme to obtain an average feature, which is the target feature of the phoneme. Similarly, the target feature of each phoneme in the speech sample can be obtained, so that the same phoneme in different speech segments has a common feature representation.
[0092] For example, the natural text corresponding to the speech sample is "I have a dream", where the phonemes corresponding to "have" include "y" and "ou3", and the phonemes corresponding to "one" include "y" and "i2". The speech features of each speech frame in all speech segments corresponding to the phoneme "y" can be averaged to use the obtained average features to assist subsequent clustering and improve the clustering effect. In addition, the averaging operation of speech features reduces the variability of the data, which can make the cluster center more stable and easier to converge.
[0093] S208, taking the target feature of each of the phonemes as an initial cluster center, and clustering the speech features of each speech frame in the speech sample based on each of the initial cluster centers to obtain a corresponding cluster label;
[0094] It should be noted that during the speech pre-training process, it is necessary to first generate a discretized target sequence for all unlabeled speech frames through clustering, and then use a self-supervised pre-training method such as MLM (Masked Langauge Model) to let the model learn the key features of the speech sample. In this process, the clustering result of the speech frame is related to the selection of the clustering initial point. Importantly, the clustering effect of the speech frame will directly affect the effect of pre-training. Therefore, in one or more embodiments of the present specification, the clustering initial point can be determined by introducing the phoneme information of the speech sample to optimize the clustering effect, thereby improving the speech pre-training effect.
[0095] Accordingly, the cluster labels in the embodiments of this specification are not manually labeled, but are automatically generated by a clustering algorithm, and the clustering algorithm may be K-means clustering, spectral clustering, and the like.
[0096] In one or more embodiments of the present specification, taking K-means clustering as an example, the target feature of each phoneme calculated in step S206 may be used as an initial clustering center, wherein each initial clustering center represents a cluster.
[0097] Exemplarily, the distance between each initial cluster center and the speech feature of each speech frame in the speech sample is calculated, and the initial cluster center closest to each speech frame in the speech sample is determined according to the distance, so as to assign each speech frame in the speech sample to the speech feature cluster where the initial cluster center is located. Then, the cluster center of each speech feature cluster is updated, and each speech frame in the speech sample is clustered based on the updated cluster centers until the clustering termination condition is met, and the cluster label corresponding to each speech frame in the speech sample is obtained.
[0098] For each speech feature of a speech frame in the speech sample, the distance between the feature and each initial cluster center is calculated, and the feature is assigned to the cluster with the closest initial cluster center. This assignment process is equivalent to assigning a cluster label to each speech frame, which indicates which cluster the speech frame belongs to.
[0099] When updating the cluster center of each speech feature cluster, for each speech feature cluster, the speech features of all speech frames in each speech feature cluster are averaged to obtain the corresponding average features, and each average feature is used as the updated cluster center of each speech feature cluster. For example, the average value of the speech features of all speech frames in each speech feature cluster is calculated, and the average value is used as the new cluster center. The allocation and update are repeated until the cluster termination condition is met, such as the change of the cluster center is less than a threshold or the maximum number of iterations is reached.
[0100] Finally, a series of new cluster centers will be generated, that is, the cluster labels corresponding to each speech frame in the speech sample will be obtained. The cluster labels reflect the similarity between each speech frame in the feature space. The cluster labels corresponding to each speech frame will be used in the subsequent self-supervised training stage to help the model learn the key features of speech.
[0101] In the embodiments of this specification, the characteristics of the phonemes themselves are taken into consideration, and the average characteristics of each phoneme are selected as the initial cluster center, which can ensure that the clustering algorithm starts from a reasonable position, which helps to converge faster and obtain high-quality clustering results, and can also better reflect the natural structure of the speech data. Moreover, averaging the speech features corresponding to the same phoneme can avoid randomness, making the clustering results more controllable, and at the same time, the clustering is more interpretable and can better distinguish different phoneme features. Furthermore, a clustering label with better discrimination is obtained, and the clustering label is used as the target of subsequent training, so that the model can learn the important features of the speech, which is helpful for model pre-training, and finally the model can achieve better performance in downstream speech tasks.
[0102] S210, using the clustering labels to train a preset network model to obtain a speech pre-training model.
[0103] The preset network model refers to a network model that can be used to process speech data, such as a convolutional neural network, a recurrent neural network, and a Transformer model. This specification does not limit the type of the preset network model. The speech pre-training model refers to a model obtained by training the preset network model using the automatically generated clustering labels as self-supervisory signals.
[0104] See also Figure 4 , is a flow chart of a self-supervised pre-training method provided in an embodiment of this specification. Figure 4 As shown, the method comprises the following steps:
[0105] S402, randomly masking the speech features of each speech frame in the speech sample to obtain corresponding mask features;
[0106] A portion of speech frames in the speech sample can be randomly selected for masking. For example, if the masking probability is set to 50%, the speech features of this portion of speech frames will be replaced with a special mask value, thereby training the model to predict the original content of the masked speech frame based on the contextual information of the unmasked speech frame and the masked speech frame.
[0107] It is understandable that the masked features include the speech features of the masked speech frame and the speech features of the unmasked speech frame. Subsequently, the speech features of all speech frames can be input into a preset network model, and the preset network model can learn how to extract useful information from the unmasked speech frame and predict the complete content of the masked speech frame.
[0108] S404, inputting each of the mask features into the preset network model to obtain a predicted label of each of the speech frames;
[0109] Optionally, the preset network model may include an encoder and a prediction head for predicting discrete labels corresponding to masked speech frames. For example, each mask feature may be input into the encoder to obtain a corresponding hidden state, and then each hidden state may be mapped to a label space through the prediction head to obtain a predicted label for each speech frame.
[0110] The encoder consists of multiple Transformer encoder layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network, and each layer gradually learns higher-level representations. The prediction head is a simple linear layer plus a softmax function, which is used to map the output of the last layer of the encoder to the label space to obtain the predicted label of the masked speech frame. In this process, the encoder and the prediction head can also be used to obtain the predicted labels of the unmasked speech frames, that is, the predicted labels of all speech frames.
[0111] S406, calculating the loss value between the predicted label and the cluster label, and iteratively training the model parameters of the preset network model according to the loss value, and obtaining the speech pre-training model when the iteration termination condition is met.
[0112] Exemplarily, the preset network model can be trained by minimizing the difference between the predicted label and the cluster label corresponding to each speech frame. For example, a cross entropy loss function is used to measure the difference between the predicted label and the cluster label corresponding to each speech frame. Specifically, the cross entropy loss between the predicted label and the cluster label corresponding to each speech frame is calculated, and the error is back-propagated according to the calculated loss value, and the model parameters of the preset network model are continuously iterated and updated until the iteration termination condition is met, such as the model converges or reaches the preset number of iterations, and the speech pre-training model can be obtained.
[0113] In the embodiments of this specification, the model can learn the key features of speech in the process of self-supervised learning, taking the speech features of the masked speech frame as input and the automatically generated clustering labels as training targets, helping the model learn to predict the clustering category to which the masked speech frame belongs, thereby learning a higher-level speech representation, and then the learned speech representation can be used for a variety of downstream tasks, such as speech recognition, speaker verification, etc. In addition, using forced alignment and clustering to generate self-supervisory signals not only utilizes the powerful potential of unlabeled audio data, but also simplifies the labeling workload required for model training.
[0114] See also Figure 5, provides a schematic diagram of the principle of another speech pre-training method for the embodiments of this specification, which can train a speech model by converting unlabeled audio data into meaningful speech representations.
[0115] like Figure 5 As shown, for an audio signal 501 used for speech pre-training, a speech feature 502 can be obtained by extracting features from the audio signal 501, and then clustering the speech feature 502. For example, the speech feature 502 is clustered using K-means to obtain a corresponding cluster label 503, which is the target that the model wants to learn.
[0116] Similarly, for the audio signal 501, the hidden feature 505 in the audio signal 501 can be extracted by the hidden feature encoder 504, and the predicted label 507 can be obtained by using the context network 506. When the hidden feature 505 is input into the context network 506, the hidden feature 505 needs to be masked, so that the training model predicts the masked part of the masked hidden feature 505 according to the context information of the unmasked hidden feature 505 and the masked hidden feature 505.
[0117] Finally, the cross entropy loss between the predicted label 507 and the clustering label 503 corresponding to each speech frame is calculated, and the error is back-propagated according to the calculated loss value, and the model parameters involved in the entire model are continuously iterated and updated until the model converges or reaches the preset number of iterations to obtain a speech pre-training model.
[0118] It should be noted that when the speech feature 502 is clustered, the corresponding relationship between the speech feature 502 and the phoneme data of the audio signal 501 in the time dimension is obtained, so that all the speech features 502 corresponding to the same phoneme are averaged to obtain the average feature of each phoneme. Further, the average feature of each phoneme can be used as the initial cluster center, and the speech feature 502 can be clustered to obtain the cluster label 503.
[0119] The speech feature 502 may be a conventional Mel-spectrogram feature or a feature extracted by a trained neural network model. Moreover, during the model training process, the intermediate layer features output by the context network 506 may be used as new speech features for cluster analysis to improve the clustering effect.
[0120] In this embodiment, by using speech phoneme information to assist clustering, the clustering results of speech features can be made more controllable and more interpretable, which is helpful for speech self-supervised training, and thus enables the obtained speech pre-training model to achieve better performance in downstream speech tasks.
[0121] See also Figure 6, is a schematic diagram of the structure of a speech pre-training device provided in an embodiment of this specification. Figure 6 As shown, the speech pre-training device 1 can be implemented as all or part of an electronic device through software, hardware or a combination of both. According to some embodiments, the speech pre-training device 1 includes a speech data acquisition module 11, a speech segment division module 12, a target feature determination module 13, a clustering label generation module 14 and a model pre-training module 15, specifically including:
[0122] A voice data acquisition module 11 is used to acquire a voice sample and phoneme data corresponding to the voice sample;
[0123] A speech segment division module 12, used for extracting speech features of each speech frame in the speech sample, and dividing the speech sample into at least one speech segment according to the speech features and the phoneme data, wherein one speech segment corresponds to one phoneme;
[0124] A target feature determination module 13, configured to determine a target feature of each phoneme according to the speech features of each speech frame in a speech segment corresponding to the same phoneme;
[0125] A cluster label generation module 14 is used to use the target feature of each of the phonemes as an initial cluster center, and cluster the speech features of each speech frame in the speech sample based on each of the initial cluster centers to obtain a corresponding cluster label;
[0126] The model pre-training module 15 is used to train the preset network model using the clustering labels to obtain a speech pre-training model.
[0127] Optionally, when the speech segment division module 12 divides the speech sample into at least one speech segment according to the speech feature and the phoneme data, it is specifically used to:
[0128] The speech features of each speech frame in the speech sample are aligned with the phoneme data to determine the speech segment corresponding to each phoneme in the time dimension.
[0129] Optionally, in the target feature determination module 13, when determining the target feature of each phoneme according to the speech features of each speech frame in the speech segment corresponding to the same phoneme, it is specifically used to:
[0130] The speech features of each speech frame in the speech segment corresponding to the same phoneme are averaged to obtain the target feature of each of the phonemes.
[0131] Optionally, when the clustering label generating module 14 clusters the speech features of each speech frame in the speech sample based on each of the initial cluster centers to obtain a corresponding cluster label, it is specifically used to:
[0132] Calculating the distance between each of the initial cluster centers and the speech features of each speech frame in the speech sample;
[0133] Determine the initial cluster center closest to each speech frame in the speech sample according to the distance, so as to assign each speech frame in the speech sample to the speech feature cluster where the initial cluster center is located;
[0134] The cluster center of each speech feature cluster is updated, and each speech frame in the speech sample is clustered based on the updated cluster centers until a clustering termination condition is met, thereby obtaining a cluster label corresponding to each speech frame in the speech sample.
[0135] Optionally, when updating the cluster center of each speech feature cluster, the cluster label generation module 14 is specifically used to:
[0136] Averaging the speech features of all speech frames in each speech feature cluster to obtain corresponding average features;
[0137] The average features are used as the updated cluster centers of the speech feature clusters.
[0138] Optionally, when the model pre-training module 15 trains the preset network model using the clustering labels to obtain the speech pre-training model, it is specifically used to:
[0139] Randomly masking the speech features of each speech frame in the speech sample to obtain corresponding masked features;
[0140] Inputting each of the mask features into the preset network model to obtain a predicted label for each of the speech frames;
[0141] The loss value between the predicted label and the cluster label is calculated, and the model parameters of the preset network model are iteratively trained according to the loss value. When the iteration termination condition is met, the speech pre-training model is obtained.
[0142] Optionally, the preset network model includes an encoder and a prediction head; when the model pre-training module 15 inputs each of the mask features into the preset network model to obtain a prediction label for each of the speech frames, it is specifically used to:
[0143] Inputting each of the mask features into the encoder to obtain a corresponding hidden state;
[0144] Each hidden state is mapped to a label space through the prediction head to obtain a predicted label for each speech frame.
[0145] Optionally, the speech feature includes a first Mel-spectrogram feature; when the speech segment division module 12 extracts the speech feature of each speech frame in the speech sample, it is specifically used to:
[0146] The speech samples are divided into frames, and the first Mel-spectrogram feature of each speech frame is extracted.
[0147] Optionally, the speech feature includes a second Mel-spectrogram feature; when the speech segment division module 12 extracts the speech feature of each speech frame in the speech sample, it is specifically used to:
[0148] The text sequence corresponding to the speech sample is input into the trained feature extraction model to obtain the second Mel-spectrogram feature of each speech frame in the speech sample.
[0149] Optionally, the feature extraction model includes at least a first encoder, a second encoder and a decoder; when the speech segment division module 12 inputs the text sequence corresponding to the speech sample into the trained feature extraction model to obtain the second Mel spectrum feature of each speech frame in the speech sample, it is specifically used to:
[0150] Using the first encoder to extract features from the text sequence to obtain a first latent variable feature;
[0151] Using the second encoder to extract features from the linear spectrum of the speech sample to obtain a second latent variable feature;
[0152] The decoder is used to decode the second latent variable feature to obtain a second Mel-spectrogram feature of each speech frame in the speech sample.
[0153] Optionally, when the voice data acquisition module 11 acquires the voice sample and the phoneme data corresponding to the voice sample, it is specifically used to:
[0154] Acquire the voice sample and the natural text corresponding to the voice sample;
[0155] The natural text is subjected to phoneme processing to obtain the phoneme data.
[0156] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.
[0157] The present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figures 2 to 5The method of the embodiment shown in the figure can be specifically executed by referring to Figures 2 to 5 The specific description of the illustrated embodiment will not be repeated here.
[0158] The present specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figures 2 to 5 The method of the embodiment shown in the figure can be specifically executed by referring to Figures 2 to 5 The specific description of the illustrated embodiment will not be repeated here.
[0159] The embodiments of this specification also provide Figure 7 The structural diagram of the electronic device shown in FIG. Figure 7 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned voice activity detection method.
[0160] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0161] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0162] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0163] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0164] For the convenience of description, the above device is described by dividing it into various units according to its functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0165] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0166] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0170] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0173] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0174] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0175] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0176] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.
Claims
1. A speech pre-training method, the method comprising: Acquire a voice sample and phoneme data corresponding to the voice sample; Extracting speech features of each speech frame in the speech sample, and dividing the speech sample into at least one speech segment according to the speech features and the phoneme data, wherein one speech segment corresponds to one phoneme; Determine the target feature of each phoneme according to the speech features of each speech frame in the speech segment corresponding to the same phoneme; The target feature of each of the phonemes is used as an initial cluster center, and based on each of the initial cluster centers, the speech features of each speech frame in the speech sample are clustered to obtain a corresponding cluster label; The preset network model is trained using the clustering labels to obtain a speech pre-training model.
2. The speech pre-training method according to claim 1, wherein the step of dividing the speech sample into at least one speech segment according to the speech feature and the phoneme data comprises: The speech features of each speech frame in the speech sample are aligned with the phoneme data to determine the speech segment corresponding to each phoneme in the time dimension.
3. The speech pre-training method according to claim 1, wherein the step of determining the target feature of each phoneme according to the speech features of each speech frame in the speech segment corresponding to the same phoneme comprises: The speech features of each speech frame in the speech segment corresponding to the same phoneme are averaged to obtain the target feature of each phoneme.
4. The speech pre-training method according to claim 1, wherein the speech features of each speech frame in the speech sample are clustered based on each of the initial cluster centers to obtain a corresponding cluster label, comprising: Calculating the distance between each of the initial cluster centers and the speech features of each speech frame in the speech sample; Determine the initial cluster center closest to each speech frame in the speech sample according to the distance, so as to assign each speech frame in the speech sample to the speech feature cluster where the initial cluster center is located; The cluster center of each speech feature cluster is updated, and each speech frame in the speech sample is clustered based on the updated cluster centers until a clustering termination condition is met, thereby obtaining a cluster label corresponding to each speech frame in the speech sample.
5. The speech pre-training method according to claim 4, wherein the updating of the cluster center of each speech feature cluster comprises: Averaging the speech features of all speech frames in each speech feature cluster to obtain corresponding average features; The average features are used as the updated cluster centers of the speech feature clusters.
6. The speech pre-training method according to claim 1, wherein the clustering labels are used to train a preset network model to obtain a speech pre-training model, comprising: Randomly masking the speech features of each speech frame in the speech sample to obtain corresponding masked features; Inputting each of the mask features into the preset network model to obtain a predicted label for each of the speech frames; The loss value between the predicted label and the cluster label is calculated, and the model parameters of the preset network model are iteratively trained according to the loss value. When the iteration termination condition is met, the speech pre-training model is obtained.
7. The speech pre-training method according to claim 6, wherein the preset network model comprises an encoder and a prediction head; The step of inputting each of the mask features into the preset network model to obtain a predicted label of each of the speech frames includes: Inputting each of the mask features into the encoder to obtain a corresponding hidden state; Each hidden state is mapped to a label space through the prediction head to obtain a predicted label for each speech frame.
8. The speech pre-training method according to claim 1, wherein the speech feature comprises a first Mel-spectrogram feature; The extracting speech features of each speech frame in the speech sample includes: The speech samples are divided into frames, and the first Mel-spectrogram feature of each speech frame is extracted.
9. The speech pre-training method according to claim 1, wherein the speech feature comprises a second Mel-spectrogram feature; The extracting speech features of each speech frame in the speech sample includes: The text sequence corresponding to the speech sample is input into the trained feature extraction model to obtain the second Mel-spectrogram feature of each speech frame in the speech sample.
10. The speech pre-training method according to claim 9, wherein the feature extraction model comprises at least a first encoder, a second encoder and a decoder; The step of inputting the text sequence corresponding to the speech sample into a trained feature extraction model to obtain a second Mel-spectrogram feature of each speech frame in the speech sample includes: Using the first encoder to extract features from the text sequence to obtain a first latent variable feature; Using the second encoder to extract features from the linear spectrum of the speech sample to obtain a second latent variable feature; The decoder is used to decode the second latent variable feature to obtain a second Mel-spectrogram feature of each speech frame in the speech sample.
11. The speech pre-training method according to claim 1, wherein the step of obtaining a speech sample and phoneme data corresponding to the speech sample comprises: Acquire the voice sample and the natural text corresponding to the voice sample; The natural text is phonemized to obtain the phoneme data.
12. A speech pre-training device, comprising: A voice data acquisition module, used to acquire a voice sample and phoneme data corresponding to the voice sample; A speech segment division module, used for extracting speech features of each speech frame in the speech sample, and dividing the speech sample into at least one speech segment according to the speech features and the phoneme data, wherein one speech segment corresponds to one phoneme; A target feature determination module, used to determine the target feature of each phoneme according to the speech features of each speech frame in the speech segment corresponding to the same phoneme; A clustering label generation module, used to take the target feature of each of the phonemes as an initial clustering center, and cluster the speech features of each speech frame in the speech sample based on each of the initial clustering centers to obtain a corresponding clustering label; The model pre-training module is used to train the preset network model using the clustering labels to obtain a speech pre-training model.
13. A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.
14. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 11.
15. A computer program product having at least one instruction stored thereon, wherein when the at least one instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.
