Training method and device of voice pre-training model, medium and equipment

Abstract

The embodiment of the invention discloses a training method and device of a voice pre-training model, a medium and equipment, and the method comprises the steps: firstly obtaining sample voice data, and carrying out the feature extraction of the sample voice data, and obtaining an initial voice acoustic feature; performing fuzzy clustering processing on the initial voice acoustic features to obtain voice feature clustering distribution; and performing self-supervised training on a pre-constructed voice pre-training model based on the voice feature clustering distribution to obtain a trained voice pre-training model. According to the technical scheme, through a soft clustering mode of fuzzy clustering, the voice pre-training model can pay more attention to the continuity of the voice, and the defect that the continuous voice is rigidly divided due to a common hard clustering mode is overcome; and the soft clustering result is used for self-supervised learning of the voice pre-training model, so that the generalization ability of the voice pre-training model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method, device, medium, and equipment for a speech pre-training model. Background Art

[0002] Speech pre-training refers to a training method that learns from a large amount of unlabeled speech data to improve the generalization ability of the model and reduce the dependence on labeled data. The audio features obtained through speech pre-training can be widely used in many types of downstream speech tasks. The speech pre-training model is a mainstream method in the field of speech pre-training.

[0003] At present, the relevant training schemes of speech pre-training models may destroy the model's ability to capture the similarity of acoustic features, resulting in poor continuity of the extracted speech features and weak generalization ability of the speech pre-training model. Summary of the invention

[0004] In a first aspect of the embodiments of the specification, a method for training a speech pre-training model is provided, which enables the speech pre-training model to pay more attention to the continuity of speech and effectively improves the generalization ability of the speech pre-training model. The method includes:

[0005] Acquiring sample speech data, and performing feature extraction on the sample speech data to obtain initial speech acoustic features;

[0006] Performing fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution;

[0007] Based on the speech feature cluster distribution, self-supervisory training is performed on the pre-constructed speech pre-training model to obtain a trained speech pre-training model.

[0008] Furthermore, in some implementations, the performing fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution includes:

[0009] Determine the initial cluster centers of a preset number of clusters, and update the initial cluster centers to obtain target cluster centers;

[0010] The initial speech acoustic features are clustered based on the target cluster center to obtain the speech feature cluster distribution corresponding to each initial speech acoustic feature belonging to the target cluster center.

[0011] Further, in some implementations, updating the initial cluster center to obtain the target cluster center includes:

[0012] Determining a membership matrix between each of the initial speech acoustic features and the initial cluster center;

[0013] Determining an intermediate cluster center based on the membership matrix and each of the initial speech acoustic features, and using the intermediate cluster center as a new initial cluster center;

[0014] The above steps are executed repeatedly until the update iteration condition is met, and the latest intermediate cluster center is used as the target cluster center.

[0015] Furthermore, in some implementations, determining a membership matrix between each of the initial speech acoustic features and the initial clustering center includes:

[0016] Determining a first similarity distance between each of the initial speech acoustic features and a current cluster center among the initial cluster centers;

[0017] Determining a second similarity distance between each of the initial speech acoustic features and all cluster centers in the initial cluster centers;

[0018] Obtaining a preset fuzzy factor, and determining membership data of each of the initial speech acoustic features belonging to the current cluster center according to the fuzzy factor, the first similarity distance, and the second similarity distance;

[0019] The membership matrix of each of the initial speech acoustic features and the initial cluster center is determined through the membership data.

[0020] Furthermore, in some implementations, clustering the initial speech acoustic features based on the target cluster center to obtain a speech feature cluster distribution corresponding to each initial speech acoustic feature belonging to the target cluster center includes:

[0021] Determining a membership vector of each of the initial speech acoustic features belonging to the target cluster center, wherein the membership vector includes a degree of belonging of each of the initial speech acoustic features to each cluster category in the target cluster center;

[0022] The membership vector is used as the speech feature cluster distribution corresponding to each of the initial speech acoustic features belonging to the target cluster center.

[0023] Further, in some implementations, the self-supervised training of the pre-constructed speech pre-training model based on the speech feature cluster distribution to obtain a trained speech pre-training model includes:

[0024] Inputting the initial speech acoustic features into a pre-built speech pre-training model to obtain a speech feature prediction distribution;

[0025] According to the speech feature prediction distribution and the speech feature cluster distribution, the model parameters of the pre-constructed speech pre-training model are updated to obtain a trained speech pre-training model.

[0026] Further, in some implementations, updating the model parameters of the pre-constructed speech pre-training model according to the speech feature prediction distribution and the speech feature cluster distribution to obtain a trained speech pre-training model includes:

[0027] Determining the loss between the speech feature prediction distribution and the speech feature cluster distribution by using the target loss function corresponding to the pre-built speech pre-training model;

[0028] Updating model parameters of the pre-built speech pre-training model according to the loss;

[0029] The above steps are executed repeatedly until the loss converges or the number of model update iterations is reached, thereby completing the self-supervised training of the pre-constructed speech pre-training model and obtaining a trained speech pre-training model.

[0030] Furthermore, in some implementations, the extracting features from the sample speech data to obtain initial speech acoustic features includes:

[0031] Performing windowing processing on the sample speech data to obtain a preprocessed speech frame;

[0032] Performing short-time Fourier transform on the preprocessed speech frame to obtain a speech frame spectrum, and performing smoothing on the speech frame spectrum to obtain a speech spectrum envelope;

[0033] Filtering the speech spectrum envelope based on a preset Mel filter, taking the logarithm of the filtering output result and performing a difference operation to obtain the Mel frequency cepstrum coefficient corresponding to the sample speech data;

[0034] The Mel-frequency cepstral coefficients are used as initial speech acoustic features corresponding to the sample speech data.

[0035] Furthermore, in some implementations, the extracting features from the sample speech data to obtain initial speech acoustic features includes:

[0036] The sample speech data is input into a trained acoustic feature extraction network, and the initial speech acoustic features are obtained as output.

[0037] Furthermore, in some embodiments, the method further comprises:

[0038] Performing denoising on the collected original speech data to obtain denoised original speech data;

[0039] Performing silence elimination processing on the denoised original speech data to obtain silence eliminated original speech data;

[0040] Performing volume normalization processing on the original voice data after the silence is eliminated to obtain original voice data with a standardized volume;

[0041] The original voice data of the standardized volume is subjected to frame processing to obtain the sample voice data.

[0042] Furthermore, in some embodiments, the method further comprises:

[0043] Acquire user voice data, and input the user voice data into the trained voice pre-training model to obtain a predicted voice feature representation;

[0044] Inputting the predicted speech feature representation into a downstream speech task processing network to obtain a speech processing result;

[0045] Among them, the downstream speech task processing network includes any one or a combination of a speech recognition conversion processing network, a language recognition processing network, a speech intention recognition processing network, a speaker identity recognition processing network and a speech enhancement processing network.

[0046] In a second aspect of the embodiments of the specification, a training device for a speech pre-training model is also proposed, comprising:

[0047] An acoustic feature extraction module, used to obtain sample speech data and perform feature extraction on the sample speech data to obtain initial speech acoustic features;

[0048] A feature soft clustering module, used for performing fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution;

[0049] The self-supervised training module is used to perform self-supervised training on the pre-constructed speech pre-training model based on the speech feature cluster distribution to obtain a trained speech pre-training model.

[0050] According to a third aspect of the embodiments of this specification, a computer program product is provided. The computer program product stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the method steps in the first aspect.

[0051] According to a fourth aspect of the embodiments of this specification, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the method in the first aspect.

[0052] According to a fifth aspect of the embodiments of the present specification, there is also provided an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method in the first aspect.

[0053] In the embodiments of this specification, sample speech data is obtained, and feature extraction is performed on the sample speech data to obtain initial speech acoustic features, and fuzzy clustering is performed on the initial speech acoustic features to obtain speech feature clustering distribution; finally, self-supervised training is performed on the pre-constructed speech pre-training model based on the speech feature clustering distribution to obtain a trained speech pre-training model. Through fuzzy clustering processing, the speech pre-training model can pay more attention to the continuity of speech, overcome the disadvantage of the commonly used hard clustering method that causes continuous speech to be rigidly divided, and the fuzzy clustering processing results are used for self-supervised learning of the speech pre-training model, effectively improving the generalization ability of the speech pre-training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the system architecture of an exemplary application environment of a training method and apparatus for a speech pre-training model to which the embodiments of this specification can be applied is shown.

[0055] Figure 2 A flowchart of a method for training a speech pre-training model provided in an embodiment of this specification.

[0056] Figure 3 A schematic diagram of a process of preprocessing original voice data to obtain sample voice data provided in an embodiment of this specification.

[0057] Figure 4 A schematic diagram of a process for extracting initial speech acoustic features corresponding to sample speech data provided in an embodiment of this specification.

[0058] Figure 5 A schematic diagram of a process for iteratively updating an initial cluster center to obtain a target cluster center is provided in an embodiment of this specification.

[0059] Figure 6 A schematic diagram of a process for performing self-supervised training on a pre-built speech pre-training model provided in an embodiment of this specification.

[0060] Figure 7 Another flowchart of self-supervised training of a pre-built speech pre-training model is provided in an embodiment of this specification.

[0061] Figure 8 A schematic diagram of the structure of a training device for a speech pre-training model provided in an embodiment of this specification.

[0062] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0064] Figure 1 A schematic diagram of the system architecture of an exemplary application environment of a training method and apparatus for a speech pre-training model to which the embodiments of this specification can be applied is shown.

[0065] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with artificial intelligence (AI) computing capabilities, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0066] The training method of the speech pre-training model provided in the embodiments of this specification is generally performed by the terminal devices 101, 102, and 103, and accordingly, the training device of the speech pre-training model is generally set in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the training method of the speech pre-training model provided in the embodiments of this specification can also be performed by the server 105, and accordingly, the training device of the speech pre-training model can also be set in the server 105, which is not particularly limited in this exemplary embodiment.

[0067] See also Figure 2, which is a flowchart of a method for training a speech pre-training model in an embodiment of this specification. In the embodiment of this specification, the method for training a speech pre-training model can be applied to a terminal device or a server, and this exemplary embodiment does not make any special restrictions on this. Figure 2 The process shown is described in detail, and the training method of the speech pre-training model in the embodiment of this specification may specifically include the following steps:

[0068] Step S210, acquiring sample speech data, and performing feature extraction on the sample speech data to obtain initial speech acoustic features;

[0069] Step S220, performing fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution;

[0070] Step S230, performing self-supervisory training on the pre-constructed speech pre-training model based on the speech feature cluster distribution to obtain a trained speech pre-training model.

[0071] According to the training method of the speech pre-training model in the embodiment of this specification, fuzzy clustering processing can make the speech pre-training model pay more attention to the continuity of speech, overcoming the shortcoming of the commonly used hard clustering method that causes continuous speech to be rigidly divided, and the fuzzy clustering processing results are used for self-supervised learning of the speech pre-training model, effectively improving the generalization ability of the speech pre-training model.

[0072] Next, the training method of the speech pre-training model in the embodiment of this specification is described in detail.

[0073] In step S210, sample speech data is acquired, and features are extracted from the sample speech data to obtain initial speech acoustic features.

[0074] In an example embodiment of the present specification, sample speech data refers to pre-collected unlabeled speech waveform data. For example, the sample speech data may be speech waveform data in an open source speech database, or speech waveform data specially collected by a collection device. This example embodiment does not specifically limit the source of the sample speech data.

[0075] The initial speech acoustic features refer to speech feature data that can be recognized by a computer and are obtained by extracting acoustic features from sample speech data. For example, the initial speech acoustic features can be obtained by extracting Mel-Frequency Cepstral Coefficients (MFCC) of the sample speech data, or can be extracted from the sample speech data by linear predictive coding (LPC). This example embodiment does not specifically limit the method for extracting the initial speech acoustic features.

[0076] By extracting features from sample speech data to obtain initial speech acoustic features, and then using the initial speech acoustic features instead of sample speech data to train the speech pre-training model, the difficulty of speech data processing can be effectively reduced, and the speech pre-training model's ability to understand sample speech data can also be effectively improved, thereby reducing the difficulty of learning.

[0077] Optionally, the sample speech data can be preprocessed before feature extraction to make the extracted initial speech acoustic features more accurate. For example, the collected original speech data can be subjected to preprocessing steps such as denoising, silence elimination, volume normalization, and frame segmentation to effectively improve the clarity and accuracy of the obtained sample speech data, reduce the impact of interference factors, and thus improve the accuracy of the initial speech acoustic features.

[0078] In step S220, fuzzy clustering processing is performed on the initial speech acoustic features to obtain speech feature cluster distribution.

[0079] In an example embodiment of the present specification, fuzzy clustering processing refers to a process of avoiding forcibly classifying each frame of speech into a certain clustering category in the process of clustering continuous initial speech acoustic features. Fuzzy clustering processing can obtain the probability that each frame of speech belongs to multiple clustering categories. Therefore, the speech feature clustering distribution obtained based on the fuzzy clustering processing can effectively improve the ability of the speech pre-training model to capture speech similarity and improve the generalization ability of the speech pre-training model.

[0080] The speech feature cluster distribution refers to the distribution feature data used to characterize that each frame of speech in the initial speech acoustic features belongs to multiple cluster categories. For example, the speech feature cluster distribution can be a membership matrix that each frame of speech in the initial speech acoustic features belongs to multiple cluster categories, or it can be a probability distribution that each frame of speech in the initial speech acoustic features belongs to multiple cluster categories. This example embodiment does not specifically limit the data representation type of the speech feature cluster distribution.

[0081] For example, the fuzzy clustering process may be to first determine the initial cluster centers of a preset number of clusters, and then update the initial cluster centers to obtain the target cluster centers, thereby clustering the initial speech acoustic features based on the target cluster centers, and determining that the initial speech acoustic features belong to the speech feature cluster distribution corresponding to at least one cluster center in the target cluster centers. Of course, other types of fuzzy clustering processes may also be used, as long as they can effectively avoid forcibly assigning each frame of speech to a certain cluster category, and this exemplary embodiment does not specifically limit this.

[0082] In step S230, self-supervised training is performed on the pre-constructed speech pre-training model based on the speech feature cluster distribution to obtain a trained speech pre-training model.

[0083] In an example embodiment of the present specification, a speech pre-training model refers to a neural network model that is trained on a large amount of unlabeled speech data to learn the general features of speech. For example, the speech pre-training model can be a HuBERT (Hidden Units BERT) pre-training model, or a Wav2Vec pre-training model. Of course, it can also be other pre-training models that can realize general feature extraction of speech data. For example, the speech pre-training model can also be a VQ-VAE (Vector Quantized-Variational AutoEncoder) pre-training model. This example embodiment does not specifically limit the network type of the speech pre-training model.

[0084] Self-supervised training refers to a machine learning method that uses auxiliary tasks to mine the representational characteristics of sample speech data as supervisory information for unlabeled sample speech data to improve the feature extraction capability of the speech pre-training model. The pre-built speech pre-training model can be self-supervised based on the speech feature clustering distribution obtained by fuzzy clustering to obtain a trained speech pre-training model.

[0085] Through fuzzy clustering processing, the speech pre-training model can pay more attention to the continuity of speech, overcoming the shortcoming of the commonly used hard clustering method that causes continuous speech to be rigidly divided. The fuzzy clustering processing results are used for self-supervised learning of the speech pre-training model, effectively improving the generalization ability of the speech pre-training model.

[0086] Next, steps S210 to S230 are described in detail.

[0087] In an exemplary embodiment of the present specification, before extracting features from the sample speech data, Figure 3 The steps in the preprocessing step are used to preprocess the collected original voice data to obtain the preprocessed sample voice data. Figure 3As shown, it may specifically include:

[0088] Step S310, performing denoising processing on the collected original voice data to obtain denoised original voice data;

[0089] Step S320, performing silence elimination processing on the denoised original speech data to obtain silence eliminated original speech data;

[0090] Step S330, performing volume normalization processing on the original voice data after silence elimination to obtain original voice data with standardized volume;

[0091] Step S340, performing frame processing on the original voice data of the standardized volume to obtain the sample voice data.

[0092] Among them, denoising processing refers to the process used to eliminate noise in original speech data. For example, the noise type and characteristics in the original speech data can be analyzed, and a suitable denoising algorithm can be selected according to the noise type and characteristics, such as noise suppression, filter, wavelet denoising, etc. to achieve denoising of the original speech data.

[0093] Silence elimination processing refers to the process used to eliminate the silent part in the original voice data. For example, the energy value of each voice frame in the original voice data can be calculated, and an energy threshold can be set. The voice frames below the energy threshold are marked as silent frames, and all silent frames in the original voice data are deleted to obtain the original voice data after silence elimination.

[0094] Volume normalization refers to the process of adjusting the voice frames with excessive volume or too low volume in the original voice data to within the standard range. For example, the dynamic range of the voice signal corresponding to the original voice data, that is, the difference between the maximum signal and the minimum signal, can be calculated. By amplifying or reducing the signal amplitude of each voice frame, the dynamic range of the entire sound signal meets the predetermined standard, thereby achieving volume normalization of the original voice data after silence elimination.

[0095] Framing processing refers to the process of dividing the original speech data into speech frames. For example, the length of the speech frame can be set according to the processing requirements, usually between 10-30 milliseconds. It can be understood that the shift length between adjacent speech frames can also be set, usually between 10-20 milliseconds, thereby creating overlapping speech frames to preserve the continuity of the speech signal.

[0096] The collected original voice data is processed through pre-processing methods such as denoising, silence elimination, volume normalization and frame processing to obtain sample voice data, improve the clarity and accuracy of the sample voice data, and reduce the influence of interference signals in the sample voice data, thereby improving the convergence efficiency of the voice pre-training model and improving the model quality of the voice pre-training model to a certain extent.

[0097] It can be understood that the preprocessing such as denoising, silence elimination, volume normalization and framing in this embodiment can be performed in any order or in a specific order to further improve the clarity and accuracy of the preprocessed sample speech data.

[0098] For example, in this embodiment, the collected original voice data is first subjected to denoising processing to obtain the denoised original voice data, which can effectively avoid the interference caused by the noise existing in the original voice data when performing the subsequent preprocessing steps. For example, when the silence elimination processing step is first implemented, the noise signal existing in the silence frame may be mistaken for a valid frame and silence elimination is not performed; then the denoised original voice data is subjected to silence elimination processing to obtain the original voice data after silence elimination, and the silence elimination processing step is set before the volume normalization processing, which can effectively avoid the operation of mistakenly amplifying the volume of the silence frame by the volume normalization processing step due to the small volume; then the original voice data after silence elimination can be subjected to volume normalization processing to obtain the original voice data of the standardized volume, and finally the original voice data of the standardized volume can be subjected to frame processing to obtain the sample voice data. Through the preprocessing process of denoising processing, silence elimination processing, volume normalization processing and frame processing in sequence, compared with the preprocessing process in any order, the interference factors in the sample voice data can be further reduced, and the accuracy, clarity and voice content validity of the preprocessed sample voice data can be effectively guaranteed.

[0099] In an exemplary embodiment of the present specification, Figure 4 The steps in the above code are used to extract the features of the sample speech data and obtain the initial speech acoustic features. Figure 4 As shown, it may specifically include:

[0100] Step S410, performing windowing processing on the sample speech data to obtain a preprocessed speech frame;

[0101] Step S420, performing short-time Fourier transform on the preprocessed speech frame to obtain a speech frame spectrum, and performing smoothing processing on the speech frame spectrum to obtain a speech spectrum envelope;

[0102] Step S430, filtering the speech spectrum envelope based on a preset Mel filter, taking the logarithm of the filtering output result and performing a difference operation to obtain the Mel frequency cepstrum coefficient corresponding to the sample speech data;

[0103] Step S440: Using the Mel-frequency cepstral coefficient as the initial speech acoustic feature corresponding to the sample speech data.

[0104] Among them, windowing processing refers to the process of adding a time domain window function to each speech frame in the sample speech data to reduce the influence of overlapping signals between frames. It is also a necessary measure for subsequent short-time Fourier transform. For example, a Hamming window can be applied to each speech frame in the sample speech data to implement windowing processing of the sample speech data to obtain a preprocessed speech frame.

[0105] Short-Time Fourier Transform (STFT) refers to a transformation method used to determine the frequency and phase of a local sinusoidal wave of a time-varying signal. The main idea is to select a time-frequency localized window function, such as a Hamming window, and assume that the analysis window function is stable (pseudo-stationary) within a short time interval. Move the window function so that the function value is a stationary signal within different finite time widths, thereby calculating the power spectrum at different times. By performing a short-time Fourier transform on the preprocessed speech frame, the speech frame spectrum corresponding to the sample speech data can be obtained.

[0106] After obtaining the speech frame spectrum, the speech frame spectrum can be smoothed to obtain the speech spectrum envelope. For example, the speech frame spectrum can be smoothed to obtain the speech spectrum envelope by using filters such as a Kalman filter, or by using harmonic integral decomposition. This example embodiment does not specifically limit the method for smoothing the speech frame spectrum to obtain the speech spectrum envelope.

[0107] A Mel filter bank can be designed and generated in advance according to the Mel frequency scale. The Mel filter bank can cover the entire spectrum range from low frequency to high frequency. Then, the speech spectrum envelope can be filtered through each Mel filter in the Mel filter bank to obtain a series of filtering output results. The logarithm of the filtering output results is taken and a differential operation is performed, so that the Mel frequency cepstral coefficients corresponding to the sample speech data can be obtained, and the Mel frequency cepstral coefficients are used as the initial speech acoustic features corresponding to the sample speech data.

[0108] The above processing steps are used to determine the Mel-frequency cepstral coefficients of the sample speech data, and then use the Mel-frequency cepstral coefficients as the initial speech acoustic features of the sample speech data. This can make the extracted initial speech acoustic features more consistent with the human auditory system, and make the initial speech acoustic features have better feature representation for the speech signal of the sample speech data, thereby improving the accuracy and comprehensibility of the initial speech acoustic features.

[0109] In an optional embodiment, the sample speech data can also be input into a trained acoustic feature extraction network to extract features of the sample speech data, thereby outputting the initial speech acoustic features. For example, the acoustic feature extraction network can be a feature extraction network based on a self-attention mechanism, such as a Transform network. Of course, the acoustic feature extraction network can also be a recurrent neural network (RNN). This example embodiment does not specifically limit the network type of the acoustic feature extraction network used to extract the initial speech acoustic features. At the same time, the model training, model evaluation and optimization of the acoustic feature extraction network are conventional technical means in this field and will not be repeated here.

[0110] By extracting the acoustic features of sample speech data through a trained feature extraction network, the extraction efficiency of the initial speech acoustic features can be effectively improved while ensuring the accuracy and effectiveness of the initial speech acoustic features, thereby improving the training efficiency of the speech pre-training model.

[0111] In an exemplary embodiment of the present specification, the following steps may be used to implement fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution, which may specifically include:

[0112] The initial cluster centers of a preset number of clusters can be determined, and the initial cluster centers can be updated to obtain target cluster centers, and the initial speech acoustic features can be clustered based on the target cluster centers to obtain the speech feature cluster distribution corresponding to the target cluster centers for each initial speech acoustic feature.

[0113] Among them, the preset number of clusters refers to the number of cluster centers pre-set for clustering each speech frame in the initial speech acoustic features. For example, the preset number of clusters can be the number of cluster centers specified based on the complexity of the speech data, or it can be the number of cluster centers randomly generated by a random number generator. The specific setting can be customized according to actual conditions. This example embodiment does not specifically limit the setting of the preset number of clusters.

[0114] The initial cluster center refers to the cluster center determined based on the initial speech acoustic features. For example, a preset number of clustering data points can be randomly selected from the initial speech acoustic features as cluster centers, or a preset number of clustering centers can be determined as initial cluster centers through the K-Means clustering algorithm. This embodiment does not specifically limit the method for determining the initial cluster center.

[0115] The target cluster center refers to the final stable cluster center obtained after iteratively updating the initial cluster center, that is, the cluster center set that is ultimately used to determine the speech feature cluster distribution of the initial speech acoustic features.

[0116] The initial cluster centers of a preset number of clusters can be determined, and the initial cluster centers can be iteratively updated to obtain target cluster centers, and the initial speech acoustic features can be clustered based on the target cluster centers to obtain the speech feature cluster distribution corresponding to the target cluster centers for each initial speech acoustic feature.

[0117] Optionally, you can Figure 5 The steps in are used to update the initial cluster center and obtain the target cluster center. Figure 5 As shown, it may specifically include:

[0118] Step S510, determining a membership matrix between each of the initial speech acoustic features and the initial cluster center;

[0119] Step S520, determining an intermediate cluster center based on the membership matrix and each of the initial speech acoustic features, and using the intermediate cluster center as a new initial cluster center;

[0120] Step S530, looping through steps S510 to S520 until an update iteration condition is met, and taking the latest intermediate cluster center as the target cluster center.

[0121] Among them, the membership matrix refers to the matrix composed of the membership degrees of the initial speech acoustic features of each frame to each initial cluster center. The membership degree can be the degree to which the initial speech acoustic features belong to the initial cluster center, or it can be simply considered as the similarity between the initial speech acoustic features and the initial cluster center.

[0122] The intermediate cluster center refers to the combination of cluster centers re-determined based on the membership matrix and each initial speech acoustic feature, and is an intermediate variable used to update the initial cluster center. The intermediate cluster center is determined by continuously determining the new membership matrix and each initial speech acoustic feature, and the intermediate cluster center is used as the new initial cluster center to achieve an iterative cycle. When the update iteration condition is met, the latest intermediate cluster center is used as the target cluster center, thereby iteratively updating the initial cluster center to obtain the target cluster center. For example, the update iteration condition can be that the number of iterative updates reaches the upper limit value, or when the number of changes in the updated intermediate cluster center reaches the change threshold, if the intermediate cluster center has not changed in multiple times, then it can be considered that the update iteration condition is met.

[0123] Optionally, the following steps may be used to determine the membership matrix of each initial speech acoustic feature and the initial cluster center, which may specifically include:

[0124] The first similarity distance between each initial speech acoustic feature and the current cluster center in the initial cluster centers can be determined, and the second similarity distance between each initial speech acoustic feature and all cluster centers in the initial cluster centers can be determined; then a preset fuzzy factor is obtained, and the membership data of each initial speech acoustic feature belonging to the current cluster center is determined based on the fuzzy factor, the first similarity distance and the second similarity distance; finally, the membership matrix of each initial speech acoustic feature and the initial cluster center can be determined through the membership data.

[0125] The current cluster center refers to any cluster center among the initial cluster centers, that is, the current cluster center may be the cluster center corresponding to the calculation of the membership degree between each initial speech acoustic feature and a certain cluster center.

[0126] The first similarity distance may be the similarity distance between each initial speech acoustic feature and the current cluster center. For example, the first similarity distance may be the Euclidean distance or the Minn distance, but the present embodiment is not limited thereto.

[0127] The second similarity distance may be the similarity distance between each initial speech acoustic feature and all cluster centers in the initial cluster centers. For example, the similarity distance between each initial speech acoustic feature and each cluster center in the initial cluster centers may be calculated separately, and the sum of all similarity distances may be taken as the second similarity distance.

[0128] The fuzzy factor refers to a preset parameter used to determine the fuzziness of the fuzzy clustering result. The fuzzy factor is generally greater than 1. For example, the fuzzy factor can be 1.5 or 1.8. It can be customized according to actual usage, and this example embodiment is not limited to this.

[0129] For example, the membership matrix can be calculated by using equation (1):

[0130]

[0131] Among them, u ij It can represent the membership matrix of the i-th initial speech acoustic feature belonging to the j-th current cluster center, c can represent the preset number of clusters, x i It can represent the acoustic features of the i-th initial speech, v j Can represent the jth current cluster center, v k It can represent any cluster center in the initial cluster centers, and m can represent a fuzzy factor. It can be understood that this is only a schematic example, and the embodiments of this specification are not limited thereto.

[0132] Optionally, the intermediate cluster center can be determined based on the membership matrix and each initial speech acoustic feature through equation (2):

[0133]

[0134] Among them, v j can represent the updated j-th intermediate cluster center, n can represent the total number of data points of the initial speech acoustic features, and x i It can represent the acoustic features of the i-th initial speech, u ij It can represent the membership matrix, and m can represent the fuzzy factor. It can be understood that this is only a schematic example, and the embodiments of this specification are not limited thereto.

[0135] Optionally, clustering of the initial speech acoustic features based on the target cluster center can be achieved through the following steps to obtain the speech feature cluster distribution corresponding to the target cluster center for each initial speech acoustic feature, which may specifically include:

[0136] The membership vector of each initial speech acoustic feature belonging to the target cluster center can be determined, and the membership vector is used as the speech feature cluster distribution corresponding to each initial speech acoustic feature belonging to the target cluster center. Among them, the membership vector can include the degree of belonging of each initial speech acoustic feature to each cluster category in the target cluster center; by using the membership vector as the speech feature cluster distribution corresponding to each initial speech acoustic feature belonging to the target cluster center, the self-supervised training of the speech pre-training model can be effectively realized, and the membership vector carries more understanding information, is more robust to interference noise, can effectively improve the generalization ability of the speech pre-training model, improve the self-supervised training effect of the speech pre-training model, and improve the model output quality of the speech pre-training model.

[0137] In an exemplary embodiment of the present specification, Figure 6 The steps in the above code implement self-supervised training of the pre-built speech pre-training model based on speech feature clustering distribution to obtain a trained speech pre-training model. Figure 6 As shown, it may specifically include:

[0138] Step S610, inputting the initial speech acoustic features into a pre-built speech pre-training model to obtain a speech feature prediction distribution;

[0139] Step S620, updating the model parameters of the pre-constructed speech pre-training model according to the speech feature prediction distribution and the speech feature cluster distribution, to obtain a trained speech pre-training model.

[0140] The speech feature prediction distribution refers to the probability distribution obtained by predicting the initial speech acoustic features through the understanding of the speech pre-training model. The speech feature clustering distribution obtained by fuzzy clustering processing can be compared with the speech feature prediction distribution output by the pre-built speech pre-training model to determine the loss, thereby updating the model parameters of the pre-built speech pre-training model and obtaining a trained speech pre-training model.

[0141] Optionally, you can Figure 7 The steps in the above are used to update the model parameters of the pre-built speech pre-training model according to the speech feature prediction distribution and speech feature cluster distribution, and obtain the trained speech pre-training model. Figure 7 As shown, it may specifically include:

[0142] Step S710, determining the loss between the speech feature prediction distribution and the speech feature cluster distribution by using the target loss function corresponding to the pre-built speech pre-training model;

[0143] Step S720, updating the model parameters of the pre-built speech pre-training model according to the loss;

[0144] Step S730, looping through steps S710 to S720 until the loss converges or the number of model update iterations is reached, completing the self-supervised training of the pre-constructed speech pre-training model, and obtaining a trained speech pre-training model.

[0145] The target loss function refers to a function of a learning criterion for guiding the speech pre-training model to perform self-supervised training, and the speech pre-training model can be solved and evaluated by minimizing the loss function. For example, the target loss function can be a cross entropy loss function or a KL divergence. This example embodiment does not specifically limit the type of the target loss function.

[0146] For example, the objective loss function can be expressed by the following equation (3):

[0147]

[0148] Among them, L can represent the loss value between the speech feature prediction distribution and the speech feature cluster distribution, y ic It can represent the cluster distribution of speech features obtained by fuzzy clustering processing, that is, the cluster probability that sample i belongs to category c; p ic It can represent the predicted distribution of speech features output by the pre-built speech pre-training model, that is, the predicted probability that sample i belongs to category c; M can represent the number of clustering categories, that is, the number of target clustering centers, and N can represent the number of sample speech data.

[0149] In an example embodiment of the present specification, user voice data can be obtained and input into a trained voice pre-training model to obtain a predicted voice feature representation; the predicted voice feature representation is input into a downstream voice task processing network to obtain a voice processing result; wherein the downstream voice task processing network may include any one or a combination of a voice recognition conversion processing network, a language recognition processing network, a voice intent recognition processing network, a speaker identity recognition processing network, and a voice enhancement processing network.

[0150] By extracting the predicted speech feature representation corresponding to the user's speech data through the trained speech pre-training model, the predicted speech feature representation can be used in any type of downstream speech tasks, including but not limited to speech recognition, language recognition, speech intent recognition, speaker identification, and speech enhancement, effectively improving the processing efficiency of various types of speech tasks and reducing the difficulty of extracting the predicted speech feature representation.

[0151] In summary, the training method of the speech pre-training model provided in the embodiments of this specification obtains sample speech data, extracts features from the sample speech data, obtains initial speech acoustic features, and performs fuzzy clustering processing on the initial speech acoustic features to obtain speech feature clustering distribution; finally, based on the speech feature clustering distribution, the pre-constructed speech pre-training model is self-supervised trained to obtain a trained speech pre-training model. The fuzzy clustering processing can make the speech pre-training model pay more attention to the continuity of speech, overcome the shortcomings of the commonly used hard clustering method that causes the continuous speech to be rigidly divided, and use the fuzzy clustering processing results for the self-supervised learning of the speech pre-training model, effectively improving the generalization ability of the speech pre-training model.

[0152] See also Figure 8 , is a structural diagram of a training device for a speech pre-training model provided in an embodiment of this specification. Figure 8As shown, the training device 800 of the speech pre-training model can be implemented as all or part of the electronic device through software, hardware or a combination of both. According to some embodiments, the training device 800 of the speech pre-training model includes an acoustic feature extraction module 810, a feature soft clustering module 820 and a self-supervised training module 830, specifically including:

[0153] The acoustic feature extraction module 810 is used to obtain sample speech data and perform feature extraction on the sample speech data to obtain initial speech acoustic features;

[0154] A feature soft clustering module 820, configured to perform fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution;

[0155] The self-supervised training module 830 is used to perform self-supervised training on the pre-constructed speech pre-training model based on the speech feature cluster distribution to obtain a trained speech pre-training model.

[0156] In an exemplary embodiment of the present specification, based on the above solution, the feature soft clustering module 820 is configured as follows:

[0157] Determine the initial cluster centers of a preset number of clusters, and update the initial cluster centers to obtain target cluster centers;

[0158] The initial speech acoustic features are clustered based on the target cluster center to obtain the speech feature cluster distribution corresponding to each initial speech acoustic feature belonging to the target cluster center.

[0159] In an exemplary embodiment of the present specification, based on the above solution, the feature soft clustering module 820 is configured as follows:

[0160] Determining a membership matrix between each of the initial speech acoustic features and the initial cluster center;

[0161] Determining an intermediate cluster center based on the membership matrix and each of the initial speech acoustic features, and using the intermediate cluster center as a new initial cluster center;

[0162] The above steps are executed repeatedly until the update iteration condition is met, and the latest intermediate cluster center is used as the target cluster center.

[0163] In an exemplary embodiment of the present specification, based on the above solution, the feature soft clustering module 820 is configured as follows:

[0164] Determining a first similarity distance between each of the initial speech acoustic features and a current cluster center among the initial cluster centers;

[0165] Determining a second similarity distance between each of the initial speech acoustic features and all cluster centers in the initial cluster centers;

[0166] Obtaining a preset fuzzy factor, and determining membership data of each of the initial speech acoustic features belonging to the current cluster center according to the fuzzy factor, the first similarity distance, and the second similarity distance;

[0167] The membership matrix of each of the initial speech acoustic features and the initial cluster center is determined through the membership data.

[0168] In an exemplary embodiment of the present specification, based on the above solution, the feature soft clustering module 820 is configured as follows:

[0169] Determining a membership vector of each of the initial speech acoustic features belonging to the target cluster center, wherein the membership vector includes a degree of belonging of each of the initial speech acoustic features to each cluster category in the target cluster center;

[0170] The membership vector is used as the speech feature cluster distribution corresponding to each of the initial speech acoustic features belonging to the target cluster center.

[0171] In an exemplary embodiment of the present specification, based on the above scheme, the self-supervised training module 830 is configured as follows:

[0172] Inputting the initial speech acoustic features into a pre-built speech pre-training model to obtain a speech feature prediction distribution;

[0173] According to the speech feature prediction distribution and the speech feature cluster distribution, the model parameters of the pre-constructed speech pre-training model are updated to obtain a trained speech pre-training model.

[0174] In an exemplary embodiment of the present specification, based on the above scheme, the self-supervised training module 830 is configured as follows:

[0175] Determining the loss between the speech feature prediction distribution and the speech feature cluster distribution by using the target loss function corresponding to the pre-built speech pre-training model;

[0176] Updating model parameters of the pre-built speech pre-training model according to the loss;

[0177] The above steps are executed repeatedly until the loss converges or the number of model update iterations is reached, thereby completing the self-supervised training of the pre-constructed speech pre-training model and obtaining a trained speech pre-training model.

[0178] In an exemplary embodiment of the present specification, based on the above solution, the acoustic feature extraction module 810 is configured as follows:

[0179] Performing windowing processing on the sample speech data to obtain a preprocessed speech frame;

[0180] Performing short-time Fourier transform on the preprocessed speech frame to obtain a speech frame spectrum, and performing smoothing on the speech frame spectrum to obtain a speech spectrum envelope;

[0181] Filtering the speech spectrum envelope based on a preset Mel filter, taking the logarithm of the filtering output result and performing a difference operation to obtain the Mel frequency cepstrum coefficient corresponding to the sample speech data;

[0182] The Mel-frequency cepstral coefficients are used as initial speech acoustic features corresponding to the sample speech data.

[0183] In an exemplary embodiment of the present specification, based on the above solution, the acoustic feature extraction module 810 is configured as follows:

[0184] The sample speech data is input into a trained acoustic feature extraction network, and the initial speech acoustic features are obtained as output.

[0185] In an exemplary embodiment of the present specification, based on the above solution, the training device 800 of the speech pre-training model further includes a speech data pre-processing unit, which is configured to:

[0186] Performing denoising on the collected original speech data to obtain denoised original speech data;

[0187] Performing silence elimination processing on the denoised original speech data to obtain silence eliminated original speech data;

[0188] Performing volume normalization processing on the original voice data after the silence is eliminated to obtain original voice data with a standardized volume;

[0189] The original voice data of the standardized volume is subjected to frame processing to obtain the sample voice data.

[0190] In an exemplary embodiment of the present specification, based on the above solution, the training device 800 of the speech pre-training model further includes a user speech data processing unit, which is configured to:

[0191] Acquire user voice data, and input the user voice data into the trained voice pre-training model to obtain a predicted voice feature representation;

[0192] Inputting the predicted speech feature representation into a downstream speech task processing network to obtain a speech processing result;

[0193] Among them, the downstream speech task processing network includes any one or a combination of a speech recognition conversion processing network, a language recognition processing network, a speech intention recognition processing network, a speaker identity recognition processing network and a speech enhancement processing network.

[0194] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0195] The embodiments of this specification also provide a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the method as described in the embodiments of this specification. The specific execution process can be found in the specific description of the embodiments of this specification, and will not be repeated here.

[0196] This specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor and executes the method as described in the embodiments of this specification. The specific execution process can be found in the specific description of the embodiments of this specification, and will not be repeated here.

[0197] The embodiments of this specification also provide Fig. 9 The structural diagram of the electronic device shown in FIG. Fig. 9 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned voice activity detection method.

[0198] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0199] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0200] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0201] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0202] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0203] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0204] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0205] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0207] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0208] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0209] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0210] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0211] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0213] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0214] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for training a speech pre-training model, the method comprising: Acquiring sample speech data, and performing feature extraction on the sample speech data to obtain initial speech acoustic features; Performing fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution; Based on the speech feature cluster distribution, self-supervisory training is performed on the pre-constructed speech pre-training model to obtain a trained speech pre-training model.

2. According to the training method of the speech pre-training model of claim 1, the fuzzy clustering processing of the initial speech acoustic features to obtain the speech feature cluster distribution comprises: Determine the initial cluster centers of a preset number of clusters, and update the initial cluster centers to obtain target cluster centers; The initial speech acoustic features are clustered based on the target cluster center to obtain the speech feature cluster distribution corresponding to each initial speech acoustic feature belonging to the target cluster center.

3. According to the training method of the speech pre-training model of claim 2, the updating of the initial cluster center to obtain the target cluster center comprises: Determining a membership matrix between each of the initial speech acoustic features and the initial cluster center; Determining an intermediate cluster center based on the membership matrix and each of the initial speech acoustic features, and using the intermediate cluster center as a new initial cluster center; The above steps are executed repeatedly until the update iteration condition is met, and the latest intermediate cluster center is used as the target cluster center.

4. According to the training method of the speech pre-training model of claim 3, the determining of the membership matrix of each of the initial speech acoustic features and the initial clustering center comprises: Determining a first similarity distance between each of the initial speech acoustic features and a current cluster center among the initial cluster centers; Determining a second similarity distance between each of the initial speech acoustic features and all cluster centers in the initial cluster centers; Obtaining a preset fuzzy factor, and determining membership data of each of the initial speech acoustic features belonging to the current cluster center according to the fuzzy factor, the first similarity distance, and the second similarity distance; The membership matrix of each of the initial speech acoustic features and the initial cluster center is determined through the membership data.

5. According to the training method of the speech pre-training model of claim 2, clustering each of the initial speech acoustic features based on the target cluster center to obtain the speech feature cluster distribution corresponding to each of the initial speech acoustic features belonging to the target cluster center, comprising: Determining a membership vector of each of the initial speech acoustic features belonging to the target cluster center, wherein the membership vector includes a degree of belonging of each of the initial speech acoustic features to each cluster category in the target cluster center; The membership vector is used as the speech feature cluster distribution corresponding to each of the initial speech acoustic features belonging to the target cluster center.

6. The training method of the speech pre-training model according to claim 1, wherein the pre-constructed speech pre-training model is subjected to self-supervision training based on the speech feature cluster distribution to obtain a trained speech pre-training model, comprising: Inputting the initial speech acoustic features into a pre-built speech pre-training model to obtain a speech feature prediction distribution; According to the speech feature prediction distribution and the speech feature cluster distribution, the model parameters of the pre-constructed speech pre-training model are updated to obtain a trained speech pre-training model.

7. The training method of the speech pre-training model according to claim 2, wherein the pre-constructed speech pre-training model is updated with model parameters according to the speech feature prediction distribution and the speech feature cluster distribution to obtain a trained speech pre-training model, comprising: Determining the loss between the speech feature prediction distribution and the speech feature cluster distribution by using the target loss function corresponding to the pre-built speech pre-training model; Updating model parameters of the pre-built speech pre-training model according to the loss; The above steps are executed repeatedly until the loss converges or the number of model update iterations is reached, thereby completing the self-supervised training of the pre-constructed speech pre-training model and obtaining a trained speech pre-training model.

8. The training method of the speech pre-training model according to claim 1, wherein the feature extraction of the sample speech data to obtain the initial speech acoustic features comprises: Performing windowing processing on the sample speech data to obtain a preprocessed speech frame; Performing short-time Fourier transform on the preprocessed speech frame to obtain a speech frame spectrum, and performing smoothing on the speech frame spectrum to obtain a speech spectrum envelope; Filtering the speech spectrum envelope based on a preset Mel filter, taking the logarithm of the filtering output result and performing a difference operation to obtain the Mel frequency cepstrum coefficient corresponding to the sample speech data; The Mel-frequency cepstral coefficients are used as initial speech acoustic features corresponding to the sample speech data.

9. The training method of the speech pre-training model according to claim 1, wherein the feature extraction of the sample speech data to obtain the initial speech acoustic features comprises: The sample speech data is input into a trained acoustic feature extraction network, and the initial speech acoustic features are obtained as output.

10. The method for training a speech pre-training model according to claim 8 or 9, further comprising: Performing denoising on the collected original speech data to obtain denoised original speech data; Performing silence elimination processing on the denoised original speech data to obtain silence eliminated original speech data; Performing volume normalization processing on the original voice data after the silence is eliminated to obtain original voice data with a standardized volume; The original voice data of the standardized volume is subjected to frame processing to obtain the sample voice data.

11. The method for training a speech pre-training model according to claim 1, further comprising: Acquire user voice data, and input the user voice data into the trained voice pre-training model to obtain a predicted voice feature representation; Inputting the predicted speech feature representation into a downstream speech task processing network to obtain a speech processing result; Among them, the downstream speech task processing network includes any one or a combination of a speech recognition conversion processing network, a language recognition processing network, a speech intention recognition processing network, a speaker identity recognition processing network and a speech enhancement processing network.

12. A training device for a speech pre-training model, the device comprising: An acoustic feature extraction module, used to obtain sample speech data and perform feature extraction on the sample speech data to obtain initial speech acoustic features; A feature soft clustering module, used for performing fuzzy clustering processing on the initial speech acoustic features to obtain speech feature cluster distribution; The self-supervised training module is used to perform self-supervised training on the pre-constructed speech pre-training model based on the speech feature cluster distribution to obtain a trained speech pre-training model.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 11.

15. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 11 are implemented.

Images