Model training method, device, equipment and storage medium

Abstract

The present disclosure provides a model training method, device, equipment and storage medium, which relate to the field of artificial intelligence, especially to the field of deep learning and speech recognition. The specific implementation scheme is: input multiple basic features corresponding to the audio sample into the initial network respectively to obtain multiple corresponding sample enhancement features, and the audio sample contains the voice of at least one speaker; cluster the multiple sample enhancement features to obtain the candidate clustering results corresponding to each speaker; according to the multiple candidate clustering results, adjust the parameters of the initial network until a feature enhancement model that meets the preset conditions is obtained; wherein the feature enhancement model is used to output the corresponding enhancement features according to the input basic features. The technology disclosed in the present disclosure improves the effect of voice separation.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and further to the fields of deep learning and speech recognition, and in particular to model training methods, devices, equipment and storage media. Background Art

[0002] In scenarios such as intelligent customer service, conference discussions, interviews, and variety shows, there are usually multiple speakers on a single channel, so it is necessary to separate the recorded voices and conduct targeted analysis. This process is usually done offline, that is, it requires inputting a long audio file, separating the given number of speakers or using a certain clustering threshold.

[0003] In the related art, the speaker separation system usually adopts a combination of TDNN (Time Delay Neural Networks), Xvector (a model for extracting speaker feature vectors) and AHC (Agglomerative Hierarchical Clustering). Summary of the invention

[0004] The present invention provides a model training method, apparatus, device and storage medium.

[0005] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0006] Inputting multiple basic features corresponding to the audio sample into the initial network respectively to obtain multiple corresponding sample enhanced features, where the audio sample includes the voice of at least one speaker;

[0007] Clustering is performed on multiple sample enhancement features to obtain candidate clustering results corresponding to each speaker;

[0008] According to multiple candidate clustering results, the parameters of the initial network are adjusted until a feature enhancement model that meets the preset conditions is obtained;

[0009] Among them, the feature enhancement model is used to output corresponding enhanced features based on the input basic features.

[0010] According to another aspect of the present disclosure, there is provided a speech processing method, comprising:

[0011] Extract features from multiple audio clips contained in the audio to be processed, respectively, to obtain basic features corresponding to the multiple audio clips, wherein the audio to be processed contains the voice of at least one speaker;

[0012] The basic features are enhanced by using the feature enhancement model to obtain enhanced features;

[0013] Clustering is performed on multiple enhanced features to obtain clustering results corresponding to each speaker;

[0014] Among them, the feature enhancement model is generated using the model training method according to the above embodiment of the present disclosure.

[0015] According to another aspect of the present disclosure, a model training device is provided, comprising:

[0016] A sample enhancement feature acquisition module, used to input multiple basic features corresponding to the audio sample into the initial network respectively to obtain multiple corresponding sample enhancement features, where the audio sample includes the voice of at least one speaker;

[0017] The first clustering module is used to perform clustering processing on multiple sample enhancement features to obtain candidate clustering results corresponding to each speaker;

[0018] A parameter adjustment module is used to adjust the parameters of the initial network according to multiple candidate clustering results until a feature enhancement model that meets preset conditions is obtained;

[0019] Among them, the feature enhancement model is used to output corresponding enhanced features based on the input basic features.

[0020] According to another aspect of the present disclosure, there is provided a speech processing device, comprising:

[0021] A feature extraction module is used to extract features from multiple audio clips contained in the audio to be processed, and obtain basic features corresponding to the multiple audio clips, wherein the audio to be processed contains the voice of at least one speaker;

[0022] A feature enhancement module is used to enhance the basic features using the feature enhancement model to obtain enhanced features;

[0023] A second clustering module is used to perform clustering processing on the multiple enhanced features to obtain clustering results corresponding to each speaker;

[0024] Among them, the feature enhancement model is generated using the model training device according to the above embodiment of the present disclosure.

[0025] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any embodiment of the present disclosure.

[0029] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to enable a computer to execute the method in any embodiment of the present disclosure.

[0030] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method in any embodiment of the present disclosure is implemented.

[0031] According to the technology disclosed in the present invention, the parameters of the initial network are adjusted by utilizing multiple clustering results of multiple sample enhancement features, that is, the parameters of the initial network are continuously updated through self-supervised learning strategies such as contrastive learning, so as to continuously improve the feature enhancement effect of the feature enhancement model, and improve the discrimination and identification ability of the speaker separation network, and give full play to the role of the clustering results to learn and update the initial network, so as to combine the feature enhancement model and the clustering model, and finally form an end-to-end speaker recognition network framework, avoiding the problem that the training results of the speaker recognition network obtained in the end do not match the test results.

[0032] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0034] Figure 1 A flow chart showing a model training method according to an embodiment of the present disclosure;

[0035] Figure 2 A specific flow chart showing the adjustment of the parameters of the initial network according to the model training method of the embodiment of the present disclosure;

[0036] Figure 3 A specific flow chart showing the construction of the first transformation matrix and the second transformation matrix of the model training method according to an embodiment of the present disclosure;

[0037] Figure 4 An example diagram showing an application of the model training method according to an embodiment of the present disclosure;

[0038] Figure 5 A flow chart showing a speech processing method according to an embodiment of the present disclosure;

[0039] Figure 6 A block diagram of a model training device according to an embodiment of the present disclosure is shown;

[0040] Figure 7 A block diagram of a speech processing device according to an embodiment of the present disclosure is shown;

[0041] Figure 8 It is a block diagram of an electronic device used to implement the model training method and / or speech processing method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] In the related art, the speaker separation system usually adopts a combination of TDNN (Time Delay Neural Networks), Xvector (a model for extracting speaker feature vectors) and AHC (Agglomerative Hierarchical Clustering).

[0044] The applicant found that the outputs of the training process and the test process do not match, which affects the effect of voice separation. To solve the above problem, the applicant proposed a solution to use clustering results to guide the update of network parameters of the feature extraction module, that is, the present disclosure provides the following model training method, speech processing method, device, equipment and storage medium.

[0045] Refer to the following Figures 1 to 4 Describe the model training method according to the embodiment of the present disclosure.

[0046] like Figure 1 As shown, the model training method according to the embodiment of the present disclosure includes the following steps:

[0047] S101: inputting multiple basic features corresponding to the audio sample into the initial network respectively to obtain multiple corresponding sample enhanced features, where the audio sample includes the voice of at least one speaker;

[0048] S102: performing clustering processing on multiple sample enhanced features to obtain candidate clustering results corresponding to each speaker;

[0049] S103: According to the multiple candidate clustering results, the parameters of the initial network are adjusted until a feature enhancement model that meets the preset conditions is obtained; wherein the feature enhancement model is used to output corresponding enhanced features according to the input basic features.

[0050] Exemplarily, in step S101, the audio sample can be obtained through an open source database, such as AiShell (a public large-scale Chinese speech database) or LibriSpeech (a public large-scale English speech database).

[0051] In one example, each audio sample may contain the voices of 2 to 7 speakers. Before step S101, the audio sample may be subjected to noise reduction processing to obtain a clean audio sample. The clean audio sample may be segmented to obtain multiple audio segments. For example, the clean audio sample may be segmented into multiple audio segments with a duration of 25 ms and a frame shift of 10 ms.

[0052] Then, multiple audio clips are respectively input into the pre-trained feature extraction model to obtain corresponding basic features. The dimension of the basic features can be 128 dimensions, and the basic features can be obtained by accumulating and averaging the features corresponding to a preset number of audio clips. The number of basic features finally obtained can be 50 to 700.

[0053] Among them, the feature extraction model can extract Mel-Frequency Cepstral Coefficients (MFCC) for each audio clip to obtain the basic features corresponding to each audio clip. In other examples of the present disclosure, frequency domain features (Filter Bank, FBank) can also be extracted for each audio clip to obtain the basic features corresponding to the audio clip, or perceptual linear prediction features (Perceptual Linear Predictive, PLP) can be extracted for the audio clip.

[0054] After multiple basic features are input into the initial network, the initial network first performs whitening on the basic features to obtain the first transformed features, then performs dimensionality reduction on the first transformed features, and finally outputs the corresponding sample enhanced features.

[0055] Exemplarily, in step S102, a pre-trained clustering model may be used to perform clustering processing on multiple sample enhancement features to obtain multiple clustering results to be selected.

[0056] In one example, the clustering model uses a probabilistic linear discriminant analysis algorithm to perform similarity determination on multiple enhanced features, and clusters the multiple enhanced features according to the similarity determination results.

[0057] Specifically, the clustering model may include a PLDA (Probability Linear Discriminant Analysis) module and an AHC (Agglomerative Hierarchical Clustering) module. The PLDA module outputs similarity values ​​between different enhanced features based on multiple input enhanced features. The greater the similarity value, the greater the possibility that the audio clips corresponding to the two enhanced features belong to the same speaker; the smaller the similarity value, the smaller the possibility that the audio clips corresponding to the two enhanced features belong to the same speaker. The AHC module can cluster multiple enhanced features and obtain multiple clustering results based on the similarity value output by the PLDA module by comparing the similarity value with a preset similarity threshold.

[0058] Exemplarily, in step S103, a self-supervised learning strategy may be adopted to adjust the parameters of the initial network using the multiple candidate clustering results obtained in step S102, and after multiple rounds of iterations, a feature enhancement model that meets preset conditions is obtained.

[0059] In one example, the strategy of contrastive learning can be used to adjust the parameters of the initial network. Specifically, the parameters of the initial network can be reversely updated according to the loss value between any two different sample enhancement features under the same clustering result, and the loss value between the sample enhancement features in two different clustering results, so that the difference between any two different sample enhancement features under the same clustering result becomes smaller and smaller, and the difference between the sample enhancement features under two different clustering results becomes larger and larger, thereby improving the feature extraction accuracy of the initial network and improving the distinguishability of the clustering results, until a feature enhancement model that meets the preset conditions is obtained.

[0060] According to the model training method of the embodiment of the present disclosure, the parameters of the initial network are adjusted by utilizing multiple clustering results of multiple sample enhancement features, that is, the parameters of the initial network are continuously updated through self-supervised learning strategies such as contrastive learning, so as to continuously improve the processing effect of the feature enhancement model, and improve the discrimination and identification ability of the speaker separation network, and give full play to the role of the clustering results to learn and update the initial network, so as to combine the feature enhancement model and the clustering model, and finally form an end-to-end speaker recognition network framework, avoiding the problem that the training results of the speaker recognition network obtained in the end do not match the test results.

[0061] like Figure 2 As shown, in one implementation, step S103 includes:

[0062] S201: From any candidate clustering result, select any two sample enhancement features as first anchor points, and calculate a first distance between the two first anchor points;

[0063] S202: Selecting a sample enhancement feature from any two candidate clustering results as a second anchor point, and calculating a second distance between the two second anchor points;

[0064] S203: Calculate a loss value according to the first distance and the second distance, and adjust parameters of the initial network according to the loss value.

[0065] For example, after the loss value is calculated, the parameters of the initial network and clustering model can be reversely updated using the gradient descent criterion, and after multiple rounds of iterations, a feature enhancement model that meets preset conditions is obtained.

[0066] It can be understood that in each round of iteration, the first anchor point can be reselected to calculate the first distance, and the second anchor point can be reselected to calculate the second distance, the loss value can be recalculated according to the first distance and the second distance, and the parameters of the initial network and clustering model can be adjusted and updated using the loss value.

[0067] In one example, the loss value can be calculated according to the ternary loss function. The ternary loss function is as follows:

[0068]

[0069] in, It is used to characterize the first distance between two different sample enhancement features xj,1 and xj,2 in the same clustering result j. It is used to characterize the second distance between the sample enhancement features xj,1 and xk≠j,2 of two different clustering results j and k, and m is used to characterize the interval parameter between different clustering results. The value of m can be 4.

[0070] According to the above implementation, by calculating the first distance between two different sample enhancement features under the same clustering result, and calculating the second distance between the sample enhancement features under two different clustering results, and updating the parameters of the initial network according to the first distance and the second distance, the clustering results of the sample enhancement features can be used to guide the updating of the parameters of the initial network, thereby improving the processing effect of the feature enhancement model, and at the same time helping to improve the discrimination and accuracy of subsequent clustering results.

[0071] In one embodiment, the initial network can be composed of a layer of deep neural network, which includes an input layer, a hidden layer and an output layer. There is a first transformation matrix between the input layer and the hidden layer, and the first transformation matrix is ​​used to transform the covariance matrix of the basic features into a unit matrix to obtain a first transformation feature; there is a second transformation matrix between the hidden layer and the output layer, and the second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature.

[0072] By calculating the product of the data matrix of the basic feature and the first transformation matrix, the first transformation feature can be obtained, so that the initial network can achieve whitening of the basic feature. Then, by calculating the product of the data matrix of the first transformation feature and the second transformation matrix, the corresponding sample enhancement feature can be obtained, so that the initial network can further achieve dimensionality reduction of the first transformation feature.

[0073] It can be understood that whitening refers to removing redundant information of the basic features and reducing the correlation between the various dimensional information of the basic features so that the various dimensional information of the basic features have the same variance, thereby obtaining the first transformed feature. Dimensionality reduction can reduce the number of dimensions of the basic features while ensuring that important information in the basic features is not lost.

[0074] In one example, after the basic features are transformed into the first transformed features using the first transformation matrix, the first transformed features may be length-regularized so that the vectors of the first transformed features are unified to unit length, and then the first transformed features are converted into sample enhanced features using the second transformation matrix.

[0075] Through the above implementation, the feature enhancement model finally trained can perform whitening and dimensionality reduction processing on the input basic features to output enhanced features. Therefore, the feature enhancement model trained according to the model training method of the embodiment of the present disclosure can reduce the correlation between the multiple dimensional information contained in the basic features to remove the redundant information therein, and the feature enhancement model can further simplify the basic features, and the output enhanced features retain the important information in the basic features. Therefore, the feature enhancement model obtained by the model training method of the embodiment of the present disclosure can improve the feature enhancement performance, which is conducive to improving the speaker separation effect.

[0076] like Figure 3 As shown, in one implementation, the first transformation matrix and the second transformation matrix are constructed in the following manner:

[0077] S301: Input multiple audio clips of the audio sample into a pre-trained feature extraction model to obtain a basic feature set;

[0078] S302: Determine initial parameters of the first transformation matrix and initial parameters of the second transformation matrix according to the basic feature set.

[0079] Exemplarily, according to the basic feature set, a whitening processing model and a dimensionality reduction processing model are trained respectively, wherein the initial parameters of the first transformation matrix can be determined according to the parameters of the whitening processing model, and the initial parameters of the second transformation matrix can be determined according to the parameters of the dimensionality reduction processing model.

[0080] According to the above implementation, the feature enhancement model obtained through training improves the feature extraction accuracy of the audio to be processed containing the voices of multiple speakers, which is beneficial to improving the clustering accuracy of multiple voice segments, thereby improving the human voice separation effect. In addition, by using the basic feature set to initialize the parameters of the first transformation matrix and the second transformation matrix of the initial network, the output of the pre-trained feature extraction model can be directly connected to the input of the feature enhancement model to improve the adaptability between the two. In addition, the initial network model can be equipped with the whitening processing capability and dimensionality reduction processing capability for the basic features, so that a feature enhancement model that meets the preset conditions can be obtained through training.

[0081] In one implementation, step S102 includes:

[0082] A pre-trained clustering model is used to perform clustering processing on multiple sample enhancement features; wherein the clustering model is trained based on a basic feature set.

[0083] In one example, the clustering model includes a PLDA module. The PLDA module can be trained and generated using the basic feature set obtained in step S401. It is understandable that the PLDA module is used to perform similarity judgment on two different sample enhancement features, that is, output a similarity value between two different enhancement features based on the input multiple enhancement features. The clustering model uses a hierarchical clustering algorithm to cluster multiple sample enhancement features according to the similarity values ​​output by the PLDA module, thereby obtaining multiple clustering results.

[0084] Therefore, by using the basic feature set to train the clustering model, the clustering model can be combined with the pre-trained feature extraction model, thereby combining the feature enhancement model with the clustering model, improving the adaptability between the feature enhancement model and the clustering model, and then improving the training efficiency of the feature enhancement model.

[0085] Refer to the following Figure 4 A specific example according to the present disclosure is described.

[0086] Reference Figure 4 As shown, the model training method of the embodiment of the present disclosure specifically includes the following steps:

[0087] Step 1: Pre-training basic feature extraction model: Use open source databases to collect a certain amount of open source data to obtain audio samples. The open source databases can be Aishell, Librispeech, etc. Pre-process each audio sample containing only one speaker's voice, including removing noise (such as environmental noise, busy tone, ringtone, silence, etc.) to obtain a clean audio sample, and then perform frame processing on the audio sample to obtain multiple audio frame samples;

[0088] Extract low-level features from each audio frame sample, such as 20-dimensional MFCC, PLP or Fbank, and obtain 20*3-dimensional basic features of each audio frame sample through first-order difference processing, second-order difference processing and regularization processing (i.e., normal distribution regularization);

[0089] Among them, each audio frame sample needs to be combined with a certain amount of context and then input into the initial network for forward calculation. The output is the probability of each speaker. The loss value is calculated according to the cross entropy function, and the network parameters are reversely updated based on the loss value using the stochastic gradient descent criterion. The above steps are repeated for each audio frame sample, and iterated until the network converges to obtain the feature extraction model.

[0090] Step 2: Pre-train the whitening processing model, dimensionality reduction model and PLDA model: Use the feature model obtained in step 1 and open source data to obtain the basic feature set; use the basic feature set to train the whitening processing model, dimensionality reduction model and PLDA model.

[0091] Step 3: Initialize the feature enhancement model using the whitening processing model and the dimensionality reduction model: Initialize the parameters of the first transformation matrix between the input layer and the hidden layer of the feature enhancement model using the parameters of the whitening processing model, and initialize the parameters of the second transformation matrix between the hidden layer and the output layer of the feature enhancement model using the dimensionality reduction model to obtain the initial feature enhancement model.

[0092] Step 4: Train the initial feature enhancement model based on the clustering results: pre-process the audio samples (each audio contains the voices of multiple speakers, for example, 2 to 7 speakers) and extract the basic feature set X, where the basic feature set X is a set of basic features corresponding to multiple audio segments of the audio sample; then, input the basic feature set into the initial feature enhancement model (NN) to obtain the corresponding enhanced feature set Y P ; Use PLDA model to enhance the feature set Y p Cluster multiple enhanced features in to obtain multiple clustering results (z p); according to the comparison learning strategy, calculate the first distance between any two enhanced features in the same clustering result, and calculate the second distance between the enhanced features in two different clustering results, and based on the first distance and the second distance, use the ternary loss function to calculate the loss value; adjust the parameters of the feature enhancement model and the PLDA model according to the loss value, and obtain the converged feature enhancement model and the PLDA model after multiple rounds of iterations.

[0093] Step 5: Test reasoning: Preprocess the test audio and extract feature set X and feature set Y respectively; cluster feature set Y to obtain multiple clustering results, each clustering result is equivalent to a set of audio clips of the same speaker's voice, thereby obtaining the result of voice separation.

[0094] Reference below Figure 5 The speech processing method according to the embodiment of the present disclosure is described. The speech processing method according to the embodiment of the present disclosure can perform human voice separation on the voice of at least one speaker contained in the audio to be processed.

[0095] like Figure 5 As shown, the speech processing method of the embodiment of the present disclosure includes the following steps:

[0096] S501: extracting features from multiple audio clips contained in the audio to be processed, respectively, to obtain basic features corresponding to the multiple audio clips, wherein the audio to be processed contains the voice of at least one speaker;

[0097] S502: using a feature enhancement model to enhance the basic features to obtain enhanced features;

[0098] S503: performing clustering processing on the multiple enhanced features to obtain clustering results corresponding to the speakers;

[0099] Among them, the feature enhancement model is generated using the model training method of the above embodiment of the present disclosure.

[0100] Exemplarily, before step S501, the audio to be processed may be pre-processed to obtain a plurality of audio segments.

[0101] Specifically, first, the audio to be processed is denoised, that is, the non-human voices such as environmental noise, busy tone, ringtone or silence in the audio to be processed are removed to obtain clean audio to be processed. Then, the clean audio to be processed is framed, for example, the clean audio to be processed can be cut into multiple audio segments with a duration of 25ms and a frame shift of 10ms.

[0102] Exemplarily, in step S501, multiple audio segments included in the audio to be processed may be respectively input into a pre-trained feature extraction model to obtain basic features corresponding to each audio segment.

[0103] In one example, the feature extraction model can extract Mel-Frequency Cepstral Coefficients (MFCC) for each audio clip to obtain basic features corresponding to each audio clip. In other examples of the present disclosure, frequency domain features (Filter Bank, FBank) can also be extracted for each audio clip to obtain basic features corresponding to the audio clip, or perceptual linear prediction features (Perceptual Linear Predictive, PLP) can be extracted for the audio clip.

[0104] Furthermore, for the extracted Mel-frequency cepstral coefficients, frequency domain features or perceptual linear prediction features, first-order difference processing, second-order difference processing and regularization processing can be performed in sequence, that is, the above features are normally distributed and regularized to obtain 20*3-dimensional multi-dimensional features as the basic features of each audio clip.

[0105] Exemplarily, in step S502, the basic features corresponding to each audio clip may be input into a pre-trained feature enhancement model, and the feature enhancement model is specifically used to sequentially perform whitening and dimensionality reduction processing on the basic features to obtain corresponding enhanced features.

[0106] The whitening process refers to removing redundant information of the basic features and reducing the correlation between information of each dimension of the basic features so that information of each dimension of the basic features has the same variance, thereby obtaining the first transformed feature.

[0107] In one example, the feature enhancement model can whiten the basic features by using PCA (Principal Components Analysis) whitening. Specifically, a data matrix is ​​obtained based on the multidimensional information contained in the basic features, the data matrix is ​​transformed to obtain the corresponding oblique variance matrix, and then the eigenvalues ​​and eigenvectors of the covariance are calculated, and the original data matrix is ​​multiplied by the transpose of the eigenvector matrix to achieve a rotation transformation of the data, and then each dimension of the transformed data matrix is ​​divided by the corresponding variance to obtain the first transformed feature.

[0108] In another example, the feature enhancement model can also use ZCA (Zero-phase Component Analysis) whitening to whiten the basic features. ZCA whitening is based on PCA whitening, and rotates the PCA whitened data back to the original feature space, so that the transformed first transformed feature can be closer to the original basic feature.

[0109] For dimensionality reduction processing, the first transformed feature can be further processed by principal component analysis, thereby reducing the number of dimensions of the basic features while ensuring that important information in the basic features is not lost.

[0110] In one example, the dimensionality reduction process specifically includes the following steps: performing data centering on the first transformed feature; obtaining the covariance matrix of the first transformed feature; determining the eigenvalues ​​and eigenvectors of the covariance based on the covariance matrix; sorting the eigenvalues ​​in descending order, selecting the largest k of them, and then using the corresponding k eigenvectors as column vectors to form an eigenvector matrix; finally, projecting the sample points onto the selected eigenvectors to obtain enhanced features.

[0111] The feature enhancement model can be composed of a layer of deep neural networks (DNN), where the deep neural network includes an input layer, a hidden layer, and an output layer. The input layer is used to input basic features, and the output layer is used to output enhanced features.

[0112] For example, in step S503, a pre-trained clustering model may be used to perform clustering processing on multiple enhanced features to obtain multiple clustering results, and the multiple clustering results correspond one-to-one to the voice of at least one speaker included in the audio to be processed. Each clustering result includes at least one audio segment.

[0113] It should be noted that the clustering model can adopt various clustering algorithms, such as K-means clustering algorithm, mean shift algorithm, density clustering (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), etc., as long as it can achieve clustering of multiple enhanced features and obtain multiple clustering results.

[0114] According to the speech processing method of the embodiment of the present disclosure, the basic features extracted from each audio segment are enhanced by utilizing a feature enhancement model to obtain enhanced features, and then clustering is performed based on the multiple enhanced features to obtain multiple clustering results. This improves the feature extraction accuracy for the audio to be processed that contains the voices of multiple speakers, which is beneficial to improving the clustering accuracy for multiple speech segments, thereby improving the human voice separation effect.

[0115] In one embodiment, the feature enhancement model includes an input layer, a hidden layer and an output layer. There is a first transformation matrix between the input layer and the hidden layer, and the first transformation matrix is ​​used to transform the covariance matrix of the basic feature into a unit matrix to obtain a first transformation feature; there is a second transformation matrix between the hidden layer and the output layer, and the second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature.

[0116] There is a first transformation matrix between the input layer and the hidden layer, and there is a second transformation matrix between the hidden layer and the output layer. The initial parameters of the first transformation matrix can be determined according to the parameters of the pre-trained whitening model, and the initial parameters of the second transformation matrix can be determined according to the parameters of the pre-trained dimensionality reduction model. The whitening model and the dimensionality reduction model can be trained separately by memory learning based on the basic feature set output by the feature extraction model.

[0117] It is understandable that, during the training process of the initial network, the parameters of the first transformation matrix and the second transformation matrix need to be continuously adjusted and updated to obtain a feature enhancement model that meets the convergence conditions.

[0118] By calculating the product of the data matrix of the basic feature and the first transformation matrix, the first transformation feature can be obtained, thereby realizing the whitening processing of the basic feature. By calculating the product of the data matrix of the first transformation feature and the second transformation matrix, the enhanced feature can be obtained, thereby realizing the dimensionality reduction processing of the first transformation feature.

[0119] In addition, after the basic features are transformed into the first transformed features using the first transformation matrix, the first transformed features can also be length-regularized so that the vectors of the first transformed features are unified to unit length, and then the first transformed features are converted into enhanced features using the second transformation matrix.

[0120] According to the speech processing method of the embodiment of the present disclosure, the basic features are whitened by using the first transformation matrix between the input layer and the hidden layer of the feature enhancement model, and the first transformation features after whitening are reduced in dimension by using the second transformation matrix between the hidden layer and the output layer, and finally the enhanced features corresponding to each audio segment are obtained, thereby reducing the correlation between the multiple dimensional information contained in the basic features, removing the redundant information therein, achieving further simplification of the basic features, and the important information in the basic features is retained in the enhanced features finally obtained. Therefore, according to the technology of the present disclosure, the feature extraction accuracy for the audio to be processed containing the voice of at least one speaker is improved, thereby improving the clustering accuracy for multiple speech segments, and further improving the human voice separation effect.

[0121] In one implementation, step S501 includes:

[0122] The feature extraction model is used to perform feature extraction processing on multiple audio clips respectively; wherein the feature extraction model includes multiple dilated convolution neural networks (TDCN).

[0123] Exemplarily, the feature extraction model may be a feature extraction module included in a pre-trained human voice separation model. The human voice separation model may be composed of a plurality of dilated convolutional layers, a fully connected layer, and an activation layer, and the plurality of dilated convolutional layers form a feature extraction module.

[0124] The spectral features extracted by the last dilated convolution layer are the basic features, and the dimension of the basic features may be 128. It is understandable that the basic features may be accumulated and averaged by a preset number of audio clips, for example, the basic features may be calculated once for every 50 audio clips.

[0125] It is understandable that dilated convolution, also known as hole convolution or expanded convolution, injects holes into the standard convolution kernel to increase the receptive field of the feature extraction model.

[0126] According to the above implementation, by using a feature extraction model with multiple dilated convolutional layers to extract features from audio clips and obtain basic features, the density of multiple dimensional information contained in the basic features can be increased, thereby improving the feature extraction effect of the audio clip.

[0127] In one implementation, step S503 includes:

[0128] Clustering multiple enhanced features using a clustering model;

[0129] The clustering model uses a probability linear discriminant analysis algorithm (Probability Linear Discriminant Analysis, PLDA) to perform similarity judgment on multiple enhanced features, and clusters the multiple enhanced features according to the similarity judgment results.

[0130] Exemplarily, the clustering model includes a PLDA module and an AHC (Agglomerative Hierarchical Clustering, hierarchical clustering algorithm) module. The PLDA module outputs similarity values ​​between different enhanced features based on multiple input enhanced features. The larger the similarity value, the greater the possibility that the audio clips corresponding to the two enhanced features belong to the same speaker; the smaller the similarity value, the smaller the possibility that the audio clips corresponding to the two enhanced features belong to the same speaker. The AHC module can cluster multiple enhanced features and obtain multiple clustering results based on the similarity value output by the PLDA module by comparing the similarity value with a preset similarity threshold.

[0131] It is understandable that, since the multiple enhanced features correspond to the multiple voice segments one by one, the multiple clustering results obtained are equivalent to the results of clustering the multiple voice segments. By comparing the similarity value with the preset similarity threshold, the multiple enhanced features can be clustered according to the comparison result and multiple clustering results can be obtained.

[0132] According to the above embodiment, clustering of multiple enhanced features is achieved by using a pre-trained clustering model, and the purpose of separating the voice of at least one speaker contained in the audio to be processed is achieved based on the clustering result.

[0133] According to another aspect of the present disclosure, a model training device is also provided.

[0134] like Figure 6 As shown, the model training device includes:

[0135] A sample enhancement feature acquisition module 601 is used to input multiple basic features corresponding to the audio sample into the initial network respectively to obtain multiple corresponding sample enhancement features, and the audio sample includes the voice of at least one speaker;

[0136] A first clustering module 602 is used to perform clustering processing on multiple sample enhancement features to obtain candidate clustering results corresponding to each speaker;

[0137] The parameter adjustment module 603 is used to adjust the parameters of the initial network according to the multiple candidate clustering results until a feature enhancement model that meets the preset conditions is obtained;

[0138] Among them, the feature enhancement model is used to output corresponding enhanced features based on the input basic features.

[0139] In one implementation, the parameter adjustment module 603 includes:

[0140] A first calculation submodule is used to select any two sample enhancement features from any clustering result to be selected as first anchor points, and calculate a first distance between the two first anchor points;

[0141] A second calculation submodule is used to select a sample enhancement feature from any two to-be-selected clustering results as a second anchor point, and calculate a second distance between the two second anchor points;

[0142] The parameter adjustment submodule is used to calculate the loss value according to the first distance and the second distance, and adjust the parameters of the initial network according to the loss value.

[0143] In one embodiment, the initial network includes an input layer, a hidden layer and an output layer. There is a first transformation matrix between the input layer and the hidden layer, and the first transformation matrix is ​​used to transform the covariance matrix of the basic features into a unit matrix to obtain a first transformation feature; there is a second transformation matrix between the hidden layer and the output layer, and the second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature.

[0144] In one embodiment, the building blocks for building the first transformation matrix and the second transformation matrix include:

[0145] A basic feature set acquisition submodule is used to input multiple audio clips of an audio sample into a pre-trained feature extraction model to obtain a basic feature set;

[0146] The initial parameter determination submodule is used to determine the initial parameters of the first transformation matrix and the initial parameters of the second transformation matrix according to the basic feature set.

[0147] In one implementation, the first clustering module 602 is further configured to:

[0148] Use the pre-trained clustering model to cluster multiple sample enhancement features;

[0149] Among them, the clustering model is trained according to the basic feature set.

[0150] According to another aspect of the present disclosure, a speech processing device is also provided.

[0151] like Figure 7 As shown, the speech processing device comprises:

[0152] A feature extraction module 701 is used to extract features from multiple audio segments contained in the audio to be processed, and obtain basic features corresponding to the multiple audio segments, wherein the audio to be processed contains the voice of at least one speaker;

[0153] The feature enhancement module 702 is used to enhance the basic features using the feature enhancement model to obtain enhanced features;

[0154] The second clustering module 703 is used to perform clustering processing on the multiple enhanced features to obtain clustering results corresponding to each speaker;

[0155] Among them, the feature enhancement model is generated using the model training device of the above embodiment of the present disclosure.

[0156] In one embodiment, the feature enhancement model includes an input layer, a hidden layer and an output layer. There is a first transformation matrix between the input layer and the hidden layer, and the first transformation matrix is ​​used to transform the covariance matrix of the basic feature into a unit matrix to obtain a first transformation feature; there is a second transformation matrix between the hidden layer and the output layer, and the second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature.

[0157] In one implementation, the feature extraction module 701 is further used to:

[0158] Using the feature extraction model, feature extraction is performed on multiple audio clips respectively;

[0159] Among them, the feature extraction model includes multiple dilated convolutional layers.

[0160] In one implementation, the second clustering module 703 is further configured to:

[0161] Clustering multiple enhanced features using a clustering model;

[0162] Among them, the clustering model uses a probabilistic linear discriminant analysis algorithm to perform similarity judgment on multiple enhanced features, and clusters the multiple enhanced features according to the similarity judgment results.

[0163] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0164] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0165] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0166] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0167] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0168] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a model training method and / or a speech processing method. For example, in some embodiments, the model training method and / or the speech processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the model training method and / or the speech processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the model training method and / or the speech processing method in any other suitable manner (e.g., by means of firmware).

[0169] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0171] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0173] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0174] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0175] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0176] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method, comprising: Inputting multiple basic features corresponding to the audio sample into the initial network respectively to obtain multiple corresponding sample enhanced features, wherein the audio sample contains the voice of at least one speaker; the initial network includes an input layer, a hidden layer and an output layer, wherein a first transformation matrix is ​​provided between the input layer and the hidden layer, and the first transformation matrix is ​​used to transform the covariance matrix of the basic features into a unit matrix to obtain a first transformation feature; and a second transformation matrix is ​​provided between the hidden layer and the output layer, and the second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature; Performing clustering processing on the plurality of sample enhancement features to obtain candidate clustering results corresponding to the speakers respectively; According to the plurality of the selected clustering results, the parameters of the initial network are adjusted until a feature enhancement model that meets the preset conditions is obtained; The feature enhancement model is used to output corresponding enhanced features according to the input basic features; The first transformation matrix and the second transformation matrix are constructed in the following manner: Input multiple audio clips of the audio sample into a pre-trained feature extraction model to obtain a basic feature set; According to the basic feature set, initial parameters of the first transformation matrix and initial parameters of the second transformation matrix are determined.

2. The method according to claim 1, wherein: According to the plurality of the candidate clustering results, adjusting the parameters of the initial network includes: From any of the to-be-selected clustering results, select any two sample enhancement features as first anchor points, and calculate a first distance between the two first anchor points; Selecting a sample enhancement feature from any two of the candidate clustering results as a second anchor point, and calculating a second distance between the two second anchor points; A loss value is calculated according to the first distance and the second distance, and a parameter of the initial network is adjusted according to the loss value.

3. The method according to claim 1 or 2, wherein: Clustering the plurality of sample enhancement features comprises: Using a pre-trained clustering model, clustering the plurality of sample enhancement features; Wherein, the clustering model is trained according to the basic feature set.

4. A speech processing method, comprising: Extract features from multiple audio clips contained in the audio to be processed, respectively, to obtain basic features corresponding to the multiple audio clips, respectively, wherein the audio to be processed contains the voice of at least one speaker; Using a feature enhancement model to enhance the basic features to obtain enhanced features; Performing clustering processing on the plurality of enhanced features to obtain clustering results corresponding to the speakers; Wherein, the feature enhancement model is generated using the model training method according to any one of claims 1 to 3.

5. The method according to claim 4, wherein: The feature enhancement model includes an input layer, a hidden layer and an output layer. There is a first transformation matrix between the input layer and the hidden layer. The first transformation matrix is ​​used to transform the covariance matrix of the basic features into a unit matrix to obtain a first transformation feature; there is a second transformation matrix between the hidden layer and the output layer. The second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature.

6. The method according to claim 4, wherein: Feature extraction is performed on multiple audio segments contained in the audio to be processed, including: Using a feature extraction model, extracting features from the plurality of audio clips respectively; Wherein, the feature extraction model includes multiple dilated convolutional layers.

7. The method according to claim 4, wherein: Clustering the plurality of enhanced features comprises: Performing clustering processing on the plurality of enhanced features using a clustering model; The clustering model uses a probabilistic linear discriminant analysis algorithm to perform similarity determination on the plurality of enhanced features, and performs clustering processing on the plurality of enhanced features according to the similarity determination result.

8. A model training device, comprising: A sample enhancement feature acquisition module, used to input multiple basic features corresponding to the audio sample into the initial network respectively to obtain multiple corresponding sample enhancement features, wherein the audio sample contains the voice of at least one speaker; the initial network includes an input layer, a hidden layer and an output layer, and there is a first transformation matrix between the input layer and the hidden layer, and the first transformation matrix is ​​used to transform the covariance matrix of the basic features into a unit matrix to obtain a first transformation feature; there is a second transformation matrix between the hidden layer and the output layer, and the second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature; A first clustering module, used for clustering the plurality of sample enhancement features to obtain candidate clustering results corresponding to each speaker; A parameter adjustment module, used to adjust the parameters of the initial network according to the plurality of clustering results to be selected, until a feature enhancement model that meets preset conditions is obtained; The feature enhancement model is used to output corresponding enhanced features according to the input basic features; The building blocks for building the first transformation matrix and the second transformation matrix include: A basic feature set acquisition submodule is used to input multiple audio clips of an audio sample into a pre-trained feature extraction model to obtain a basic feature set; The initial parameter determination submodule is used to determine the initial parameters of the first transformation matrix and the initial parameters of the second transformation matrix according to the basic feature set.

9. The device according to claim 8, wherein: The parameter adjustment module comprises: A first calculation submodule, configured to select any two sample enhancement features from any of the to-be-selected clustering results as first anchor points, and calculate a first distance between the two first anchor points; A second calculation submodule is used to select a sample enhancement feature from any two of the to-be-selected clustering results as a second anchor point, and calculate a second distance between the two second anchor points; A parameter adjustment submodule is used to calculate a loss value according to the first distance and the second distance, and adjust the parameters of the initial network according to the loss value.

10. The device according to claim 8 or 9, wherein: The first clustering module is further configured to: Using a pre-trained clustering model, clustering the plurality of sample enhancement features; Wherein, the clustering model is trained according to the basic feature set.

11. A speech processing device, comprising: A feature extraction module, configured to extract features from a plurality of audio segments contained in the audio to be processed, respectively, to obtain basic features corresponding to the plurality of audio segments, respectively, wherein the audio to be processed contains the voice of at least one speaker; A feature enhancement module, used to enhance the basic features using a feature enhancement model to obtain enhanced features; A second clustering module, used for clustering the plurality of enhanced features to obtain clustering results corresponding to the speakers; Wherein, the feature enhancement model is generated using the model training device according to any one of claims 8 to 10.

12. The device according to claim 11, wherein The feature enhancement model includes an input layer, a hidden layer and an output layer. There is a first transformation matrix between the input layer and the hidden layer. The first transformation matrix is ​​used to transform the covariance matrix of the basic features into a unit matrix to obtain a first transformation feature; there is a second transformation matrix between the hidden layer and the output layer. The second transformation matrix is ​​used to perform dimensionality reduction processing on the first transformation feature to obtain an enhanced feature.

13. The device according to claim 11, wherein: The feature extraction module is also used for: Using a feature extraction model, extracting features from the plurality of audio clips respectively; Wherein, the feature extraction model includes multiple dilated convolutional layers.

14. The device according to claim 11, wherein: The second clustering module is further used for: Performing clustering processing on the plurality of enhanced features using a clustering model; The clustering model uses a probabilistic linear discriminant analysis algorithm to perform similarity determination on the plurality of enhanced features, and performs clustering processing on the plurality of enhanced features according to the similarity determination result.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images