Blind speech separation method based on double-branch convolutional neural network fusion model

Abstract

The application discloses a blind speech separation method based on a double-branch convolutional neural network fusion model, which comprises the following steps: acquiring a blind speech signal and a blind speech signal amplitude spectrum, wherein the blind speech signal is a double-mixed speech signal; preprocessing the acquired blind speech signal, and extracting deep fusion features of the blind speech signal according to a double-branch convolutional neural network fusion model; marking the blind speech signal by using a classifier according to the deep fusion features of the blind speech signal; inputting the blind speech signal amplitude spectrum into a speech separation model matched with the marking label, and obtaining a target speech signal, so that blind speech separation is completed. The application can separate double-mixed speech signals.

Description

TECHNICAL FIELD

[0001] The application relates to a blind speech separation method based on a dual-branch convolutional neural network fusion model and belongs to the technical field of speaker recognition and speech separation. BACKGROUND

[0002] In recent years, deep learning technology has been widely used in speech signal processing, and scholars have proposed various deep learning-based speech separation methods. Research on single-channel speech separation includes speaker speech separation, speech and noise separation, and singing voice separation, which mainly analyzes target speech and interfering speech in the time-frequency domain to extract target speech from mixed signals.

[0003] Scholars have modeled deep neural networks (DNN), convolutional neural networks (CNN), and long short-term memory networks (LSTM) to achieve speech separation.

[0004] These research works all achieve speech separation according to the nonlinear mapping relationship between the input mixed speech features and the target speech features obtained by training various deep neural networks. Compared with shallow models, deep learning methods have more advantages in solving single-channel speech separation problems. When dealing with two-speaker mixed speech separation problems, the speaker mixing is usually discussed according to the speaker mixing situation, including male-female mixing (Male-Female, M-F) and same-sex mixing, which includes male-male mixing (Male-Male, M-M) and female-female mixing (Female-Female, F-F). Previous speech separation research has discussed gender combinations separately, i.e., directly performing speech separation on matched models. However, the gender combination of mixed speech is unknown in real speech separation. If the mixed speech gender combination is first determined, and then the corresponding model is used for speech separation, this is more suitable for actual blind speech separation.

[0005] Determining the speaker gender combination is a problem in the field of speaker recognition. In recent years, speaker recognition technology has developed rapidly, and deep learning-based speaker recognition methods have gradually become mainstream. The main advantage of deep learning over traditional methods is its powerful representation ability, which can extract highly abstract embedding features from speech for speaker recognition. Since in most cases, the speech of two people exists simultaneously in mixed speech, one common gender speech exists in both the male-female combination and the two same-sex combinations when training the model, which makes it much more difficult to recognize the mixed speech gender combination than traditional pure speech speaker recognition. It is very difficult to directly identify the three gender combinations using traditional single acoustic features, and it is easy to produce false judgments.

[0006] Therefore, the application proposes a blind speech separation method based on a dual-branch convolutional neural network fusion model. SUMMARY

[0007] The application aims to overcome the deficiencies in the prior art and provide a blind speech separation method based on a dual-branch convolutional neural network fusion model, which can separate dual mixed speech signals.

[0008] To achieve the above-mentioned purpose, the application is implemented by using the following technical solutions:

[0009] The application provides a blind speech separation method based on a dual-branch convolutional neural network fusion model, which includes the following steps:

[0010] Obtain a blind speech signal and a blind speech signal amplitude spectrum, wherein the blind speech signal is a dual mixed speech signal;

[0011] Preprocess the obtained blind speech signal, and input the preprocessed blind speech signal into a dual-branch convolutional neural network fusion model to obtain deep fusion features of the blind speech signal;

[0012] According to the deep fusion features of the blind speech signal, use a classifier to label the blind speech signal;

[0013] Input the blind speech signal amplitude spectrum into a speech separation model matched with the labeled label to obtain a target speech signal, and complete blind speech separation.

[0014] Further, the speech separation model is a male-male mixed speech separation model, a male-female mixed speech separation model, or a female-female mixed speech separation model.

[0015] Further, the dual-branch convolutional neural network fusion model includes a fusion layer and two branch convolutional neural networks.

[0016] Each branch convolutional neural network is provided with an input layer, two convolutional layers, two pooling layers, and a full connection layer.

[0017] Among them, each pooling layer is arranged after the corresponding convolutional layer.

[0018] Further, the input of the preprocessed blind speech signal into the dual-branch convolutional neural network fusion model to obtain the deep fusion features of the blind speech signal includes:

[0019] Extract filter bank features and mel cepstral coefficient features of the preprocessed blind speech signal;

[0020] Input the filter bank features and the mel cepstral coefficient features into the dual-branch convolutional neural network fusion model;

[0021] The full connection layer of one branch convolutional neural network integrates the mel-frequency cepstral coefficient features after convolution and pooling to obtain deep mel-frequency cepstral coefficient features;

[0022] The full connection layer of another branch convolutional neural network integrates the filter bank features after convolution and pooling to obtain deep filter bank features;

[0023] The fusion layer fuses the deep mel-frequency cepstral coefficient features and the deep filter bank features to obtain deep fusion features.

[0024] Further, the classifier is a support vector machine, and the deep fusion features are input into the trained support vector machine to label the blind speech signal by using the support vector machine to detect the type of gender combination of the blind speech signal.

[0025] Further, the speech separation model matched with the labeled label includes:

[0026] When the labeled label is 0, the speech separation model matched with the labeled label is a male-male mixed speech separation model;

[0027] When the labeled label is 1, the speech separation model matched with the labeled label is a male-female mixed speech separation model;

[0028] When the labeled label is 2, the speech separation model matched with the labeled label is a female-female mixed speech separation model.

[0029] Further, the speech separation model matched with the labeled label and the blind speech signal amplitude spectrum input includes:

[0030] The speech separation model matched with the labeled label and the blind speech signal amplitude spectrum input obtains an estimated ratio mask of the target speech signal;

[0031] The estimated ratio mask of the target speech signal is used to determine the target speech signal amplitude spectrum;

[0032] Based on the principle of inverse short-time Fourier transform, the target speech signal is obtained by using the target speech signal amplitude spectrum, and blind speech separation is completed.

[0033] Further, the estimated ratio mask of the target speech signal is used to determine the target speech signal amplitude spectrum, which includes the following formula:

[0034]

[0035] In the formula, is a target speech signal amplitude spectrum feature function, k is a time frame, f is a frequency, Y is a blind speech signal amplitude spectrum, is an estimated ratio mask of the target speech signal.

[0036] Further, the predicted target speech signal obtained by the target speech signal amplitude spectrum based on the inverse short-time Fourier transform principle comprises the following formula:

[0037]

[0038] In the formula, is the rth predicted target speech signal function, t is continuous time, ISTFT(·) is an inverse short-time Fourier transform function, S r (k,f) is the real amplitude spectrum feature of the rth speech signal, is an e exponential function, j is an imaginary unit, is the phase of the blind speech signal.

[0039] Further, the r is 1 or 2.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] In the gender combination type recognition stage of the unknown blind speech signal, the deep features of the blind speech signal are extracted by using the double-branch convolutional neural network fusion model, so that the deep fusion features used for recognition contain more rich speech information, thereby improving the accuracy of recognizing the gender combination type of the blind speech signal; in the speech separation stage of the blind speech signal, based on the recognition result of the classifier, a speech separation model matched with the blind speech signal is selected for speech separation, so that the target speech signal with high quality, high restoration degree and high intelligibility is obtained, which is superior to the traditional speech separation method. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 Fig. 1 shows an embodiment flowchart of a blind speech separation method based on a double-branch convolutional neural network fusion model of the present application;

[0043] Figure 2 Fig. 2 shows an embodiment flowchart of training a double-branch convolutional neural network and a classifier of the present application;

[0044] Figure 3 Fig. 3 shows an embodiment structure diagram of a branch convolutional neural network of the present application;

[0045] Figure 4 Fig. 4 shows an embodiment flowchart of extracting MFCC features and Fbank features of the present application;

[0046] Figure 5 Fig. 5 (a) is a feature distribution diagram of t-SNE dimension reduction MFCC features of three types of mixed speech signals;

[0047] Figure 5(b) is a feature distribution diagram of t-SNE dimensionality-reduced Fbank features of the three types of mixed speech signals;

[0048] Figure 6 (a) is a feature distribution diagram of t-SNE dimensionality-reduced DMFCC features of the three types of mixed speech signals;

[0049] Figure 6 (b) is a feature distribution diagram of t-SNE dimensionality-reduced DFbank features of the three types of mixed speech signals;

[0050] Figure 6 (c) is a feature distribution diagram of t-SNE dimensionality-reduced DMFCC+DFbank features of the three types of mixed speech signals;

[0051] Figure 7 An embodiment flowchart of training a speech separation model is shown.

[0052] Figure 8 The quality of the target speech signal of the M-F gender combination mixed speech signal is shown.

[0053] Figure 9 The quality of the target speech signal of the M-M gender combination mixed speech signal is shown.

[0054] Figure 10 The quality of the target speech signal of the F-F gender combination mixed speech signal is shown.

[0055] Figure 11 An embodiment flowchart of a blind speech separation method based on a double-branch convolutional neural network fusion model is shown. DETAILED DESCRIPTION

[0056] The application will be further described below with reference to the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0057] In the description of the present application, it needs to be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0058] In the description of the present application, it needs to be understood that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0059] The present application will be further described below in conjunction with the drawings. The following examples are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0060] In the description of the present application, it needs to be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0061] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "linking" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0062] Embodiment 1

[0063] The embodiment provides a blind speech separation method based on a double-branch convolutional neural network fusion model.

[0064] Reference Figure 11 The blind speech separation method based on the double-branch convolutional neural network fusion model of the embodiment comprises the following steps:

[0065] S1 acquires a blind speech signal and a blind speech signal amplitude spectrum, wherein the blind speech signal is a double-mixed speech signal.

[0066] S2 pre-processes the acquired blind speech signal, and inputs the pre-processed blind speech signal into a double-branch convolutional neural network fusion model to acquire a deep fusion feature of the blind speech signal.

[0067] In applications, because the speech signal has the characteristic of short-time stationarity, before performing Fast Fourier transform (FFT) on the speech signal, the speech signal needs to be pre-processed. Common pre-processing methods include frame division, windowing, pre-emphasis and the like.

[0068] In actual applications, compared with traditional feature extraction, the embodiment deeply mines the features of the blind speech signal, so that the deep fusion feature contains more information of gender combination categories, so as to improve the accuracy of recognition and judgment of the double-mixed speech signal, facilitate selection of a suitable speech separation model in a subsequent step, and improve the quality of the separated target speech signal.

[0069] S3 labels the blind speech signal by using a classifier according to the deep fusion feature of the blind speech signal.

[0070] Preferably, the classifier is a support vector machine (SVM).

[0071] In applications, the deep fusion feature is input into the trained support vector machine, and the support vector machine is used to label the blind speech signal, so as to detect the type of gender combination of the blind speech signal.

[0072] S4 inputs the blind speech signal amplitude spectrum into a speech separation model matched with the labeled tag to obtain a target speech signal, and completes blind speech separation.

[0073] In applications, the speech separation model is a male-male mixed speech separation model, a male-female mixed speech separation model, or a female-female mixed speech separation model.

[0074] In actual applications, when the labeled tag is 0, the speech separation model matched with the labeled tag is a male-male mixed speech separation model, that is, the blind speech signal is judged as a male-male mixed speech signal; when the labeled tag is 1, the speech separation model matched with the labeled tag is a male-female mixed speech separation model, that is, the blind speech signal is judged as a male-female mixed speech signal; and when the labeled tag is 2, the speech separation model matched with the labeled tag is a female-female mixed speech separation model, that is, the blind speech signal is judged as a female-female mixed speech signal.

[0075] In the gender combination type recognition stage of the unknown blind speech signal, the deep features of the blind speech signal are extracted by using the double-branch convolutional neural network fusion model, so that the deep fusion features used for recognition contain richer speech information, thereby improving the accuracy of recognizing the gender combination type of the blind speech signal; in the speech separation stage of the blind speech signal, based on the recognition result of the classifier, a speech separation model matched with the blind speech signal is selected for speech separation, so as to obtain a target speech signal with high quality, high restoration degree and high intelligibility, which is superior to the traditional speech separation method.

[0076] Embodiment 2

[0077] Based on embodiment 1, this embodiment details the specific method of extracting the deep fusion features of the preprocessed blind speech signal and the specific method of inputting the blind speech signal amplitude spectrum into a speech separation model matched with the labeled tag to obtain a target speech signal.

[0078] The deep fusion features of the preprocessed blind speech signal of this embodiment include the following steps.

[0079] S21 extracts filter bank features and mel frequency cepstral coefficient features of the preprocessed blind speech signal.

[0080] S22 inputs the filter bank features and the mel frequency cepstral coefficient features into a double-branch convolutional neural network fusion model to obtain deep mel frequency cepstral coefficient features and deep filter bank features.

[0081] In applications, the double-branch convolutional neural network fusion model includes a fusion layer and two branch convolutional neural networks. Each branch convolutional neural network is provided with an input layer, two convolutional layers, two pooling layers, and a full connection layer; wherein each pooling layer is arranged after the corresponding convolutional layer.

[0082] In actual application, the full connection layer of one branch convolutional neural network is used to integrate the mel cepstrum coefficient features after convolution and pooling, to obtain deep mel cepstrum coefficient features;

[0083] The full connection layer of another branch convolutional neural network is used to integrate the filter bank features after convolution and pooling, to obtain deep filter bank features;

[0084] S23 fuses the deep mel cepstrum coefficient features and the deep filter bank features by using a fusion layer to obtain deep fusion features

[0085] In this embodiment, the amplitude spectrum of the blind speech signal is input into the speech separation model matched with the label, and the target speech signal is obtained, including the following steps:

[0086] S41 inputs the amplitude spectrum of the blind speech signal into the speech separation model matched with the label, to obtain an estimated ratio mask of the target speech signal, for reference Figure 7 .

[0087] In application, based on the DNN / CNN mapping relationship of the speech separation model, the estimated ratio mask matrix of each frame of target speaker is obtained by using the spectrum of the mixed speech signal, and the estimated ratio mask of the target speech signal is obtained according to the estimated ratio mask matrix of each frame of target speaker, specifically including the following formula:

[0088]

[0089] In the formula, K represents the total number of speech frames, f(·) represents the DNN / CNN mapping relationship of the speech separation model, represents the estimated ratio mask matrix of the kth frame of target speaker, Y k represents the spectrum of the mixed speech signal.

[0090] S42 determines the amplitude spectrum of the target speech signal by using the estimated ratio mask of the target speech signal, specifically including the following formula:

[0091]

[0092] In the formula, is the amplitude spectrum feature function of the target speech signal, k is the time frame, f is the frequency, Y is the amplitude spectrum of the blind speech signal, is the estimated ratio mask of the target speech signal.

[0093] S43 obtains the target speech signal by using the amplitude spectrum of the target speech signal based on the principle of inverse short-time Fourier transform, to complete blind speech separation, specifically including the following formula:

[0094]

[0095] wherein, is the rth target speech signal function, and r is 1 or 2 in application. t is continuous time, ISTFT() is inverse short-time Fourier transform function, S r (k,f) is the real amplitude spectrum feature of the rth speech signal, is e exponential function, j is imaginary unit, is the phase of the blind speech signal.

[0096] Embodiment 3

[0097] Based on Embodiment 1 or 2, this embodiment details a blind speech separation method based on a double-branch convolutional neural network fusion model.

[0098] The gender combination type recognition of the blind speech signal in this embodiment is completed based on a double-branch convolutional neural network fusion model and a support vector machine; the separation of the blind speech signal in this embodiment is completed based on a speech separation model of different gender combination types.

[0099] Firstly, this embodiment constructs a double-branch convolutional neural network fusion model.

[0100] In application, the filter bank feature and the mel cepstral coefficient feature are input into the double-branch convolutional neural network fusion model, and the deep mel cepstral coefficient feature and the deep filter bank feature are obtained at the full connection layer of the branch convolutional neural network. The deep fusion feature can be obtained at the fusion layer of the double-branch convolutional neural network fusion model.

[0101] In actual application, each branch convolutional neural network includes an input layer, two convolutional layers, two pooling layers, a full connection layer, and a fusion layer arranged in sequence. Referring to Figure 3 The training process of each branch convolutional neural network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a full connection layer, and an output layer arranged in sequence.

[0102] 1.1 Network setting of convolutional layer

[0103] The convolutional layer is the core of the convolutional neural network, and the input feature of this layer is calculated by weighted summation of the input feature and the convolution kernel. The specific operation is as follows:

[0104]

[0105] wherein, is the output feature of the l1th convolutional layer, is the convolution kernel weight parameter of the l1th convolutional layer, and * is convolution operation, is the output feature of the l1-1th convolutional layer, represents the bias parameters of the l1-th layer of the convolutional layer, and ReLU() is the activation function, which can increase the nonlinearity of the convolutional neural network.

[0106] In the application, the kernel size of the first convolutional layer is 3×3, and the number of kernels is 32. The kernel size of the second convolutional layer is 3×3, and the number of kernels is 48.

[0107] In this embodiment, the convolution operation of the convolutional layer ensures that each unit in the output feature is only related to a portion of the input feature, which can be used to extract local features of the speech signal. Since the size of the convolutional kernel is much smaller than the size of the input feature, the convolution operation of the convolutional layer significantly reduces the network parameters of the convolutional neural network.

[0108] 1.2 Network settings for the pooling layer

[0109] In this embodiment, a pooling layer is set after each convolutional layer to perform feature selection on the output features of the convolutional layer.

[0110] Pooling layers typically include mean pooling and max pooling methods. In the field of speaker recognition, max pooling is preferred. The pooling layer in this embodiment can retain the most effective information of the input features through max pooling.

[0111] In the application, the pooling kernel size of the first pooling layer is 2×2, and the stride is 2. The pooling kernel size of the second pooling layer is 2×2, and the stride is 1.

[0112] 1.3 Network settings for the fully connected layer.

[0113] Since there are many convolutional and pooling kernels, the output feature dimension after the second pooling layer is very large. Therefore, a fully connected layer is set after the second pooling layer to integrate the output features after convolution and pooling.

[0114] The fully connected layer in this embodiment includes a first fully connected layer and a second fully connected layer. The first fully connected layer has 1024 neurons, and the second fully connected layer has 256 neurons.

[0115] The specific operations for each fully connected layer are as follows:

[0116]

[0117] In the formula: For the l2nd layer of the fully connected layer τ The value of each neuron, For the j-th neuron in the l2-1 layer of the fully connected layer and the j-th neuron in the l2 layer... τ The weights of each neuron, This represents the value of the j-th neuron in the l2-1-th layer of the convolutional neural network. For the l2nd layer of the fully connected layer τ The bias parameter of each neuron, where J is the total number of neurons in the fully connected layer.

[0118] 1.4 Network settings for the output layer.

[0119] The output layer contains three nodes, representing the probabilities that the blind speech signal is a male-male mixed speech signal, a male-female mixed speech signal, and a female-female mixed speech signal, respectively, as shown in the following formula:

[0120]

[0121] In the formula, Let represent the probability of the output of the i-th neuron in the output layer. In this application, each neuron in the output layer corresponds to a mixed speech signal of a gender combination type. Softmax() is the activation function of the output layer. For the fully connected layer, layer l3-1, ... τ The weights of the i-th neuron in the l3-th layer and the i-th neuron in the l3-th layer. This represents the value of the i-th neuron in the l3-1-th layer of the convolutional neural network. represents the bias parameters of the i-th neuron in the l3-th layer of the fully connected layer. i This represents the raw output score of the blind speech signal in the range (-∞, +∞). exp(o i (to use the e-index to express o) i Mapped to (0,+∞), when applied, exp(o i After normalization, the result is obtained And the highest probability is taken as the recognition result, i.e., the maximum. The corresponding mixed speech signal type is the type of blind speech signal.

[0122] In practical applications, the total number of neurons in the output layer of this embodiment is 3, therefore, for or Next, this embodiment trains a convolutional neural network, referring to... Figure 2 .

[0123] 2.1 Generate mixed speech with three gender combinations

[0124] Multiple single speech signals are acquired, and the acquired single speech signals are combined in pairs to obtain three types of mixed speech signals.

[0125] Among them, single speech signals include male single speech signals and female single speech signals; the three types of mixed speech signals include male-male mixed speech signals, male-female mixed speech signals, and female-female mixed speech signals.

[0126] In the application, the sampling rate of each monophonic signal is 16 kHz, and those skilled in the art can adjust the sampling rate of each monophonic signal according to actual needs.

[0127] 2.2 Preprocessing each mixed voice signal

[0128] Since the voice signal has the characteristic of short-time stationarity, the mixed voice signal is preprocessed before being subjected to Fast Fourier transform (FFT). Common preprocessing methods include frame division, windowing, pre-emphasis and the like, which are described in detail in Figure 4 .

[0129] 2.3 Extracting filter bank features and mel-frequency cepstrum coefficient features of each mixed voice signal

[0130] The preprocessed mixed voice signal y(t) is subjected to Fast Fourier transform to convert it to the frequency domain for analysis. The frequency spectrum Y(k,f) of the mixed voice signal is obtained by performing the following steps on the mixed voice signal: k The square of the frequency spectrum is taken to obtain the power spectrum E(k,f) of the mixed voice signal.

[0131] The actual frequency is converted into the mel frequency by using the mel-frequency filter bank. Specifically, reference is made to Figure 4 In this embodiment, the mel frequency of the mixed voice signal is obtained according to the power spectrum of the mixed voice signal, and the specific formula is as follows:

[0132] Mel(f) = 2595log 10 (1 + f / 700)

[0133] In the formula, Mel(f) is the mel frequency of the mixed voice signal, and f is the frequency.

[0134] The obtained mel frequency of the mixed voice signal is taken as the logarithm to reduce the data range and extract the filter bank (Fbank) features.

[0135] Reference is made to Figure 4 The filter bank features are further subjected to discrete cosine transform (DCT) to retain the real part of the frequency spectrum and reduce the data dimension. The first-order difference spectrum and the second-order difference spectrum of the mel-frequency cepstrum features are obtained. The MFCC features of each frame of voice signal are obtained through the above steps. Since the voice signal is continuous in time sequence, and the MFCC features are static, the first-order features or the second-order features of the mel-frequency cepstrum coefficients are used to increase the correlation between the features of the adjacent frames and obtain dynamic information.

[0136] In applications, since Convolutional Neural Networks (CNNs) perform well on samples conforming to a standard distribution, it is sometimes necessary to calculate the variance and mean for standardization. Furthermore, this embodiment normalizes the MFCC and Fbank features to better train the dual-branch CNN fusion model.

[0137] 2.4 Weights and biases of each branch convolutional neural network in training the dual-branch convolutional neural network fusion model.

[0138] The training process of the branch convolutional neural network in this embodiment includes a forward propagation stage and a backward propagation stage.

[0139] The forward propagation phase involves randomly initializing the weights and biases, calculating them layer by layer starting from the input layer, and finally obtaining the recognition result of the output layer. The backpropagation phase involves calculating the cross-entropy loss function value, using the gradient descent algorithm to constrain the loss function, and gradually optimizing the weights w and biases b of each layer of the model.

[0140] In applications, for multi-class gender combination tasks, the cross-entropy loss function is expressed as follows:

[0141]

[0142] In the formula, J1(w,b) is the cross-entropy loss function, and y i Indicates y i The labeling is performed when the type of the mixed speech signal corresponding to the output layer neuron is the same as the type of the training signal. i The label, y i =1, otherwise y i =0.

[0143] In practical applications, the update formula for the initial parameters {w,b} during backpropagation is:

[0144]

[0145]

[0146] In the formula, and These represent the weight vector and bias vector of the l-th layer of the convolutional neural network after the t-th iteration, respectively. This represents the iteration speed of the convolutional neural network.

[0147] In this embodiment, the Stochastic Gradient Descent (SGD) algorithm is used to train the model, with a learning rate of 0.01, a batch size of 16, and an iteration period of 200.

[0148] Then, a dual-branch convolutional neural network fusion model is constructed by using the trained branch convolutional neural network to obtain deep features.

[0149] 3.1 Extracting deep Mel frequency cepstral coefficient features of various mixed speech signals

[0150] The MFCC features of various mixed speech signals are input into one branch convolutional neural network of the dual-branch convolutional neural network fusion model, i.e., the first convolutional neural network, to obtain deep Mel frequency cepstral coefficient (DMFCC) features. In applications, DMFCC = (m1, m2, …, m 1024 ), and the reference Figure 1 .

[0151] 3.2 Extracting deep filter bank features of various mixed speech signals

[0152] The Fbank features of various mixed speech signals are input into the other branch convolutional neural network of the dual-branch convolutional neural network fusion model, i.e., the second convolutional neural network, to obtain deep filter bank (DFbank) features. In applications, DFbank = (f1, f2, …, f 1024 ), and the reference Figure 1 .

[0153] 3.3 Extracting deep fusion features of various mixed speech signals

[0154] The deep Mel frequency cepstral coefficient features and the deep filter bank features are serially fused in the fusion layer of the dual-branch neural network to obtain deep fusion features. In applications, the deep fusion features are represented as (m1, m2, …, m 1024 , f1, f2, …, f 1024 ).

[0155] In actual applications, the network parameters between the input layer and the first fully connected layer of one branch convolutional neural network are saved, the output features of the first fully connected layer are extracted, and 1024-dimensional deep Mel frequency cepstral coefficient features (m1, m2, …, m 1024 ) are obtained. The network parameters between the input layer and the first fully connected layer of the other branch convolutional neural network are saved, the output features of the first fully connected layer are extracted, and 1024-dimensional deep filter bank features (f1, f2, …, f 1024 ) are obtained. Then, the two kinds of deep features are serially fused to obtain 2048-dimensional deep fusion features (m1, m2, …, m 1024 , f1, f2, …, f 1024 ).

[0156] Then, the classifier is trained using the deep fusion features.

[0157] 4.1 Training support vector machine based on deep fusion features for gender combination recognition of mixed speech signals.

[0158] In the application, the classification decision function g(x) used in training the SVM is:

[0159]

[0160] where sign(·) is the sign function, y n is the label of the sample, in the application, is the weight, x is the deep fusion feature DMFCC+DFbank train of the training sample, β is the intercept of the input feature, K(x n , x) is the kernel function, the most widely used radial basis kernel function is used in the embodiment. N is the number of training samples.

[0161] where a n , n = 1, 2, …, N are obtained by solving the following optimization problem:

[0162]

[0163]

[0164] where β is obtained by the following formula:

[0165]

[0166] Then, the speech separation model applied to mixed speech signals of different gender combinations is trained.

[0167] 5.1 Training speech separation model using mixed speech signals of different gender combinations

[0168] The speech separation model is trained using male-male mixed speech signals, female-female mixed speech signals, and male-female mixed speech signals, respectively, to obtain a male-male mixed speech separation model, a male-female mixed speech separation model, or a female-female mixed speech separation model, as described in reference Figure 7 .

[0169] The speech separation model used in the embodiment is the speech separation model described in reference 1 and reference 2.

[0170] Document 1: ZHANG Xiaolei and WANG Deliang. A deep ensemble learning method for monaural speech separation[J]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2016, 24(5): 967-977. doi:10.1109 / TASLP.2016.

[0171] Document 2: T. Kounovsky and J. Malek, "Single channel speech enhancement using convolutional neural network," 2017 IEEE International Workshop of Electronics, Control, Measurement, Signals and their Application to Mechatronics (ECMSM), 2017, pp. 1-5, doi: 10.1109 / ECMSM.2017.7945915.

[0172] The training data of the speech separation model of the embodiment randomly selects two speakers, and two male speakers (M1, M2) and two female speakers (F1, F2) are randomly selected from the GRID English corpus to form two pairs.

[0173] In the application, the mixed speech signal is the time domain signal of two single speech signals added together, which belongs to single-channel speech separation:

[0174] y(t) = s1(t) + s2(t)

[0175] Wherein, s1(t) and s2(t) represent the single speech signals of two speakers respectively, and y(t) represents the mixed speech signal.

[0176] During training, the mixed speech signal is converted to the frequency domain for analysis. The input of the deep neural network is the amplitude spectrum of the mixed speech, and the training target adopts the ideal ratio mask (IRM), which reflects the energy ratio relationship between the target signal and the mixed signal. The target speech signal s rIRM of (t) is as follows:

[0177]

[0178] In the formula, M r (k,f) is an ideal ratio mask of the rth target speech signal, S r (k,f) is the real amplitude spectrum of the rth speech signal, k is a time frame, f is a frequency, ε is a parameter, and is a positive number.

[0179] A very small positive number ε is added to the denominator of the above formula to avoid the denominator being 0.

[0180] The training of the deep neural network is constrained by using a minimum mean square error loss function to minimize the error J2(w,b) between the IRM of the target speech and the estimated ratio mask (RM) :

[0181]

[0182] where K represents the total number of speech frames, and is obtained by separating the DNN / CNN mapping relationship f(·) of the speech model represents the estimated ratio mask matrix of the kth frame of the target speaker, Y k represents the spectrum of the mixed speech signal. In applications, the minimum mean square error uses the l2 norm to measure.

[0183] In the training process of the speech separation model, the structure of the deep neural network DNN of the speech separation model is 257-1024-1024-1024-257, and the values of each layer represent the number of neurons of each layer. The activation function of the first few layers uses Relu(), and the activation function of the output layer uses Sigmoid(). The batch size is 128 and the iteration period is 200 during training.

[0184] In the training process of the speech separation model, the convolution kernel size of the first convolution layer and the second convolution layer of the one-dimensional convolutional neural network CNN of the speech separation model is 1x3, and the step size is 1 and 2 respectively; the pooling kernel size of the first pooling layer and the second pooling layer is 1x3, and the step size is 1, using mean pooling; the structure of the fully connected layer is 1024-1024-257; in addition, the activation function of the first few layers uses Relu(), and the activation function of the output layer uses Sigmoid().

[0185] Finally, the blind speech signal is separated.

[0186] 6.1 Utilize the double-branch convolutional neural network fusion model to extract the deep fusion features of the unknown blind speech signal.

[0187] 6.2 Based on the trained support vector machine, the gender combination type of the blind speech signal is identified by using the deep fusion feature.

[0188] 6.3 According to the identification result, the type of the speech separation model is determined, and the blind speech signal amplitude spectrum is input into the determined speech separation model to realize blind speech separation.

[0189] In the speech separation stage, the blind speech signal amplitude spectrum is input into the matched speech separation model to obtain the estimated ratio mask of the target speech signal According to the formula The Hadamard product of the estimated target speech signal amplitude spectrum feature According to the formula:

[0190]

[0191] The The phase of the blind speech signal is combined to write a complex signal, and the estimated speech time domain signal, i.e., the target speech signal, is reconstructed by using the inverse short-time Fourier transform (ISTFT) The separation of the unknown blind speech signal is completed.

[0192] Embodiment 4

[0193] On the basis of any one of embodiments 1-3, this embodiment verifies the performance of the blind speech separation method of the application by using the GRID English corpus. The corpus contains 34 speakers (18 males and 16 females), each speaker has 1000 sentences, and each sentence is about 1-2 seconds. The training data of the speech separation model of the application randomly selects two speakers, and randomly selects two male speakers (M1, M2) and two female speakers (F1, F2) from the corpus to form two two-person combinations for training and testing. The training sentences consist of the first 500 sentences, and the remaining sentences are used as the verification set and the test set. 100 sentences are selected as the verification set, and 60 sentences are selected as the test set. Then, the mixed speech is generated, the male-male combination is represented as M-M, the female-female combination is represented as F-F, and the male-female combination is generated by combining the speech of two male speakers and two female speakers, represented as M-F.

[0194] The accuracy of the identification result is used to evaluate the performance of the blind speech gender combination identification.

[0195] To verify the separability of the gender combination and the effectiveness of the features proposed in the present application, the distributions of the MFCC, Fbank, DMFCC, DFbank and DMFCC+DFbank features are visualized by t-distributed Stochastic Neighbor Embedding (t-SNE) dimensionality reduction, with reference to Figure 5 and Figure 6 .

[0196] It can be seen from Figure 5 that the distributions of the MFCC features and the Fbank features of the three types of mixed speech signals are relatively disordered, and therefore, it is not easy to distinguish the types of the mixed speech signals according to the distributions of the MFCC features and the Fbank features; it can be seen from Figure 6 that the DMFCC features and the DFbank features obtained by the multi-layer convolution and pooling of the CNN for the three types of mixed speech signals have a certain degree of distinguishability, and therefore, it is feasible to distinguish the types of the mixed speech signals according to the distributions of the DMFCC features and the DFbank features.

[0197] It can be seen from (a), (b) and (c) of Figure 6 that the distributions of the deep fusion features of the three types of mixed speech signals have the greatest distinguishability, and therefore, the deep fusion features have the greatest gender combination distinguishability.

[0198] To verify the effectiveness of the DMFCC features, the DFbank features and the deep fusion features extracted by the CNN in the present application, hereinafter referred to as deep features, the MFCC features, the Fbank features, the shallow fusion features and the deep features are respectively used as the inputs of the SVM for gender combination classification, and the influence of each feature on the accuracy of the recognition result, hereinafter referred to as recognition rate, is analyzed, as shown in Table 1, in which DMFCC+DFbank represents the deep fusion features, and MFCC+Fbank represents the shallow fusion features.

[0199] Table 1 Recognition rate (%) of different features under SVM

[0200]

[0201] It can be seen from Table 1 that the average recognition rate of the gender combination classification is increased by 20.55% for DMFCC compared with MFCC, and the average recognition rate is increased by 21.11% for DFbank compared with Fbank. It can be seen that the deep features extracted by the dual-branch convolutional neural network fusion model can better represent the gender combination information compared with the shallow features. In addition, the gender combination classification effect of DMFCC+DFbank is the best, which is increased by 19.44% compared with MFCC+Fbank, increased by 3.34% compared with DMFCC, and increased by 1.67% compared with DFbank. It can be seen that directly classifying the calculated shallow features has poor recognition effect and is not conducive to subsequent speech separation; and DMFCC+DFbank contains more deep gender combination information, which is helpful for subsequent accurate speech separation.

[0202] Table 2 Recognition rate (%) of deep features under different classifiers

[0203]

[0204] As shown in Table 2, in order to verify the influence of different classifiers on the accuracy of the recognition result, the SVM classifier and the Softmax classifier are used for gender combination classification of the mixed speech signal, and the influence of different classifiers on the recognition rate of each feature is analyzed, which is shown in Table 2.

[0205] It can be seen from Table 2 that based on the features of DMFCC, DFbank and DMFCC+DFbank, the average recognition rate of the SVM classifier is increased by 0.58%, 0.55% and 2.78% respectively compared with the Softmax classifier, and it can be seen that the recognition rate of the SVM classifier is better than the traditional Softmax classifier based on the extracted deep features.

[0206] In actual application, the dual-branch convolutional neural network fusion model is used in combination with the SVM classifier for gender combination classification of blind speech signals.

[0207] In addition, the speech quality perceptual evaluation (PESQ), signal distortion ratio (SDR) and short-time objective intelligibility (STOI) are used to measure the quality, restoration and intelligibility of the obtained target speech signal, and the numerical value of the index is positively correlated with the speech separation performance.

[0208] In order to verify the effectiveness of the blind speech signal separation method proposed in the application, the mixed speech signals of known gender types are subjected to matched, cross test, baseline and blind separation processing, and the PESQ and STOI of the target speech signals obtained by the four processing methods are analyzed.

[0209] In application, the speech separation model matched with the known mixed speech signal is used for speech separation, which is recorded as matched; the speech separation model not matched with the known mixed speech signal is used for speech separation, which is recorded as cross test; the traditional speech separation method is used for speech separation of the known mixed speech signal, which is recorded as baseline; the blind speech separation method recorded in any one of embodiments 1-3 is used for speech separation of the mixed speech signal, which is recorded as blind separation.

[0210] The cross test is specifically: if the gender type of the known mixed speech signal is male-male mixed, a female-female mixed speech separation model is used; if the gender type of the known mixed speech signal is female-female mixed, a male-male mixed speech separation model is used; if the gender type of the known mixed speech signal is male-female mixed, a male-male mixed speech separation model is used.

[0211] In actual application, the network of the speech separation model is DNN or CNN.

[0212] Reference Figure 8 It can be seen that for male-female mixed speech signals, the speech separation effect of blind speech is very close to that of matched speech:

[0213] When the network of the speech separation model is DNN, the PESQ, STOI and SDR indexes of blind separation and matched are only different by 0.029, 0.006 and 0.058 dB respectively; when the network of the speech separation model is CNN, the PESQ, STOI and SDR indexes of blind separation and matched are only different by 0.049, 0.001 and 0.056 dB respectively, which shows that the speech separation result of the method of the application is reliable.

[0214] In addition, for male-female mixed speech signals, the speech separation effect of blind speech has a greater advantage compared with that of cross test:

[0215] When the network of the speech separation model is DNN, the PESQ, STOI and SDR indexes of blind separation are 0.584, 0.262 and 4.163 dB higher than those of cross test respectively, and when the network of the speech separation model is CNN, the PESQ, STOI and SDR indexes of blind separation are 0.239, 0.079 and 4.133 dB higher than those of cross test respectively. It can be seen that the importance of selecting a matched speech separation model.

[0216] In addition, compared with the speech separation effect of the baseline, the speech separation effect of the blind speech is more advantageous for mixed male and female speech signals:

[0217] When the network of the speech separation model is a DNN, the PESQ, STOI and SDR indexes of the blind separation are respectively 0.279, 0.002 and 0.980 dB higher than those of the baseline; when the network of the speech separation model is a CNN, the PESQ, STOI and SDR indexes of the blind separation are respectively 0.051, 0.011 and 0.324 dB higher than those of the baseline; it can be seen that the target speech signal obtained by the speech separation method has higher quality, restoration and intelligibility.

[0218] Reference Figure 10 It can be seen that, compared with mixed male speech signals, the blind speech separation method of the application is more suitable for mixed male and female speech signals and mixed female speech signals.

[0219] In combination Figure 9 And Figure 10 It can be seen that, compared with mixed speech signals of a single gender, the blind speech separation method of the application is more suitable for mixed speech signals of two genders, i.e., mixed male and female speech signals.

[0220] Embodiment 5

[0221] The embodiment provides a blind speech separation system of a double-branch convolutional neural network fusion model, comprising:

[0222] A collection module is configured to acquire a blind speech signal and a blind speech signal amplitude spectrum, wherein the blind speech signal is a double mixed speech signal.

[0223] A deep feature extraction module is configured to pre-process the acquired blind speech signal and extract deep fusion features of the pre-processed blind speech signal.

[0224] A marking module is configured to mark the blind speech signal by using a classifier according to the deep fusion features of the blind speech signal.

[0225] A separation module is configured to input the blind speech signal amplitude spectrum into a speech separation model matched with the marking label, to obtain a target speech signal and complete blind speech separation.

[0226] The specific function implementation of each functional module is referred to the related content in the methods of embodiments 1-4.

[0227] In summary, for the problem of misjudgment in directly identifying the gender combination of mixed speech signals using a single acoustic feature, the application makes full use of the excellent ability of the double-branch convolutional neural network fusion model to extract deep features. In the gender combination classification stage of unknown blind speech signals, the deep features of filter bank features (Fbank) and Mel frequency cepstral coefficient features (MFCC) are extracted, so that the speech features used for identification contain more rich speech information, to improve the identification performance, and based on the deep fusion features and the classifier, high-accuracy discrimination of the gender combination of blind speech signals is realized; in the speech separation stage, based on the identification result of the classifier, the speech separation model matched with the blind speech signal is selected for speech separation, which can obtain high-quality, high-recovery and high-intelligibility target speech signals superior to traditional speech separation methods.

[0228] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0229] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0230] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0231] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are generated to realize the computer-implemented processes, and the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0232] The embodiments of the present application are described above with reference to the drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, which are all within the protection of the present application.

Claims

1. A blind speech separation method based on a dual-branch convolutional neural network fusion model, characterized in that, The method comprises the following steps: obtaining a blind speech signal and a blind speech signal amplitude spectrum, the blind speech signal being a double mixed speech signal; preprocessing the obtained blind speech signal, and inputting the preprocessed blind speech signal into a double-branch convolutional neural network fusion model to obtain a deep fusion feature of the blind speech signal; labeling the blind speech signal according to the deep fusion feature of the blind speech signal by using a classifier; inputting the blind speech signal amplitude spectrum into a speech separation model matched with the labeled label to obtain a target speech signal, and completing blind speech separation.

2. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 1, characterized in that, The speech separation model is a male-male mixed speech separation model, a male-female mixed speech separation model, or a female-female mixed speech separation model.

3. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 1, characterized in that, The double-branch convolutional neural network fusion model comprises a fusion layer and two branch convolutional neural networks; Each branch convolutional neural network is provided with an input layer, two convolutional layers, two pooling layers, and a full connection layer. Among them, each pooling layer is arranged after the corresponding convolutional layer.

4. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 3, characterized in that, The inputting of the preprocessed blind speech signal into the double-branch convolutional neural network fusion model to obtain the deep fusion feature of the blind speech signal comprises: extracting filter bank features and mel-frequency cepstral coefficient features of the preprocessed blind speech signal; inputting the filter bank features and the mel-frequency cepstral coefficient features into the double-branch convolutional neural network fusion model; integrating the convolutional and pooled mel-frequency cepstral coefficient features by using the full connection layer of one branch convolutional neural network to obtain a deep mel-frequency cepstral coefficient feature; integrating the convolutional and pooled filter bank features by using the full connection layer of another branch convolutional neural network to obtain a deep filter bank feature; fusing the deep mel-frequency cepstral coefficient feature and the deep filter bank feature by using the fusion layer to obtain a deep fusion feature.

5. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 1, characterized in that, The classifier is a support vector machine, the deep fusion feature is inputted into the trained support vector machine, and the support vector machine is used to label the blind speech signal to detect the type of the gender combination of the blind speech signal.

6. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 1, characterized in that, The inputting of the blind speech signal amplitude spectrum into the speech separation model matched with the labeled label comprises: when the labeled label is 0, the speech separation model matched with the labeled label is a male-male mixed speech separation model; when the labeled label is 1, the speech separation model matched with the labeled label is a male-female mixed speech separation model; when the labeled label is 2, the speech separation model matched with the labeled label is a female-female mixed speech separation model.

7. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 1, characterized in that, The inputting of the blind speech signal amplitude spectrum into the speech separation model matched with the labeled label to obtain the target speech signal comprises: inputting the blind speech signal amplitude spectrum into the speech separation model matched with the labeled label to obtain an estimated ratio mask of the target speech signal; determining a target speech signal amplitude spectrum by using the estimated ratio mask of the target speech signal; obtaining the target speech signal by using the target speech signal amplitude spectrum based on the principle of inverse short-time Fourier transform, and completing blind speech separation.

8. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 7, characterized in that, The determination of the target speech signal amplitude spectrum by using the estimated ratio mask of the target speech signal comprises the following formula: wherein is the target speech signal amplitude spectrum feature function, k is the time frame, f is the frequency, Y is the blind speech signal amplitude spectrum, is the estimated ratio mask of the target speech signal.

9. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 7, characterized in that, The obtaining of the predicted target speech signal by using the target speech signal amplitude spectrum based on the principle of inverse short-time Fourier transform comprises the following formula: wherein is the rth predicted target speech signal function, t is continuous time, ISTFT(·) is an inverse short-time Fourier transform function, S r (k,f) is the rth speech signal's true amplitude spectrum feature, is the e exponential function, j is the imaginary unit, is the blind speech signal's phase.

10. The blind speech separation method based on the dual-branch convolutional neural network fusion model according to claim 9, characterized in that, The r is 1 or 2.

Images