A small sample voice separation method based on data generation

Abstract

The present application relates to a kind of small sample voice separation method based on data generation, belong to voice separation field, comprising the following steps: S1: collection and construction are separated voice data set and target speaker clean voice data set;S2: data set is preprocessed;S3: using the voice matching model based on LSTM, each voice in the preprocessed voice data set to be separated is carried out frame level target speaker matching, according to the matching result, voice segment is judged as: not containing target speaker, only containing one target speaker, containing multiple target speakers;S4: using the voice generation model based on GAN containing multiple target speakers, expands the voice segment to be separated;S5: construct and train the voice separation model based on TCN, and the voice data of single target speaker is obtained by separating the voice containing multiple target speakers to be separated, finally, the complete voice of single speaker is spliced.

Description

Technical Field

[0001] This invention belongs to the field of speech separation and relates to a few-sample speech separation method based on data generation. Background Technology

[0002] Speech separation is a problem stemming from the well-known "cocktail party effect," where humans can effectively select and track a particular speaker's voice amidst a complex mixture of sounds. Research on speech separation has significant theoretical and practical value in speech communication, acoustic target detection, and sound signal enhancement. Speech separation can be broadly divided into two directions: first, separating multiple sound sources within a speech signal; and second, separating a single sound source from noise and other interference within a speech signal. Separating multiple sound sources has always been a challenging problem in the field due to the greater number of factors involved. Current research on separating multiple sound sources still faces the following issues:

[0003] (1) Insufficient speech samples for model training. When we want to separate the speech of a specific speaker, it is difficult to collect a large number of speech samples of that target speaker, which affects the training of the model and makes it difficult for the model to learn the complete speech features of the target speaker.

[0004] (2) Speech separation of multiple target speakers is difficult. It is easy to separate only one target speaker by classifying all other sounds as noise. However, if it is necessary to separate the speech of multiple target speakers, the above method is less effective. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a few-sample speech separation method based on data generation.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A few-sample speech separation method based on data generation includes the following steps:

[0008] S1: Collect multiple segments of speech data to be separated and clean speech data of the target speaker, and construct the speech dataset to be separated and the clean speech small sample dataset of the target speaker respectively;

[0009] S2: Preprocess the speech dataset to be separated and the clean speech dataset of the target speaker;

[0010] S3: Use an LSTM-based speech matching model to perform frame-level target speaker matching on each speech segment in the preprocessed speech dataset to be separated. Based on the matching results, the speech segments are judged as: having no target speaker, having only one target speaker, or having multiple target speakers.

[0011] S4: Using a GAN-based speech generation model containing multiple target speakers, the speech data to be separated containing multiple target speakers is processed to expand the speech segment dataset to be separated.

[0012] S5: Construct and train a TCN-based speech separation model to separate the speech of multiple target speakers into speech data of a single target speaker, and finally splice the complete speech of a single speaker.

[0013] Furthermore, step S1 specifically includes the following steps: When collecting data, a recording device is used to record the speaker and the real scene in which they are located, to obtain the speech dataset to be separated, V0 = {v1, v2, ... v...} i , ..., v n}i∈[1,n], where n represents the total number of speech segments in the speech dataset to be separated, v i Let ci represent the i-th segment of speech to be separated, and the duration of each segment is Time minutes. Collect clean speech signals from all target speakers without noise interference to obtain a small sample dataset of clean speech from the target speakers: Clean = {c1, c2, ..., ci}. i c m}j∈[1,m], where m represents the number of target speakers, c j This represents the clean speech of the j-th target speaker, with each clean speech segment lasting in Time minutes.

[0014] Furthermore, the preprocessing in step S2 includes:

[0015] S21: Use a threshold method combining short-time energy and zero-crossing rate to perform endpoint detection on the acquired speech to be separated and remove blank speech parts;

[0016] S22: The speech segment to be separated obtained in step S21 is subjected to noise reduction processing using a deep learning model based on frequency domain masking.

[0017] Furthermore, step S3 specifically includes the following steps:

[0018] S31: Divide the speech segment to be separated and the clean speech of the target speaker into p frames, and extract the Mel-spectral features X of the speech segment to be separated in each frame. a ={x 11 x 12 , ..., x 1k , ..., x 1p , ..., x lk , ..., x np}, where x lkThis represents the feature vector extracted from the k-th frame of the l-th segment of speech to be separated, where l∈[1,n], k∈[1,p]; and the Mel-spectral features Y of the clean speech of the target speaker in each frame are extracted. b ={y 11 y 12 , ..., y 1k , ..., y 1p , ..., y jk , ..., y mp}, where y jk This represents the feature vector extracted from the k-th frame of the clean speech of the j-th target speaker.

[0019] S32: Construct an LSTM-based speech matching model, and integrate Mel-spectral features X a With Y b Each frame of speech features in each segment of speech to be separated is arranged in a row from top to bottom in frame order; the clean speech features of each frame of the target speaker in each segment are also arranged in a row from top to bottom in frame order, forming a batch. The Mel-spectral features in the batch are processed through three layers of LSTM to obtain a feature vector containing the input speech features. A linear mapping layer is then added after the last LSTM layer to map the output of the last LSTM layer into a low-dimensional embedding feature vector. Then, the similarity between the embedding feature vector of each frame of speech to be separated and the embedding feature of the center vector of the clean speech of each target speaker is calculated to obtain a similarity matrix with dimensions n×pm. The average value of all elements in the similarity matrix is ​​taken as the threshold to identify the target speaker who is speaking in the segment of speech to be separated.

[0020] S33: Divide the speech segments to be separated into speech segments without a target speaker, with one target speaker, and with multiple target speakers; use a speech matrix of dimension n×p to record the number of target speakers speaking in each frame of the similarity matrix; based on the statistical results of the speech matrix, select speech frames in each speech segment to be separated where the number of current speakers is equal to 0, 1, or greater than or equal to 2, retain only speech frames where the number of current target speakers is greater than or equal to 2, and merge the retained speech frames to obtain speech segments M = {M1, M2, ..., M...} of multiple target speakers in the speech segment to be separated. n Further speech separation was performed on it.

[0021] Furthermore, step S32 specifically includes the following steps:

[0022] S321: The speech batch to be separated consists of n×p Mel spectral features, including segments of speech to be separated. Each segment is divided into p frames, and the features of each frame are X. a Corresponding to the characteristic elements within;

[0023] The speaker clean speech batch consists of m×p Mel-spectral feature vectors, containing m segments of the target speaker clean speech. Each segment is divided into p frames, and the features of each frame are related to Y. b Corresponding to the characteristic elements within;

[0024] S322:X a After processing with three layers of LSTM, the feature vector of each frame of speech is obtained. Finally, a linear mapping layer is used to reduce the dimensionality of the feature vector, resulting in f(x). lk ;W), where x lk Let f be the speech signal of the kth frame of the l-th segment of speech to be separated, f be the linear mapping function after the last LSTM layer, and W be the parameters of the function, represented by a one-dimensional vector.

[0025] S323: Define the embedding feature vector as f(x) lk Normalizing W using the L2 norm, the embedding feature vector of the k-th frame of the l-th segment of the speech to be separated is represented as e. l，k :

[0026]

[0027] Where ||f(x) lk ;W)||2 represents the model output f(x) lk The L2 norm of W);

[0028] S324: Y b After model processing, the embedding feature vector e of the clean speech of the target speaker is obtained. j，k Let represent the embedding feature vector of the clean speech in frame k for the j-th target speaker; the average of the embedding feature vectors of the clean speech signal in frames p is used to obtain the speaker's center feature vector. j Let represent the center feature vector of the clean speech signal of the j-th target speaker:

[0029]

[0030] S325: Calculate each of X a The obtained e l，k With each target speaker's clean speech center feature vector (Centre) j cosine similarity S l，k，j :

[0031] S l，k，j =w·cos(e l，k Centre j )+b

[0032] Where w and b are the weights and biases that the neural network can learn, respectively, and the calculation results will form a similarity matrix with dimensions n×pm;

[0033] S326: Calculate the average value of all elements in the similarity matrix and set it as the speaker determination threshold. When the matrix element value is greater than or equal to the threshold, set the frame to 1 and determine that the corresponding target speaker is speaking in the frame; otherwise, set it to 0 and determine that the speaker is not speaking in the frame.

[0034] Furthermore, step S326 involves calculating each of X... a The obtained e l，k With each target speaker's clean speech center feature vector (Centre) j cosine similarity S l，k，j The specific process is as follows:

[0035] By X a The obtained e 1，1 Calculate the cosine similarity with Centre1, and use the result as the first element of the similarity matrix. The element in this row is e. 1，1 The cosine similarity calculation results with all Centres; the next line is e. 1，2 The cosine similarity calculation results for all Centres are used, and so on, until e is calculated. 1，p Cosine similarity results with all Centres;

[0036] Next, we calculate e. 2，1 The cosine similarity with all Centres is calculated, and the result is used as the first element of the (p+1)th row of the similarity matrix, where the element of this row is e. 2，1 The cosine similarity is calculated with all Centres; and so on, up to e. n，p The cosine similarity with all Centres yields a similarity matrix of dimension n×pm.

[0037] Furthermore, step S4 specifically includes the following steps:

[0038] S41: Extract Mel-spectral features H = {h1, h2, ..., h...} from the speech segment M to be separated, which contains multiple target speakers. v , ...h n}, h v Let v represent the v-th segment of speech to be separated, where v∈[1,n];

[0039] S42: Construct a GAN-based speech generation model, consisting of a generator and a discriminator;

[0040] The Mel spectral features H containing the speech of multiple target speakers are used as the input to the generator. After passing through a convolutional layer, they are fed into an upsampled network block. Each upsampled network block contains nested residual blocks. Finally, after passing through a convolutional layer, the speech signal output corresponding to the input features is obtained.

[0041] The discriminator employs a multi-scale architecture to discriminate the original audio. It also performs frequency reduction processing on the original audio using average pooling before inputting it into the next discriminator for discrimination. The speech signal generated by the generator and the original speech signal are input into the discriminator D, passed through a convolutional layer, and then fed into a downsampling network block. There are four downsampling network blocks. Finally, the discriminator passes through two convolutional layers to obtain the discrimination result. By continuously training the generator and discriminator, a speech generation model that can generate speech from multiple target speakers is obtained.

[0042] S43: Input the speech segment M to be expanded into the trained speech generation model to obtain n segments of generated speech to be separated, each containing multiple target speakers. Merge these segments with the original speech to be separated to obtain a new speech segment Data = {data1, data2, ..., data...} n , ...data 2n}

[0043] Furthermore, the generator has four upsampling network blocks, namely 8x, 8x, 2x, and 2x. After each layer of upsampling, a residual stack with dilated convolution is added. Each residual stack has three layers, and the dilation coefficient of each layer is 1, 3, and 9. The generator processes the one-dimensional feature H containing multiple target speakers into a speech signal containing multiple target speakers through four layers of upsampling and convolution.

[0044] The discriminator consists of three sub-discriminators D1, D2, and D3. The sub-discriminators have the same network structure and operate at different sampling frequencies. The input of D1 is the speech at the normal sampling frequency, the input of D2 is the speech obtained by downsampling once, and the input of D3 is the speech obtained by downsampling twice.

[0045] Furthermore, step S5 specifically includes the following steps:

[0046] S51: Generate a speech signal with silence segments using Praat, and splice it to the end of the speech signal to be separated that contains multiple target speakers, so that all the speech signals to be separated that contain multiple target speakers have the same duration.

[0047] S52: Construct a speech separation model based on TCN, consisting of an encoder, a separator, and a decoder;

[0048] The encoder extracts features from the input waveform of the speech to be separated, which contains multiple target speakers, through a 1*1 convolution, and transforms the speech waveform into a feature matrix.

[0049] The separator feeds the feature matrix obtained from the encoder into a TCN-based temporal convolutional network to learn the feature matrix of each speaker. It uses a 1*1 convolution to first increase the number of channels, distinguish different speakers by different channel dimensions, and then divides the feature matrix into the corresponding speaker's feature matrix.

[0050] The feature matrix obtained by the separator is fed into the decoder, and the corresponding speech signal is obtained by inverse transformation, thus realizing the separation of multiple target speakers;

[0051] S53: Fuse the clean speech of the target speaker matched in each segment of speech to be separated in step S3 to obtain speech containing multiple target speakers, thereby obtaining the training speech data of the separation model; perform iterative training on the speech separation model according to the signal-to-noise ratio evaluation value, and determine that the speech separation model training is completed when the signal-to-noise ratio evaluation value reaches the maximum value, thus obtaining the trained speech separation model.

[0052] S54: Input the speech waveforms to be separated, which contain multiple target speakers, into the trained speech separation model, and output multiple speech signals containing a single speaker to achieve speech separation of multiple target speakers.

[0053] Furthermore, in step S52, the input of the encoder is the waveform of the speech to be separated from multiple target speakers in the time domain. The waveform is a one-dimensional vector with dimension I, where I = Time * p. The encoder transforms the one-dimensional vector into a two-dimensional feature matrix of 256 × P through a one-dimensional convolution Conv1D(1,256,20, stride = 10, padding = 0), where P = (I - 20) / 10 + 1.

[0054] In the separator, the obtained 256×P feature matrix is ​​input into the TCN temporal convolutional network composed of stacked CNN networks. The feature matrix of each speaker is learned, and the number of channels is expanded by convolution. Then, different channels are distinguished for different speakers, and then the system is divided into multiple parts according to the number of speakers.

[0055] Finally, the feature matrices of different speakers learned by the separator are fed into the decoder for decoding to obtain the speech waveform of the corresponding speaker. The 256×P feature matrix is ​​transformed into a one-dimensional vector I with the same scale as the input.

[0056] The beneficial effects of this invention are as follows: Addressing the problem of high model performance requirements and long training time when directly performing speech separation on entire audio segments, this invention performs frame-level speech matching on the speech to be separated before inputting it into the speech separation model. This categorizes the speech into three types: speech without a speaker, speech with only one speaker, and speech with multiple speakers. By first separating the speech without a speaker and speech with only one speaker, further speech separation is only performed on the speech containing multiple speakers, thus saving performance and time. Simultaneously, model training often suffers from insufficient existing speech data, failing to provide enough data for the model and resulting in incomplete learning of the speech features of different speakers. This invention expands the speech data to include multiple target speakers, enabling the speech separation model to learn the individual speech features of each target speaker when multiple target speakers are present, thereby achieving better speech separation.

[0057] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0058] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0059] Figure 1 A schematic diagram of the overall process of the data-generated few-sample speech separation method described in this invention;

[0060] Figure 2 The structure diagram of the speech matching model described in this invention;

[0061] Figure 3 The structure diagram of the speech generation model for generating multiple target speaker voices described in this invention;

[0062] Figure 4 The structural diagram of the speech separation model for separating multiple target speakers described in this invention. Detailed Implementation

[0063] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0064] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0065] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0066] This invention proposes a speech separation method for extracting the target speaker's speech in a multi-person dialogue scenario. The implementation process of this method is as follows. Figure 1 As shown. The present invention includes the following steps:

[0067] S1: Collect n segments of speech data to be separated and m clean speech data from target speakers. Construct a speech dataset to be separated and a small sample dataset of clean speech from the target speakers for each collected speech segment. Details are as follows:

[0068] During data collection, recording equipment is used to record the speaker and the real-world scene, resulting in a speech dataset V0 = {v1, v2, ... v} to be separated. i , ..., v n}i∈[1,n], where n represents the total number of speech segments in the speech dataset to be separated, v iLet $C_i$ represent the $i$-th segment of speech to be separated, and each segment has a duration of 1 minute. A small sample dataset of clean speech from all target speakers, free from noise interference, is obtained by collecting clean speech signals from all target speakers. The dataset is named Clean = {c1, c2, ..., c_i}. i c m}j∈[1,m], where m represents the number of target speakers, c j Let m represent the clean speech of the j-th target speaker, with each clean speech segment lasting 1 minute. In this example, m = 10, meaning the clean speech small sample dataset contains clean speech from 10 target speakers.

[0069] S2: Preprocessing of the decoupled speech dataset and the clean speech dataset of the target speaker. A deep learning model is used to perform endpoint detection and noise reduction on the decoupled speech dataset. Details are as follows:

[0070] S21: Use a threshold method combining short-time energy and zero-crossing rate to perform endpoint detection on the speech to be separated collected in S11 and remove blank speech parts.

[0071] S22: The speech segment to be separated obtained in S21 is denoised using a deep learning model based on frequency domain masking (such as RNNoise).

[0072] The specific content of S21 is as follows:

[0073] S211: Calculate the short-time energy and short-time average zero-crossing rate of the speech signal to be separated, and calculate the ratio of energy to zero-crossing rate (energy-to-zero ratio). The formulas for calculating short-time energy and zero-crossing rate are as follows:

[0074]

[0075] Where E is the short-time energy value, S(y) is the speech signal, i is the number of frames, y is the number of sampling points of the speech signal, and Y is the window length.

[0076]

[0077] Where Z is the zero-crossing rate, and sgn[] is a function for determining the sign, defined as follows:

[0078]

[0079] Plot the zero-energy ratio curve based on the calculation results, with time on the horizontal axis and zero-energy ratio on the vertical axis.

[0080] S212: Select a relatively low threshold D based on the speech energy-to-zero ratio curve. In this example, D = 15, as the energy-to-zero ratio of the speech signal is mostly above this threshold. Find all intersections between the energy-to-zero ratio curve and the threshold D. The segments between any two adjacent intersections are the detected speech segments.

[0081] S213: The threshold needs to be determined and adjusted based on the characteristics of the speech signal to be separated. The energy change characteristics of the initial segment of the speech are also different, with a larger change at the beginning and a slower change at the end. Therefore, before determining the final starting point, it is necessary to collect 10 frames before and after the start and calculate their short-time energy and average zero-crossing rate as the basis for determining the starting point.

[0082] The specific content of S22 is as follows:

[0083] S221: The noisy speech signal undergoes a short-time Fourier transform, converting the speech signal from the time domain to frequency domain features.

[0084] S222: Frequency domain signal features are input into a deep learning prediction model. The model multiplies each frequency point in the frequency domain by a coefficient less than or equal to 1 to suppress noise. The model learns the coefficient to be multiplied at each frequency point based on the energy changes of the frequency points. Finally, the model predicts and outputs the frequency domain mask of the speech segment.

[0085] S223: Multiply the frequency domain signal by the frequency domain mask to obtain the noise-reduced frequency domain signal.

[0086] S224: The denoised frequency domain signal is transformed by inverse short-time Fourier transform to obtain the denoised speech signal in the time domain.

[0087] S3: Utilize an LSTM-based deep learning model to perform frame-level target speaker matching on each segment of the preprocessed speech dataset to be separated. Based on the matching results, determine the number of target speakers in the segment and categorize the speech segments into three types: no target speaker, one target speaker, and multiple target speakers. Continue speech separation for segments containing multiple target speakers. Details are as follows:

[0088] S31: Divide the speech to be separated and the clean speech of the target speaker into frames, each divided into p = 600 frames. Extract the Mel-spectral features X of the speech to be separated from each frame. a ={x 11 x 12 , ..., x 1k , ..., x 1p , ..., x lk , ..., x np}, where x lk Let Y represent the feature vector extracted from the k-th frame of the l-th segment of the speech to be separated. Similarly, extract the feature Y of the clean speech of the target speaker in each frame. b ={y 11 y 12 , ..., y 1k , ..., y 1p, ..., y jk , ..., y mp}, where y jk This represents the feature vector extracted from the k-th frame of the clean speech of the j-th target speaker.

[0089] S32: Construct an LSTM-based speech matching model. Process S31 to obtain Mel-spectral features X containing multiple target speakers and their clean speech. a With Y b according to Figure 2 The sequence is as follows: each frame of speech features in each segment of speech to be separated is arranged in a row, from top to bottom in frame order; each frame of clean speech features of each target speaker is arranged in a row, from top to bottom in frame order, forming a batch. The Mel-spectral features in the batch are processed through three layers of LSTM to obtain a feature vector containing the input speech features. A linear mapping layer is added after the last LSTM layer to map the output of the last LSTM layer into a low-dimensional embedding feature vector. Then, the similarity between the embedding feature vector of each frame of speech to be separated and the embedding feature of the center vector of the clean speech of each target speaker is calculated to obtain a similarity matrix with dimensions n×pm. The average value of all elements in the similarity matrix is ​​taken as a threshold to identify the target speaker speaking in the segment of speech to be separated.

[0090] The speech to be separated, btach, consists of n×p Mel-spectral feature vectors, containing n segments of speech to be separated. Each segment is divided into p frames, and the features of each frame are related to X. a Correspondingly, the speaker's clean speech batch consists of m×p Mel-spectral feature vectors, containing m segments of the target speaker's clean speech. Each segment is divided into p frames, and the features of each frame correspond to Y. b Corresponding. X a After processing with three layers of LSTM, the feature vector of each frame of speech is obtained. Finally, a linear mapping layer is used to reduce the dimensionality of the feature vector. The output of the entire network is f(x). lk ;W), where x lk Let f be the speech signal of the k-th frame of the l-th segment of the speech to be separated, f be the linear mapping function after the last LSTM layer, and W be the parameters of this function, represented by a one-dimensional vector. The embedding feature vector is defined as the L2 norm normalized output of the network, and the embedding feature vector of the k-th frame of the l-th segment of the speech to be separated is represented as e. l，k , ||f(x lk ;W)||2 represents the model output f(x) lk The L2 norm of W). Similarly, Y bThe above model processing yields the embedding feature vector e of the clean speech of the target speaker. j，k Let represent the embedding feature vector of the clean speech in the k-th frame for the j-th target speaker. The center feature vector of the speaker is obtained by averaging the embedding feature vectors of the clean speech signals in the p-th frames. j Let represent the center feature vector of the clean speech signal of the j-th target speaker. Calculate each of X a The obtained e l，k With each target speaker's clean speech center feature vector (Centre) j cosine similarity S l，k，j ,S l，k，j =w·cos(e l，k Centre j )+b, where w and b are the weights and biases that the neural network can learn, respectively. The calculation results will form a similarity matrix of dimension n×pm. The specific process is as follows: From X a The obtained e 1，1 Calculate the cosine similarity with Centre1, and use the result as the first element of the similarity matrix. The element in this row is e. 1，1 The cosine similarity calculation results with all Centres; the next line is e. 1，2 The cosine similarity calculation results for all Centres are used, and so on, until e is calculated. 1，p The cosine similarity results with all Centres are then calculated. Next, e is calculated. 2，1 The cosine similarity with all Centres is calculated, and the result is used as the first element of the (p+1)th row of the similarity matrix, where the element of this row is e. 2，1 The cosine similarity is calculated with all Centres; and so on, up to e. n，p The cosine similarity with all centers yields an n×pm similarity matrix. The average of all elements in the similarity matrix is ​​calculated and set as the speaker determination threshold. When the threshold is greater than or equal to the matrix element value, the frame is set to 1, indicating that the corresponding target speaker is speaking in that frame; otherwise, it is set to 0, indicating that the speaker is not speaking.

[0091] S33: Divide the speech segments to be separated into speech segments with no target speaker, with one target speaker, and with multiple target speakers. Use an n×p speech matrix to record the number of target speakers speaking in each frame of the similarity matrix. Let num be the number of segments in the first frame of the first speech segment to be separated whose similarity to m speakers is greater than a threshold. 1，1 , as the first element of the speech matrix; the number of speakers in the k-th frame of the l-th segment of speech to be separated is num.l，k The k-th element in the l-th row of the speech matrix is ​​used as the basis for further analysis. This process continues until the p-th frame of the n-th segment of speech to be separated, forming an n×p speech matrix. Based on the statistical results of the speech matrix, speech frames with 0, 1, or more than or equal to 2 speakers in each segment of speech to be separated are selected. Only speech frames with more than or equal to 2 target speakers are retained, and these retained speech frames are merged to obtain multiple target speaker speech segments M = {M1, M2, ..., M...} for the segment of speech to be separated. n Further speech separation was performed on it.

[0092] S4: Augment the speech data to be separated, which contains multiple target speakers. Construct and train a GAN-based speech generation model containing multiple target speakers. Input the speech data to be augmented, which contains multiple target speakers, into the model to obtain the generated speech data to be separated, which contains multiple target speakers. The specific steps are as follows:

[0093] S41: Extract Mel-spectral features H = {h1, h2, ..., h...} from the speech data to be separated, which contains multiple target speakers. i , ...M t}, h v Let u represent the u-th segment of speech to be separated, where u∈[1,t].

[0094] S42: Construct a GAN-based speech generation model, consisting of a generator and a discriminator. Mel-spectral features H, containing the speech of multiple target speakers, are input into the generator. After passing through a Conv layer, they are fed into an upsampling network block. There are four upsampling network blocks: 8x, 8x, 2x, and 2x. Each upsampling network block contains nested residual blocks to achieve frequency-to-time domain conversion. Each residual block has three layers with dilation values ​​of 1, 3, and 9, respectively. Finally, a convolutional layer is passed to obtain the speech signal corresponding to the input features. The speech signal generated by the generator and the original speech signal are fed into the discriminator D for discrimination. The discriminator adopts a multi-scale architecture. In addition to discriminating the original audio, it also performs average pooling to down-convert the original audio before inputting it into the next discriminator, aiming to capture high-frequency results in the audio. The generator and discriminator are continuously trained to obtain a speech generation model capable of generating speech from multiple target speakers.

[0095] To enhance the temporal correlation of speech during generation, a residual stack with dilated convolutions is added after upsampling at each layer, as shown in the structure below. Figure 3As shown, this residual block allows for greater overlap in receptive fields at relatively large time steps, thus enabling the generated speech from multiple target speakers to have better long-range relevance. The generator processes the one-dimensional feature H containing multiple target speakers into a speech signal containing multiple target speakers through four layers of upsampling and convolution. The discriminator consists of three sub-discriminators (D1, D2, D3), which have the same network structure and operate at different sampling frequencies. The input of D1 is speech at the normal sampling frequency, the input of D2 is speech obtained by downsampling once, and the input of D3 is speech obtained by downsampling twice. By separating different frequencies, the discriminator calculates the differences in speech at different frequencies, thereby optimizing the training effect.

[0096] S43: Input the speech segment M to be expanded into the speech generation model trained in S42 to obtain n segments of generated speech segments containing multiple target speakers. Merge these segments with the original speech segments to be separated to obtain a new speech segment Data = {data1, data2, ..., data...} n , ...data 2n}

[0097] S5: Construct a TCN-based speech separation model. Input the speech data containing multiple target speakers to be separated, along with the augmented speech data obtained in S4, into the speech separation model for training. Then, input the speech data containing multiple target speakers into the trained speech separation model. Finally, separate the speech data containing only one target speaker. The specific steps are as follows:

[0098] S51: Generate a speech signal with silence segments using Praat, and splice it to the end of the speech signal to be separated, which contains multiple target speakers, so that the duration of all the speech signals to be separated containing multiple target speakers is equal.

[0099] S52: Construct a TCN-based speech separation model, consisting of an encoder, a separator, and a decoder. The encoder extracts features from the input waveform of the speech to be separated, which contains multiple target speakers, using a 1*1 convolution, transforming the speech waveform into a feature matrix. The separator feeds the feature matrix obtained from the encoder into a TCN-based temporal convolutional network to learn the feature matrix for each speaker. Using a 1*1 convolution, it first expands the number of channels to distinguish different speakers by different channel dimensions, and then separates the feature matrix for the corresponding speaker. The feature matrix obtained from the separator is fed into the decoder, and an inverse transform is performed to obtain the corresponding speech signal, achieving the separation of multiple target speakers.

[0100] The encoder's input is the time-domain speech waveform of the speech to be separated from multiple target speakers. The waveform is a one-dimensional...

[0101] A one-dimensional vector I = Time * p = 1 * 60 * 600 = 36000 is transformed into a 1 × 256 × 3599 two-dimensional feature matrix by the encoder through a one-dimensional convolution Conv1D(1, 256, 20, stride = 10, padding = 0), where 3599 = (36000 - 20) / 10 + 1. In the separator, the resulting 1 × 256 × 3599 feature matrix is ​​input into the circuit processed as follows... Figure 4 The TCN temporal convolutional network, composed of stacked CNN networks (Block A), learns the feature matrix for each speaker. It then uses a 1x1 convolution to first increase the number of channels, transforming the 1×256×3599 feature matrix into a 1×512×3599 matrix. Next, it distinguishes between different speakers based on different channel dimensions, then divides the matrix into multiple parts based on the speaker, such as splitting the 1×512×3599 matrix into two 1×256×3599 matrices to obtain the feature matrices for two speakers. Finally, the feature matrices learned by the separator are fed into the decoder for decoding, obtaining the corresponding speaker's speech waveform. This transforms the 1×256×3599 feature matrix into a 1×36000 one-dimensional vector with the same scale as the input.

[0102] S53: Training the speech separation model. The clean speech of the target speaker matched in each segment of speech to be separated in S53 is fused to obtain speech containing multiple target speakers, thus obtaining the training speech data for the separation model. The separation model is iteratively trained based on the signal-to-noise ratio (SNR) evaluation value. When the SNR evaluation value reaches its maximum value, the speech separation model is considered to have completed training, resulting in a well-trained speech separation model.

[0103] S54: Achieve speech separation for multiple target speakers. Input the speech waveforms containing multiple target speakers into the trained speech separation model, which outputs multiple speech signals containing a single speaker. Finally, based on the similarity matrix filtered by the threshold in S3 and the separated speech signals of a single speaker, concatenate the complete speech of that single speaker.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A small sample speech separation method based on data generation, characterized by: Includes the following steps: S1: Collect multiple segments of speech data to be separated and clean speech data of the target speaker, and construct the speech dataset to be separated and the clean speech small sample dataset of the target speaker respectively; S2: Preprocess the speech dataset to be separated and the clean speech dataset of the target speaker; S3: Use an LSTM-based speech matching model to perform frame-level target speaker matching on each speech segment in the preprocessed speech dataset to be separated. Based on the matching results, the speech segments are judged as: having no target speaker, having only one target speaker, or having multiple target speakers. S4: Using a GAN-based speech generation model containing multiple target speakers, the speech data to be separated containing multiple target speakers is processed to expand the speech segment dataset to be separated. S5: Construct and train a TCN-based speech separation model to separate the speech of multiple target speakers into speech data of a single target speaker, and finally splice the complete speech of a single speaker.

2. The small-sample speech separation method based on data generation of claim 1, characterized in that: Step S1 specifically includes the following steps: When collecting data, use a recording device to record the speaker and the real scene in which they are located, to obtain the speech dataset to be separated, V0 = {v1, v2, ..., v...} i , ..., v n }i∈[1,n], where n represents the total number of speech segments in the speech dataset to be separated, v i Let i represent the i-th segment of speech to be separated, and the duration of each segment is Time minutes; collect the clean speech signals of all target speakers without noise interference to obtain the target speaker clean speech small sample dataset Clean={c1, c2, ..., c j c m }j∈[1,m], where m represents the number of target speakers, c j This represents the clean speech of the j-th target speaker, with each clean speech segment lasting in Time minutes.

3. The small-sample speech separation method based on data generation of claim 1, characterized in that: The preprocessing in step S2 includes: S21: Use a threshold method combining short-time energy and zero-crossing rate to perform endpoint detection on the acquired speech to be separated and remove blank speech parts; S22: The speech segment to be separated obtained in step S21 is subjected to noise reduction processing using a deep learning model based on frequency domain masking.

4. The small-sample speech separation method based on data generation of claim 1, characterized in that: Step S3 specifically includes the following steps: S31: Divide the speech segment to be separated and the clean speech of the target speaker into p frames, and extract the Mel-spectral features X of the speech segment to be separated in each frame. a ={x 11 x 12 , ..., x 1k , ..., x 1p , ...x lk , ..., x np }, where x lk This represents the feature vector extracted from the k-th frame of the l-th segment of speech to be separated, where l∈[1,n], k∈[1,p]; and the Mel-spectral features Y of the clean speech of the target speaker in each frame are extracted. b ={y 11 y 12 , ..., y 1k , ..., y 1p , ...y jk , ..., y mp }, where y jk This represents the feature vector extracted from the k-th frame of the clean speech of the j-th target speaker. S32: Construct an LSTM-based speech matching model, and integrate Mel-spectral features X a With Y b Each frame of speech features in each segment of speech to be separated is arranged in a row from top to bottom in frame order; the clean speech features of each frame of the target speaker in each segment are also arranged in a row from top to bottom in frame order, forming a batch. The Mel-spectral features in the batch are processed through three layers of LSTM to obtain a feature vector containing the input speech features. A linear mapping layer is then added after the last LSTM layer to map the output of the last LSTM layer into a low-dimensional embedding feature vector. Then, the similarity between the embedding feature vector of each frame of speech to be separated and the embedding feature of the center vector of the clean speech of each target speaker is calculated to obtain a similarity matrix with dimensions n×pm. The average value of all elements in the similarity matrix is ​​taken as the threshold to identify the target speaker who is speaking in the segment of speech to be separated. S33: divide the speech segment to be separated into speech segments containing no target speaker, containing one target speaker, containing multiple target speakers; use a speaker matrix with a dimension of n x p to record the number of target speakers who are speaking in each frame in the similarity matrix; according to the statistical result of the speaker matrix, screen out speech frames in which the current number of target speakers is equal to 0, 1 and greater than or equal to 2, only keep the speech frames in which the current number of target speakers is greater than or equal to 2, and combine the kept speech frames to obtain multiple target speaker speech segments M = {M1, M2,..., M n} of the speech segment to be separated, and further separate the speech segments.

5. The small-sample speech separation method based on data generation of claim 4, characterized in that: Step S32 specifically includes the following steps: S321: The speech to be separated batch consists of n x p mel-frequency spectral features, where there are n segments of speech to be separated, each of which is divided into p frames, and the feature of each frame corresponds to the feature element in X a ; The speaker clean speech batch is composed of m x p mel-frequency spectrum feature vectors, where there are m segments of target speaker clean speech, each of which is divided into p frames, and the features of each frame correspond to the feature elements in Y b ​ S322: X a After three-layer LSTM processing, the feature vector of each frame of speech is obtained, and finally a linear mapping layer is used to reduce the dimension of the feature vector, resulting in f(x lk ; W), where x lk is the kth frame of the lth segment of the speech signal to be separated, f is the linear mapping function after the last LSTM layer, W is the parameter of the function, represented by a one-dimensional vector; S323: define the embedding feature vector as the L2 norm normalization of f(x lk ; W), and denote the embedding feature vector of the kth frame of the lth segment of the speech to be separated as e l，k : where ||f(x lk ; W)||2 is the L2 norm of the model output f(x lk ; W). S324: Y b The embedding feature vector e of the target speaker clean speech obtained through the model processing j，k , represents the embedding feature vector of the jth target speaker, the kth frame clean speech; the embedding feature vectors of p frames of clean speech signals are averaged to obtain the speaker center feature vector Centre j , represents the center feature vector of the jth target speaker clean speech signal: S325: calculate each by X a The e l，k Cosine similarity S j of each target speaker clean speech center feature vector Centre l，k，j : S l，k，j = w * cos(e l，k , Centre j ) + b Where w and b are the weights and biases that the neural network can learn, respectively, and the calculation results will form a similarity matrix with dimensions n×pm; S326: Calculate the average value of all elements in the similarity matrix and set it as the speaker determination threshold. When the matrix element value is greater than or equal to the threshold, set the frame to 1 and determine that the corresponding target speaker is speaking in the frame; otherwise, set it to 0 and determine that the speaker is not speaking in the frame.

6. The small-sample speech separation method based on data generation of claim 5, characterized in that: Step S326 calculates each of the X a The e l，k The cosine similarity S j of each target speaker clean speech central feature vector Centre l，k，j The specific process is as follows: By X a The e 1，1 Cosine similarity with Centre1, the result of the calculation is taken as the first element of the similarity matrix, and the elements of this row are e 1，1 The result of the cosine similarity calculation with all Centres; the next row is e 1，2 The result of the cosine similarity calculation with all Centres, and so on, until e 1，p The result of the cosine similarity calculation with all Centres; Next compute e 2，1 The cosine similarity with all Centres, the result of the computation is taken as the first element of the p+1 row of the similarity matrix, the row elements are e 2，1 The cosine similarity with all Centres, the result of the computation is taken as the first element of the p+1 row of the similarity matrix, the row elements are e n，p The cosine similarity with all Centres, the result of the computation is taken as the first element of the p+1 row of the similarity matrix, the row elements are e 7. The small-sample speech separation method based on data generation of claim 1, characterized in that: Step S4 Specifically, the following steps are included: S41: Extracting mel-spectrum features H = {h1, h2,..., hn} from the to-be-separated speech segments M containing multiple target speakers. v ,...h n} represent the vth segment of to-be-separated speech, v e [1, n]. v ​ S42: Construct a GAN-based speech generation model, consisting of a generator and a discriminator; The Mel spectral features H containing the speech of multiple target speakers are used as the input to the generator. After passing through a convolutional layer, they are fed into an upsampled network block. Each upsampled network block contains nested residual blocks. Finally, after passing through a convolutional layer, the speech signal output corresponding to the input features is obtained. The discriminator employs a multi-scale architecture to discriminate the original audio. It also performs frequency reduction processing on the original audio using average pooling before inputting it into the next discriminator for discrimination. The speech signal generated by the generator and the original speech signal are input into the discriminator D, passed through a convolutional layer, and then fed into a downsampling network block. There are four downsampling network blocks. Finally, the discriminator passes through two convolutional layers to obtain the discrimination result. By continuously training the generator and discriminator, a speech generation model that can generate speech from multiple target speakers is obtained. S43: Put the speech segment M to be separated that needs to be expanded into the speech generation model trained to obtain n generated speech segments containing multiple target speakers, and merge the original speech to be separated to obtain new speech segments Data = {data1, data2,..., data n ,...data 2n}.

8. The small-sample speech separation method based on data generation of claim 7, characterized in that: The generator has four upsampling network blocks, namely 8x, 8x, 2x, and 2x. After each upsampling layer, a residual stack with dilated convolution is added. Each residual stack has three layers, and the dilation coefficient of each layer is 1, 3, and 9. The generator processes the one-dimensional feature H containing multiple target speakers into a speech signal containing multiple target speakers through four layers of upsampling and convolution. The discriminator consists of three sub-discriminators D1, D2, and D3. The sub-discriminators have the same network structure and operate at different sampling frequencies. The input of D1 is the speech at the normal sampling frequency, the input of D2 is the speech obtained by downsampling once, and the input of D3 is the speech obtained by downsampling twice.

9. The small-sample speech separation method based on data generation of claim 1, characterized in that: Step S5 specifically includes the following steps: S51: Generate a speech signal with silence segments using Praat, and splice it to the end of the speech signal to be separated that contains multiple target speakers, so that all the speech signals to be separated that contain multiple target speakers have the same duration. S52: Construct a speech separation model based on TCN, consisting of an encoder, a separator, and a decoder; The encoder extracts features from the input waveform of the speech to be separated, which contains multiple target speakers, through a 1*1 convolution, and transforms the speech waveform into a feature matrix. The separator feeds the feature matrix obtained from the encoder into a TCN-based temporal convolutional network to learn the feature matrix of each speaker. It uses a 1*1 convolution to first increase the number of channels, distinguish different speakers by different channel dimensions, and then divides the feature matrix into the corresponding speaker's feature matrix. The feature matrix obtained by the separator is fed into the decoder, and the corresponding speech signal is obtained by inverse transformation, thus realizing the separation of multiple target speakers; S53: Fuse the clean speech of the target speaker matched in each segment of speech to be separated in step S3 to obtain speech containing multiple target speakers, thereby obtaining the training speech data of the separation model; perform iterative training on the speech separation model according to the signal-to-noise ratio evaluation value, and determine that the speech separation model training is completed when the signal-to-noise ratio evaluation value reaches the maximum value, thus obtaining the trained speech separation model. S54: Input the speech waveforms to be separated, which contain multiple target speakers, into the trained speech separation model, and output multiple speech signals containing a single speaker to achieve speech separation of multiple target speakers.

10. The small-sample speech separation method based on data generation of claim 9, characterized in that: In step S52, the input of the encoder is the waveform of the speech to be separated from multiple target speakers in the time domain. The waveform is a one-dimensional vector with dimension I, where I = Time * p. The encoder transforms the one-dimensional vector into a two-dimensional feature matrix of 256 × P through a one-dimensional convolution Conv1D(1,256,20,stride=10,padding=0), where P = (I-20) / 10 + 1. In the separator, the obtained 256xP feature matrix is input into a TCN time convolution network composed of a stacked CNN network, the feature matrix of each speaker is obtained by learning, the channel number is expanded by using convolution, then different channel dimensions of different speakers are distinguished, and then the feature matrix of each speaker is divided into multiple parts according to the number of speakers; Finally, the feature matrix of different speakers learned by the separator is put into the decoder for decoding to obtain the speech waveform corresponding to the speaker, and the 256xP feature matrix is converted into a one-dimensional vector of the same size as the input I.

Images