A speech separation method based on air flow estimation

Abstract

The present application belongs to the technical field of digital audio processing, and specifically provides a speech separation method based on air flow estimation, to solve the problem of reduced separation accuracy of existing methods in low harmonic component scenarios. The present application estimates the glottal air flow of the speaker and the speech harmonic component, and on this basis, suppresses the air flow of the harmonic frame in units of frames, thereby generating an air flow feature, making up for the deficiency of traditional methods in audio representation ability in low harmonic component scenarios. The air flow feature and the original audio data are spliced to form new training and testing samples, thereby improving the accuracy of non-harmonic speech separation while maintaining the separation effect of traditional methods on normal speech.

Description

Technical Field

[0001] This invention belongs to the field of digital audio processing technology, specifically providing a speech separation method based on airflow estimation. Background Technology

[0002] In telephone and video conferencing, there are often situations where multiple speakers speak at the same time. The goal of speech separation is to separate the audio of a single speaker from the audio of multiple people speaking at the same time. Speech separation is often used to improve the performance of tasks such as speech recognition and sentiment analysis.

[0003] Currently, speech separation methods are mainly divided into traditional methods and deep learning methods. Traditional methods rely on techniques such as sound source localization and beamforming for separation, while deep learning methods are divided into two categories: spectrogram-based methods and audio time-series-based methods. Deep learning methods have significantly improved separation performance compared to traditional methods. However, both traditional and deep learning speech separation methods are designed for general scenarios where the speaker is speaking normally. These methods all analyze and model harmonic components in speech. In certain special scenarios, such as when the speaker intentionally lowers their voice or is breathing heavily, resulting in lower harmonic components, existing methods lack sufficient audio representation capabilities, leading to a significant decrease in separation accuracy. Therefore, it is necessary to research methods to improve speech separation accuracy in scenarios with low harmonic components. Summary of the Invention

[0004] The purpose of this invention is to provide a speech separation method based on airflow estimation to solve the problem of reduced separation accuracy of existing methods in low harmonic component scenarios.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A speech separation method based on airflow estimation, comprising: model construction and speech separation, characterized in that:

[0007] The model construction includes the following steps:

[0008] A1. Constructing airflow features: Suppose there are M audio samples for training, each containing the voice of a single speaker and having the same number of audio samples K. Process each audio sample as follows:

[0009] A1-1. Estimating airflow: Input the input audio C into the glottal airflow estimation model to obtain the airflow estimation sequence W corresponding to the audio;

[0010] A1-2. Estimating harmonic components:

[0011] The audio data is divided into audio frames of fixed length, with adjacent frames partially overlapping, resulting in I audio frames.

[0012] Define a harmonic component array H of length I. For each audio frame, input audio frame i into the fundamental frequency estimation algorithm to obtain the fundamental frequency component of the audio frame. If the audio frame does not have a fundamental frequency component, set the i-th element H[i] of H to 1; otherwise, set the i-th element H[i] of H to 0, 1≤i≤I.

[0013] A1-3. Characteristics of generated gas flow rate:

[0014] Define a gas flow characteristic array F of length K, and calculate the gas flow characteristics:

[0015] F[k]=W[k]·H[z],1≤k≤K

[0016] Where F[k] represents the air flow characteristic of the k-th audio sample point, W[k] represents the estimated air flow value corresponding to the k-th audio sample point, z is the frame number corresponding to the k-th audio sample point, and H[z] represents the z-th element value of the harmonic component array H;

[0017] A2. Generate training samples:

[0018] A2-1. Randomly select N audio samples from the training audio to form a group, and select a total of P times to form P groups of audio;

[0019] A2-2. Process each group of audio as follows:

[0020] Normalize the data of the nth audio element in the audio group to obtain the normalized audio data sequence c. n , 1≤n≤N;

[0021] Normalize the elements of the airflow feature array F of the nth audio in the audio group to obtain the normalized airflow feature data sequence f. n ;

[0022] f n With c n The combined feature T is obtained by splicing. n T n =[c n f n ];

[0023] The combined feature T of all audio within the group n By performing vector addition, we obtain the training sample X corresponding to the audio group p. p :

[0024] X p =T1+T2+……+T N 1≤p≤P

[0025] A3. Training the speech separation model: The generated training samples X... p The input is fed into the speech separation model for training, and the training is completed using the backpropagation algorithm.

[0026] The speech separation includes the following steps:

[0027] B1. Constructing airflow characteristics: Let the test audio C' be a mixed voice containing N speakers, with K audio samples; generate airflow characteristics based on the test audio:

[0028] B1-1. Estimate air flow rate: Following step A1-1, obtain the air flow rate estimation sequence W' corresponding to the test audio C';

[0029] B1-2. Estimating harmonic components: Following step A1-2, obtain the harmonic component array H' of the test audio C';

[0030] B1-3. Generate airflow characteristics: Following step A1-3, obtain the airflow characteristic array F' of the test audio C';

[0031] B2. Generate test samples:

[0032] B2-1. Normalize the data of the test audio C' to obtain the normalized audio data sequence c';

[0033] B2-2. Normalize the elements of the airflow feature array F' of the test audio C' to obtain the normalized airflow feature data sequence f';

[0034] B2-3. Concatenate f' and c' to obtain the test sample X', where X' = [c'f'];

[0035] B3. Speech Separation: Input the test sample X' into the speech separation model for separation to obtain the individual speech of each of the N speakers.

[0036] Based on the above technical solution, the beneficial effects of the present invention are as follows:

[0037] This invention provides a speech separation method based on airflow estimation. This method estimates the glottal airflow and harmonic components of the speaker. Based on this, the airflow of harmonic frames is suppressed on a frame-by-frame basis to generate airflow features. This method makes up for the shortcomings of traditional methods in audio representation in low harmonic component scenarios. The airflow features and the original audio data are spliced ​​together to form new training and testing samples. Thus, while improving the accuracy of non-harmonic speech separation, it also maintains the separation effect of traditional methods on normal speech. Attached Figure Description

[0038] Figure 1This is a flowchart illustrating the speech separation method based on airflow estimation in this invention, which includes two stages: model building and speech separation.

[0039] Figure 2 The diagram shows the effect of airflow estimation in the speech separation method based on airflow estimation in this invention. (a) is the waveform of the input audio, and (b) is the data diagram of the airflow estimation sequence. Detailed Implementation

[0040] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0041] This embodiment provides a speech separation method based on airflow estimation, the process of which is as follows: Figure 1 As shown, this method consists of two stages: model building and speech separation.

[0042] The model construction includes the following steps:

[0043] A1. Constructing airflow features: Generate airflow features based on the training audio. Assume there are M audio samples for training, and each audio sample contains the voice of a single speaker with the same number of audio samples K. For each audio sample, perform the following processing:

[0044] In this embodiment, a total of 1332 audio samples are used for training, i.e., M = 1332. Each audio sample is a mono audio with a sampling rate of 22500Hz, containing only the voice of a single speaker. The audio duration is 10 seconds, and the number of audio samples is 225000, i.e., K = 225000. The following steps are illustrated using the processing of the first female audio sample as an example. The waveform of this audio sample is shown below. Figure 2 As shown in (a);

[0045] A1-1. Estimate airflow: Input the input audio C into the existing glottal airflow estimation model to obtain the airflow estimation sequence W corresponding to the audio, that is, each sample point C[k] (1≤k≤K) of C corresponds to the kth estimated value W[k] of the airflow estimation sequence W;

[0046] In this embodiment, the input audio C is input into an existing glottal airflow estimation model to obtain the corresponding airflow estimation sequence W = [0.001, 0.001, ..., 0.095]. W has a total of 225,000 elements, corresponding to 225,000 sample points of the original audio. The data of the airflow estimation sequence W is as follows: Figure 2 As shown in (b);

[0047] A1-2. Estimating harmonic components:

[0048] A1-2-1. Divide the audio data into audio frames of fixed length, with overlap between adjacent frames, to obtain I audio frames;

[0049] In this embodiment, the audio data is divided into audio frames with a length of 512 sampling points. There is a half-frame overlap between adjacent frames, and the total number of audio frames obtained is I = 878.

[0050] A1-2-2. Define a harmonic component array H of length I;

[0051] A1-2-3. For each audio frame: input the audio frame i (1≤i≤I) data into the existing fundamental frequency estimation algorithm to estimate the fundamental frequency component of the audio frame; if the audio frame does not have a fundamental frequency component, set the i-th element H[i] of H to 1, otherwise set the i-th element H[i] of H to 0;

[0052] In this embodiment, the data from frame 1 to frame 878 are sequentially input into the fundamental frequency estimation algorithm YIN to estimate the fundamental frequency component of each audio frame, thereby determining the value of each element of the harmonic component array H; taking frame i=137 as an example, the audio data of frame 137 is input into the fundamental frequency estimation algorithm YIN for fundamental frequency component estimation, and it is found that the fundamental frequency component of the audio frame does not exist, so the 137th element H

[137] =1 is set. After performing this step, the harmonic component array H=[1,1,…,1] is finally obtained;

[0053] A1-3. Characteristics of generated gas flow rate:

[0054] A1-3-1. Define a gas flow characteristic array F of length K;

[0055] A1-3-2. Calculation of gas flow characteristics:

[0056] F[k]=W[k]·H[z], 1≤k≤K,

[0057] Where F[k] is the kth element of the air flow feature array F, and z is the frame number corresponding to the audio sample point numbered k;

[0058] In this embodiment, taking the 34900th sample point as an example, i.e., k = 34900, the 34900th element of the gas flow estimation sequence W, W

[34900] = 0.632, the frame number corresponding to the 34900th sample point, z = 137, and H

[137] = 1, then the 34900th element of the gas flow feature array, F

[34900] = W

[34900] ·H

[137] = 0.632, is calculated; after performing this step, the gas flow feature array F = [0.001, 0.001, ..., 0.095] is finally obtained;

[0059] A2. Generate training samples:

[0060] A2-1. Randomly select N audio samples from the training audio to form a group, and select a total of P times to form P groups of audio;

[0061] In this embodiment, N=2 audio samples are randomly selected from the training audio to form a group, and a total of P=10000 selections are made to form 10000 audio groups.

[0062] A2-2. For each group of audio, do the following:

[0063] In this embodiment, the first group is taken as an example;

[0064] A2-2-1. For each n satisfying 1≤n≤N, perform the following:

[0065] In this embodiment, N = 2, therefore for each n satisfying 1 ≤ n ≤ 2, the following is performed:

[0066] A2-2-1-1. Normalize the data of the nth audio element in the current audio group to obtain the normalized audio data sequence c. n ;

[0067] In this embodiment, taking the n=1th audio as an example, the first audio C1 is normalized. For each k satisfying 1≤k≤225000, the kth element c1[k] of the normalized audio data sequence c1 is calculated:

[0068]

[0069] in, and These are the maximum value of 0.351 and the minimum value of -0.303 among the 225,000 audio sample data points, respectively. Taking the 34900th sample point as an example, i.e., k = 34900, and C1

[34900] = 0.225, then...

[0070]

[0071] After performing this step, the normalized audio data sequence c1 = [0.025, 0.021, ..., 0.087] is obtained;

[0072] A2-2-1-2. Normalize the elements of the airflow feature array F corresponding to the nth audio frequency to obtain the normalized airflow feature data sequence f. n ;

[0073] In this embodiment, taking the n=1th audio signal as an example, the first airflow feature F1 is normalized. For each k satisfying 1≤k≤225000, the kth element f1[k] of the normalized airflow feature data sequence f1 is calculated:

[0074]

[0075] Among them, F1 max and F1 min These are the maximum value of 1.127 and the minimum value of 0 among 225,000 gas flow characteristic data, respectively, which are F1. max =1.127, F1 min =0, taking the 34900th sample point as an example, i.e., k = 34900, and F1

[34900] = 0.632, then

[0076]

[0077] After performing this step, the normalized gas flow characteristic data sequence f1 = [0.001, 0.001, ..., 0.084] is obtained;

[0078] A2-2-1-3. f n The K data points are concatenated into c n After obtaining K data points, a combined feature T of length 2K is obtained. n That is, T n =[c n f n ];

[0079] In this embodiment, taking the n=1th audio as an example, f1=[0.001,0.001,…,0.084] is concatenated to c1=[0.025,0.021,…,0.087] to obtain the combined feature T1=[0.025,0.021,…,0.084] with a length of 2K;

[0080] A2-2-2. Combine the features T of all audio within the group. n (1≤n≤N) Perform vector addition to generate the training sample X corresponding to the audio group p. p :

[0081] X p =T1+T2+……+T n

[0082] In this embodiment, N=2. Taking audio group p=1 as an example, the combined features T1=[0.025,0.021,…,0.084] and combined features T2=[0.010,0.051,…,0.291] of the two audio samples in the group are vector-added to generate the training sample X1 corresponding to the audio group p=1:

[0083] X1=T1+T2=[0.035,0.072,…,0.375]

[0084] A3. Training the speech separation model: The generated training samples X... p (1≤p≤P) is input into the speech separation model for training, and the backpropagation algorithm is used to complete the training.

[0085] In this embodiment, the generated training sample X p (1≤p≤10000) are input into the speech separation model SepFormer for training, and the backpropagation algorithm is used to complete the training.

[0086] The speech separation includes the following steps:

[0087] B1. Constructing airflow characteristics: Generate airflow characteristics based on the test audio. Assume that the test audio contains the mixed voices of N speakers and the number of audio samples is K.

[0088] In this embodiment, a total of 8,000 audio samples are used for testing. Each audio sample is a mono audio with a sampling rate of 22,500 Hz, and each contains the mixed voices of N=2 speakers. The audio duration is 10 seconds, and the number of audio samples is 225,000, i.e., K=225,000. The following steps are explained using the processing of the first test audio sample as an example.

[0089] B1-1. Estimate air flow rate: Following step A1-1, obtain the air flow rate estimation sequence W' corresponding to the test audio C';

[0090] In this embodiment, the input audio C' is input into the existing glottal airflow estimation model to obtain the airflow estimation sequence W' = [0.003, 0.005, ..., 0.004] corresponding to the audio. W' has a total of 225,000 elements, corresponding to 225,000 sample points of the original audio.

[0091] B1-2. Estimating harmonic components: Following step A1-2, obtain the harmonic component array H' corresponding to each audio frame of the test audio C';

[0092] In this embodiment, following steps A1-2, the harmonic component array H' = [1,1,…,1] corresponding to each audio frame of the test audio C' is obtained;

[0093] B1-3. Generate airflow characteristics: Following step A1-3, obtain the airflow characteristic array F' corresponding to the test audio C';

[0094] In this embodiment, following steps A1-3, the air flow characteristic array F' = [0.003, 0.005, ..., 0.004] corresponding to the test audio C' is obtained;

[0095] B2. Generate test samples:

[0096] B2-1. Normalize the data of the test audio C' to obtain the normalized audio data sequence c';

[0097] In this embodiment, C' is normalized, and for each k satisfying 1≤k≤225000, the k-th element c'[k] of the normalized audio data sequence c' is calculated:

[0098]

[0099] Among them, C' max and C' min These are the maximum value of 2.120 and the minimum value of -1.529 from the 225,000 audio sample data points, respectively, i.e., C' max =2.120, C' min =-1.529, taking the 34900th sample point as an example, i.e., k=34900, and C'

[34900] =-0.239, then

[0100]

[0101] After performing this step, the normalized audio data sequence c' = [0.004, 0.011, ..., 0.028] is obtained;

[0102] B2-2. Normalize the elements of the air flow feature array F' corresponding to the test audio C' to obtain the normalized air flow feature data sequence f';

[0103] In this embodiment, F' is normalized, and for each k satisfying 1≤k≤225000, the k-th element f'[k] of the normalized gas flow characteristic data sequence f' is calculated:

[0104]

[0105] Among them, F' max and F' min These are the maximum value of 1.326 and the minimum value of 0 among 225,000 gas flow characteristic data, respectively, i.e., F' max =1.326, F' min=0, taking the 34900th sample point as an example, i.e., k = 34900, and F'

[34900] = 0.611, then

[0106]

[0107] After performing this step, the normalized gas flow characteristic data sequence f' = [0.002, 0.003, ..., 0.002] is obtained;

[0108] B2-3. Concatenate the K data points of f' to the K data points of c' to obtain a test sample X' of length 2K, i.e., X' = [c'f'];

[0109] In this embodiment, f' = [0.002, 0.003, ..., 0.002] is concatenated to c' = [0.004, 0.011, ..., 0.028] to obtain a combined feature X' of length 2K, i.e., X' = [0.004, 0.011, ..., 0.002].

[0110] B3. Speech Separation: Input the test sample X' into the speech separation model for separation to obtain the individual speech of each of the N speakers;

[0111] In this embodiment, the test sample X' is input into the speech separation model for separation, resulting in N=2 individual voices of each speaker.

[0112] The performance of the method of the present invention is compared with that of the SepFormer speech separation method below. The separation effect is measured by SI-SNR. Both methods are trained and tested using the same example dataset, and the results are shown in Table 1, where the data are the average values ​​of the test audio.

[0113] Table 1

[0114] method SI-SNR SepFormer 0.17 Method of the present invention 5.7

[0115] As can be seen, the SI-SNR of the method of the present invention is significantly improved compared with the traditional SepFormer method, indicating that the present invention achieves better speech separation effect.

[0116] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.

Claims

1. A speech separation method based on airflow estimation, comprising: Model building and speech separation, characterized by: The model construction includes the following steps: A1. Constructing gas flow characteristics: Assume a total of Each audio file is used for training, and each audio file contains the voice of a single speaker and has the same number of audio samples. For each audio file, the following processing is performed: A1-1. Estimate airflow: Input audio The data is input into the glottal airflow estimation model to obtain the airflow estimation sequence corresponding to the audio audio. ; A1-2. Estimating harmonic components: The audio data is divided into audio frames of fixed length, with adjacent frames partially overlapping. One audio frame; Define the length as harmonic component array For each audio frame, perform the following: The input is fed into the fundamental frequency estimation algorithm to obtain the fundamental frequency component of the audio frame; if the audio frame does not have a fundamental frequency component, then... The element Set to 1, otherwise, set to 1. The element =0, Finally, the harmonic component array is obtained. ; A1-3. Characteristics of generated gas flow rate: Define the length as air flow characteristic array Calculate the gas flow characteristics: ， in, Indicates the first Airflow characteristics of each audio sample point Indicates the first The estimated airflow value corresponding to the audio sample point, z is the th audio sample point. The frame number corresponding to each audio sample point. Represents the array of harmonic components The Each element value; A2. Generate training samples: A2-1. Randomly select from the training audio. Each audio file is grouped into a set, and a total of [number] audio files are selected. Next, forming Group audio; A2-2. Process each group of audio as follows: For audio group number The audio data is normalized to obtain a normalized audio data sequence. , ; with the first Taking one audio file as an example, for the first audio file... Normalization is performed on the audio data sequence. The element for: in, For the first audio The One element, and The first audio The maximum and minimum values ​​in the audio sample data; For audio group number Airflow feature array of each audio audio element The elements are normalized to obtain a normalized gas flow characteristic data sequence. ; with the first Taking one audio signal as an example, the first airflow characteristic... Normalization is performed on the gas flow characteristic data sequence. The element for: in, The first gas flow characteristic The One element, and The first gas flow characteristic The maximum and minimum values ​​in the gas flow characteristic data; Will and Combined features are obtained by splicing. , ; Combined features of all audio within the group By performing vector addition, the audio group is obtained. Corresponding training samples : ， A3. Training the speech separation model: The generated training samples... The input is fed into the speech separation model for training, and the training is completed using the backpropagation algorithm. The speech separation includes the following steps: B1. Constructing airflow characteristics: Set the test audio For inclusion The mixed voice of the speaker, with an audio sample count of [number missing]. ; Generate airflow characteristics based on test audio: B1-1. Estimate airflow: Follow step A1-1 to obtain the test audio. Corresponding gas flow estimation sequence ; B1-2. Estimating Harmonic Components: Following step A1-2, obtain the test audio. harmonic component array ; B1-3. Generating airflow characteristics: Following step A1-3, obtain the test audio. air flow characteristic array ; B2. Generate test samples: B2-1. Test audio The data is normalized to obtain a normalized audio data sequence. ; B2-2. Test audio air flow characteristic array The elements are normalized to obtain a normalized gas flow characteristic data sequence. ; B2-3. [The following is a list of items / items] and Test samples were obtained by splicing. , ; B3. Speech Separation: Test samples... The input is fed into a speech separation model for separation, and the result is... Each speaker's individual voice recording.

Images