Frame-level multi-channel speaker verification method under large-scale self-organizing microphone array

Abstract

The application discloses a frame-level multi-channel speaker verification method under a large-scale self-organizing microphone array, a space-time processing block is added before a pooling layer of a single-channel speaker verification system, and context relationships in a channel, between channels and across time are modeled respectively, and the performance of a far-field ASV is further improved. The method comprises the following steps: 1) a space-time processing block composed of a cross-frame processing layer (CFL) and a cross-channel processing layer (CCL) is added before the pooling layer; and 2) in order to make the channel weight of a noise channel zero, a softmax operator of the cross-channel processing layer is improved into a sparsemax operator. Results on a Libri-adhoc-simu dataset show that a multi-channel ASV system of the STB realizes an equal error rate (EER) of less than 33% of an oracle one-best baseline; and results on a Libri-adhoc40 dataset show that the multi-channel ASV system of the STB realizes an equal error rate of less than 27% of the oracle one-best baseline, and also realizes an equal error rate of less than 9% of a speech-level cross-channel self-attention ASV system, and achieves superior performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech recognition, and particularly relates to a frame-level multi-channel speaker verification method. BACKGROUND

[0002] In recent years, with the rapid development of voice intelligent devices, the demand for far-field voice interaction continues to grow. Due to the attenuation of voice signals, noise interference and spatial reverberation, far-field automatic speaker verification (ASV) is still challenging.

[0003] To solve the above problems, it has been proved in the past research that single-channel and multi-channel speech enhancement has made substantial improvements to the ASV system. These studies mainly focus on single-channel front-end or multi-channel front-end of a single device, and the microphone array with known geometry is an important method to improve the performance of ASV. Compared with the fixed microphone array, the ad-hoc microphone array is composed of a group of randomly placed microphone nodes in the acoustic environment, which provides more flexibility and allows users to virtually form a microphone array system using their own mobile devices. Unlike traditional microphone array settings, the spatial arrangement and number of microphones of the ad-hoc microphone array are not known in advance, which hinders the application of traditional multi-channel ASV technology.

[0004] A method has recently been proposed for ASV on ad-hoc microphone array, which fuses the sentence-level representation of each channel after the pooling layer to improve the performance of ASV. However, this method cannot fully utilize the frame-level information. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the application provides a frame-level multi-channel speaker verification method under a large-scale self-organizing microphone array, a space-time processing block is added before the pooling layer of a single-channel speaker verification system, the context relationship of intra-channel, inter-channel and cross-time is modeled respectively, and the performance of the far-field ASV is further improved. The method comprises the following steps: 1) a space-time processing block composed of a cross-frame processing layer (CFL) and a cross-channel processing layer (CCL) is added before the pooling layer; 2) in order to make the channel weight of the noise channel zero, the softmax operator of the cross-channel processing layer is improved into a sparsemax operator. The results on the Libri-adhoc-simu dataset show that the multi-channel ASV system of the STB realizes an equal error rate (EER) of less than 33% of the oracle one-best baseline; the results on the Libri-adhoc40 dataset show that the multi-channel ASV system of the STB realizes an equal error rate of less than 27% of the oracle one-best baseline, and also realizes an equal error rate of less than 9% of the utterance-level cross-channel self-attention ASV system, achieving superior performance.

[0006] The technical solution adopted by the application to solve the technical problems comprises the following steps:

[0007] Step 1: constructing a space-time processing block, including two parts: a cross-channel processing layer and a cross-frame processing layer;

[0008] Step 2: the cross-frame processing layer:

[0009] Step 2-1: using to represent the input of the cross-frame processing layer, using to represent the input feature of the cth channel of the cross-frame processing layer, c = 0, 1, …, C-1; T, C and N respectively represent the number of time frames, the number of input channels and the number of features of each channel;

[0010] Step 2-2: using h to represent the number of attention heads, for the ith attention head, a cross-frame query matrix with a dimension of E is obtained through linear transformation a cross-frame key matrix and a cross-frame value matrix

[0011]

[0012] wherein, and are learnable weight matrices and bias parameters, * ∈ {Q, K, V}, d k = E / h;

[0013] Step 2-3: Calculate the attention score of the i-th attention head of the cross-frame processing layer

[0014]

[0015] wherein P cf is the attention score from the previous cross-frame processing layer;

[0016] Step 2-4: Obtain the attention score by subspace connection to obtain the output of the attention layer

[0017]

[0018] wherein, denotes the weight matrix of the linear projection layer of the cross-frame processing layer;

[0019] Step 2-5: Obtain input into a feedforward neural network FFN with ReLU activation, and the output of the feedforward neural network FFN is the output of the current cross-frame processing layer;

[0020] Step 3: Cross-channel processing layer

[0021] Step 3-1: Denote the input of the cross-channel processing layer as wherein denotes the input feature at time t, t = 0, 1, …, T-1;

[0022] Step 3-2: For the i-th attention head, obtain the cross-channel query matrix with dimension E by linear transformation cross-channel key matrix and cross-channel value matrix

[0023]

[0024] wherein, is a learnable weight matrix, is a bias parameter;

[0025] Step 3-3: The output of the i-th attention head, i.e., the attention score is as follows:

[0026]

[0027] wherein P cc is the attention score from the previous cross-channel processing layer;

[0028] Step 3-4: attention score The output of the attention layer is obtained through the subspace connection

[0029]

[0030] wherein, represents the weight matrix of the linear projection layer;

[0031] Step 3-5: the input a feedforward neural network FFN with ReLU activation, and the output of the feedforward neural network FFN is the output of the current cross-channel processing layer;

[0032] Step 4: a plurality of stacked space-time processing blocks are added before the pooling layer of the single-channel speaker verification system, so as to realize the multi-channel speaker verification system.

[0033] Preferably, the Sparsemax operator is replaced with a softmax operator in the attention score in step 2-3:

[0034]

[0035] Preferably, in the cross-channel processing layer, layer normalization is applied before the attention module and the FFN module respectively, and residual connection is applied between the input and output of the attention module and the FFN, so as to alleviate the gradient vanishing problem of the cross-channel processing layer.

[0036] Preferably, in the cross-frame processing layer, layer normalization is applied before the attention module and the FFN module respectively, and residual connection is applied between the input and output of the attention module and the FFN, so as to alleviate the gradient vanishing problem of the cross-frame processing layer.

[0037] The beneficial effects of the present application are as follows:

[0038] 1. The present application proposes a multi-channel ASV system under an ad-hoc microphone array, which constructs a space-time processing block before a pooling layer, so as to fully utilize frame-level multi-channel information in a manner independent of the number and arrangement of microphones, has flexibility, and realizes superior performance.

[0039] 2. The present application uses a sparsemax operator, so that the channel weight of the noise channel is zero, and the performance of the system is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The space-time processing block network structure constructed by the present application.

[0041] Figure 2A multi-channel ASV system structure diagram proposed by the present application.

[0042] Figure 3 The EER results of the present application in the six nearest channels of the single-channel ASV. DETAILED DESCRIPTION

[0043] The present application is further illustrated below in conjunction with the drawings and examples.

[0044] In order to further improve the performance of the ASV system in the far field scene, the ad-hoc microphone array has been widely studied in recent years, however, the spatial arrangement and number of microphones of the ad-hoc microphone array are not known in advance, which will hinder the application of traditional speaker verification technology under the ad-hoc microphone array. Therefore, the present application proposes a new multi-channel ASV system under the ad-hoc microphone array, which constructs a space-time processing block before the pooling layer, to fully utilize the frame-level multi-channel information independently of the number and arrangement of microphones, and replaces the softmax operator in the residual self-attention module with the sparsemax operator, so that the channel weight of the noise channel is zero, and a more superior performance is achieved.

[0045] A frame-level multi-channel speaker verification method under a large-scale self-organizing microphone array, comprising the following steps:

[0046] Step 1: as shown in Figure 1 , a space-time processing block is constructed, including two parts: a cross-channel processing layer and a cross-frame processing layer;

[0047] Step 2: cross-frame processing layer:

[0048] Step 2-1: use to represent the input of the cross-frame processing layer, , represents the input feature of the cth channel of the cross-frame processing layer, c = 0, 1, …, C-1; T, C and N represent the number of time frames, the number of input channels and the number of features of each channel respectively;

[0049] Step 2-2: use h to represent the number of attention heads, for the ith attention head, a cross-frame query matrix with dimension E is obtained through linear transformation cross-frame key matrix and cross-frame value matrix

[0050]

[0051] wherein, and are learnable weight matrix and bias parameter respectively, * ∈ {Q, K, V}, d k = E / h;

[0052] Step 2-3: Calculate the attention score of the i-th attention head of the cross-frame processing layer

[0053] The general Softmax operator has a limitation for ad-hoc microphone array, the element of its output is not zero, which cannot be used for channel selection; secondly, if some channels are very noisy, when their weights are non-zero, it may harm the system; to solve this problem, the Sparsemax operator is used to replace the Softmax operator in CCL;

[0054]

[0055] wherein P cf is the attention score from the previous cross-frame processing layer;

[0056] Step 2-4: Obtain the attention score by subspace connection to obtain the output of the attention layer

[0057]

[0058] wherein, denotes the weight matrix of the linear projection layer of the cross-frame processing layer;

[0059] Step 2-5: Obtain input a feedforward neural network FFN with ReLU activation, and the output of the feedforward neural network FFN is the output of the current cross-frame processing layer;

[0060] Step 3: Cross-channel processing layer;

[0061] Step 3-1: Denote the input of the cross-channel processing layer as wherein denotes the input feature at t time, t = 0, 1, …, T-1;

[0062] Step 3-2: For the i-th attention head, obtain the cross-channel query matrix with dimension E by linear transformation cross-channel key matrix and cross-channel value matrix

[0063]

[0064] wherein, is a learnable weight matrix, is a bias parameter;

[0065] Step 3-3: The output of the i-th attention head, i.e. the attention score is as follows:

[0066]

[0067] wherein P cc is the attention score from the previous cross-channel processing layer;

[0068] Step 3-4: the attention score is obtained by subspace connection to get the output of the attention layer

[0069]

[0070] wherein, denotes the weight matrix of the linear projection layer;

[0071] Step 3-5: the is input into a feedforward neural network FFN with ReLU activation, and the output of the feedforward neural network FFN is the output of the current cross-channel processing layer;

[0072] Step 4: as Figure 2 shown, a plurality of stacked space-time processing blocks are added before the pooling layer of the single-channel speaker verification system to realize a multi-channel speaker verification system. Specific embodiments:

[0074] (1) Data preparation

[0075] Three data sets are used in the experiment: Librispeech corpus, Librispeech under ad-hoc microphone array simulation (Libri-adhoc-simu) and Libri-adhoc-40. Each node of the ad-hoc microphone array of Libri-adhoc-simu and Libri-adhoc-40 has only one microphone, so one channel refers to one node.

[0076] In the experiment, 960 hours of data are selected to train the single-channel ASV system, and 10 hours of data are selected as the development set.

[0077] Libri-adhoc40 collects data by playing the subsets “train-clean-100”, “dev-clean”, “test-clean” of Librispeech in a large room, which has strong reverberation and small additional noise. 20 channels are randomly selected for each training set and development set speech, and 20, 30 and 40 channels are selected for each test set speech, corresponding to three test scenarios.

[0078] Libri-adhoc-simu uses a subset of Librispeech, "train-clean-100", as the training data, which contains 251 speakers; uses "dev-clean" subset as the development set and "test-clean" as the test set, which contains 40 different speakers respectively. For each utterance, we simulate a room, the length and width of the room are randomly selected from the range of [5, 25] meters, the height of the room is randomly selected from the range of [2.7, 4] meters, multiple microphones and a loudspeaker are randomly placed in the room, and the distance between the source and the wall is greater than 0.2 meters, and the distance between the source and the microphone is at least 0.3 meters. The image source model is used to simulate the reverberation environment, and T 60 is selected from the range of [0.2, 0.4] seconds. The diffuse noise generator is used to simulate the uncorrelated diffuse noise, and the noise source for the training set and the development set is a large-scale noise library containing more than 20,000 noise segments, and the noise source for the test is the noise segment from the CHiME-3 and NOISEX-92 databases. 20 channels are randomly generated for training, and 20, 30 and 40 channels are used for testing.

[0079] (2) Model structure

[0080] The network structure of single-channel ASV includes three parts: a front-end residual convolutional neural network (Residual Convolution Neural Network, ResNet), a self-attentive pooling (Self-Attentive Pooling, SAP) layer and a fully connected layer, which are jointly optimized in an end-to-end manner and an angle prototype loss function. ResNet converts the original features into high-level abstract representations, and then the SAP layer and the fully connected layer process the frame-level representations into more abstract utterance-level representations. The residual block width is set to {16, 32, 64, 128}, the embedding size in the fully connected layer is 512, the number of spatio-temporal processing blocks is 2, the number of attention heads is 4, and the output dimension of each self-attention layer and FFT layer is 256.

[0081] Two baseline models of the ASV system are compared:

[0082] 1. Oracle one-best ASV: As a single-channel baseline, the channel closest to the sound source in a physical sense is selected as the input of the single-channel ASV model. Note: For the Oracle one-best baseline, the distance between the loudspeaker and the microphone is known in advance. To prove the rationality of the Oracle one-best, the six closest channels are selected as the input of the single-channel ASV, as shown in Figure 3 .

[0083] 2. Utterance-level Cross-Channel Self-Attention+ASV (UCCA+ASV): After the single-channel ASV pooling layer, an utterance-level cross-channel self-attention layer and a global fusion layer are added.

[0084] (3) Model training

[0085] During the training process, a fixed length of 2 seconds is randomly extracted from each utterance, a 40-dimensional mel filter is used as the input, and mean and variance normalization (MVN) is applied. For each round of training, a maximum of 100 utterances per speaker are randomly sampled to reduce class imbalance, no data augmentation is performed, the Adam optimizer is used, the initial learning rate is 0.001, and it is reduced by 5% every 5% round. During training, first, the single-channel ASV system is trained on the Librispeech corpus for 200 rounds, then the parameters of the ResNet layer and the SAP layer are fixed and sent to the multi-channel ASV system. Finally, the multi-channel ASV system is trained using the Libri-adhoc-simu and Libri-adhoc40 data respectively.

[0086] (4) Experimental results

[0087] Table 1 Comparison of EER (%) on Libri-adhoc-simu data and Libri-adhoc40 data

[0088]

[0089]

[0090] Table 2 Relative EER reduction (EERR) of using only CCL or CFL relative to the STB-ASV model

[0091]

[0092] Table 1 lists the performance comparison on Libri-adhoc-simu and Libri-adhoc40 datasets, from which we can see that the proposed method achieves superior performance in all test scenarios. Specifically, compared with oracle one-best, STB-ASV achieves more than 33% relative EER reduction on Libri-adhoc-simu dataset and more than 27% relative EER reduction on Libri-adhoc40 dataset. Compared with UCCA-ASV system, STB-ASV achieves more than 9% relative EER reduction on Libri-adhoc40 dataset, which proves the effectiveness of frame-level information modeling.

[0093] Meanwhile, the Sparsemax operator also achieves better performance than softmax. On the Libri-adhoc40 dataset, STB-ASV with Sparsemax operator achieves 9.22% EER in the mismatched 40-channel test environment, which reduces 12.4% EER compared with UCCA-ASV with Sparsemax operator. The above results show the effectiveness of the proposed model under ad-hoc microphone array.

[0094] In addition, model simplification tests are conducted to prove the importance of CCL and CFL. By training two variants of multi-channel ASV using only CCL or only CFL, it is concluded from Table 2 that when either attention is removed, the EER decreases significantly.

Claims

1. A method for speaker verification at frame level multi-channel under large-scale self-organizing microphone array, characterized in that, Comprising the following steps: Step 1: constructing a spatio-temporal processing block, including two parts: a cross-channel processing layer and a cross-frame processing layer; Step 2: the cross-frame processing layer: Step 2-1: using to represent the input of the cross-frame processing layer, using to represent the input feature of the cth channel of the cross-frame processing layer, c = 0, 1, …, C-1; T, C and N represent the number of time frames, the number of input channels and the number of features of each channel respectively; Step 2-2: using h to represent the number of attention heads, for the i th attention head, through linear transformation, the cross-frame query matrix with dimension E, the cross-frame key matrix and the cross-frame value matrix are obtained Wherein, and are learnable weight matrix and bias parameter respectively, * ∈ {Q, K, V}, d k = E / h; Step 2-3: the attention score of the i th attention head of the cross-frame processing layer is calculated Wherein P cf is the attention score from the previous cross-frame processing layer; Step 2-4: the attention score is connected through subspace to obtain the output of the attention layer Wherein, represents the weight matrix of the linear projection layer of the cross-frame processing layer; Step 2-5: input the feedforward neural network FFN with ReLU activation, and the output of the feedforward neural network FFN is the output of the current cross-frame processing layer; Step 3: the cross-channel processing layer; Step 3-1: using to represent the input of the cross-channel processing layer, wherein represents the input feature of t time, t = 0, 1, …, T-1; Step 3-2: for the i th attention head, through linear transformation, the cross-channel query matrix with dimension E, the cross-channel key matrix and the cross-channel value matrix are obtained Wherein, is a learnable weight matrix, is a bias parameter; Step 3-3: the output of the i th attention head, that is, the attention score is as follows: Wherein P cc is the attention score from the previous cross-channel processing layer; Step 3-4: the attention score is connected through subspace to obtain the output of the attention layer Wherein, represents the weight matrix of the linear projection layer; Step 3-5: input the feedforward neural network FFN with ReLU activation, and the output of the feedforward neural network FFN is the output of the current cross-channel processing layer; Step 4: adding multiple stacked spatio-temporal processing blocks before the pooling layer of the single-channel speaker verification system to realize the multi-channel speaker verification system; In the step 2-3, replace the Sparsemax operator with the softmax operator in the attention score: In the cross-channel processing layer, layer normalization is applied before the attention module and the FFN module respectively, and residual connection is applied between the input and output of the attention module and the FFN to alleviate the gradient vanishing problem of the cross-channel processing layer; In the cross-frame processing layer, layer normalization is applied before the attention module and the FFN module respectively, and residual connection is applied between the input and output of the attention module and the FFN to alleviate the gradient vanishing problem of the cross-frame processing layer.

Images