Speaker recognition method, device and equipment based on multi-domain acoustic feature fusion

Abstract

The invention relates to a speaker recognition method based on multi-domain acoustic feature fusion. The speaker recognition method comprises the following steps: acquiring voice data to be recognized; respectively extracting a spectrogram and a score spectrogram of each piece of voice data; enabling the spectrogram and the fractional spectrogram to pass through a filter bank to respectively obtain a first Fbank feature and a second Fbank feature, and combining the first Fbank feature and the second Fbank feature into an acoustic feature; fusing the acoustic features by adopting a feature fusion method; inputting the fused acoustic features into a speaker recognition network model to obtain voiceprint features of the voice of the speaker to be recognized; and obtaining a recognition result according to the voiceprint features of the voice of the speaker to be recognized. According to the method, high-precision and high-discrimination voiceprint features are obtained, and the speaker recognition accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of voiceprint recognition, and in particular relates to a speaker recognition method, device and equipment based on multi-domain acoustic feature fusion. Background Art

[0002] Speaker recognition is a biometric technology that can determine whether the registered and tested utterances belong to the same speaker. It has been widely used in financial payment, telecommunications anti-fraud, criminal investigation and other fields. Existing speaker recognition methods with excellent performance mainly focus on the construction and optimization of speaker embedding models, such as Thin-ResNet, ECAPA-TDNN and MFA-conformer.

[0003] In the whole speaker recognition task, it is necessary to extract acoustic features from the input speech before training the speaker embedding model. However, most of the acoustic features used in the best speaker recognition methods currently convert time domain signals into time-frequency domain signals, such as Mel-frequency cepstral coefficients (MFCC) or filter banks (FBank). There are other common acoustic features in the time-frequency domain, such as linear predictive coding (LPC), perceptual linear prediction (PLP), subband-based cepstral coefficients (SBC), and wavelet transform features. There are also some that directly use the original waveform of speech data to train speaker recognition models.

[0004] However, the current research and exploration of acoustic features for speaker recognition (SR) tasks is relatively limited. Existing methods only use part of the information in the time domain or frequency domain, while a large part of the information between the time domain and the frequency domain has not been used. This unused information may also contain information related to speaker characteristics. If this information can be fully mined to construct better acoustic features, the effect of speaker recognition methods will be significantly improved.

[0005] Research has pointed out that the fractional domain can represent the information between the time domain and the frequency domain. Similar to the frequency domain signal obtained by Fourier transform (FT), the fractional domain signal can be obtained by fractional Fourier transform (FRFT). On the time-frequency plane, if the horizontal axis is regarded as the time axis and the vertical axis is regarded as the frequency axis, the traditional Fourier transform can be regarded as the process of rotating the signal 90 degrees counterclockwise from the time axis to the frequency axis. FRFT can be rotated at any angle, so it is regarded as a generalized form of FT. By adjusting the rotation angle, FRFT enhances the application scope of traditional FT, enabling it to cope with more diverse signal processing problems. Summary of the invention

[0006] The purpose of the present invention is to provide a speaker recognition method, device and equipment based on multi-domain acoustic feature fusion, which can solve the problem that a large amount of information between the time domain and frequency domain of acoustic features in the existing speaker recognition method is not utilized;

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A speaker recognition method based on multi-domain acoustic feature fusion includes the following steps:

[0009] Acquire voice data to be recognized;

[0010] Extract the spectrogram and fractional spectrogram of each speech data respectively;

[0011] The spectrogram and the fractional spectrogram are passed through a filter bank to obtain a first Fbank feature and a second Fbank feature respectively, and the first Fbank feature and the second Fbank feature are combined into an acoustic feature;

[0012] The acoustic features are fused using feature fusion method;

[0013] The fused acoustic features are input into the speaker recognition network model to obtain the voiceprint features of the speaker to be recognized;

[0014] The recognition result is obtained based on the voiceprint features of the speaker to be recognized.

[0015] Preferably, the extracting the spectrogram and fractional spectrogram of each speech data respectively comprises: extracting the spectrogram and fractional spectrogram of each speech data respectively by short-time Fourier transform and short-time fractional Fourier transform.

[0016] Preferably, the discrete representation of the fractional spectrum graph is:

[0017]

[0018] X p The representation of (i, k) is:

[0019]

[0020] Where p represents the fractional order, and 0≤p<1, preferably p is 0.5, K p (m, k) is the kernel function of the fractional Fourier transform, i represents the i-th frame of the speech data, k represents the k-th frequency interval, T is the total number of frames of the speech data, B represents the number of frequency intervals, m is the m-th sampling point in a frame, and 0≤m<N, N is the number of sampling points in a frame, w(·) is the window function, u i (·) represents the signal of the i-th frame in U, and h represents the interval between two consecutive frames.

[0021] Preferably, the dimension of the acoustic feature can be expressed as:

[0022] (2, T, F)

[0023] Among them, 2 represents two feature channels consisting of the first Fbank feature and the second Fbank feature, T represents the number of frames, and F represents the dimension of the acoustic features of each frame.

[0024] Preferably, the feature fusion method for fusing acoustic features includes: using a convolution module to expand the dimension of the acoustic features from (2, T, F) to (C, T, F), that is, using C convolution kernels to learn the features of two input channels.

[0025] Preferably, the fusing of acoustic features by using a feature fusion method includes: using a convolution module to compress the dimension of the acoustic features from (2, T, F) to (1, T, F), that is, fusing the features of two feature channels.

[0026] Preferably, the speaker recognition network model is an MFA-conformer model or an ECAPA-TDNN model.

[0027] According to another aspect of the present invention, there is also provided a speaker recognition device based on multi-domain acoustic feature fusion, comprising:

[0028] An acquisition module, used for acquiring voice data to be recognized;

[0029] An extraction module, used for respectively extracting a spectrogram and a fractional spectrogram of each speech data;

[0030] A first processing module is used for passing the spectrum graph and the fractional spectrum graph through a filter bank to obtain a first Fbank feature and a second Fbank feature, and combining the first Fbank feature and the second Fbank feature into an acoustic feature;

[0031] A fusion module, used to fuse acoustic features using a feature fusion method;

[0032] The second processing module is used to input the fused acoustic features into the speaker recognition network model to obtain the voiceprint features of the speech of the speaker to be recognized;

[0033] The recognition module is used to obtain the recognition result according to the voiceprint features of the speaker to be recognized.

[0034] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned speaker recognition method based on multi-domain acoustic feature fusion are performed.

[0035] According to another aspect of the present invention, there is also provided a speaker recognition device based on multi-domain acoustic feature fusion, characterized in that it comprises:

[0036] Memory, for storing software applications,

[0037] The processor is used to execute the software application program, and each program of the software application program correspondingly executes the steps in the above-mentioned speaker recognition method based on multi-domain acoustic feature fusion.

[0038] The present invention extracts the spectrogram and fractional spectrogram of speech data by short-time Fourier transform and short-time fractional Fourier transform, fuses the spectrogram features and fractional spectrogram features, and proposes two multi-domain acoustic feature fusion methods (MAFF1 and MAFF2). Since the fractional spectrograms generated by the fractional Fourier transform using different orders p will have a significant impact on the performance, the present invention trains each order with a p value between 0 and 0.9 and an interval of 0.1, and selects the order with the best performance to obtain the final model. The present invention obtains voiceprint features with high precision and high discrimination, and solves the problem that a large amount of information between the time domain and the frequency domain of acoustic features in the existing speaker recognition method is not utilized, thereby improving the accuracy of speaker recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0041] Example 1

[0042] Before implementing the speaker recognition method of the present invention, the speaker recognition method of the present invention is first used to train a speaker recognition network model. The training method of the speaker recognition network model includes:

[0043] S1. Obtain a speech data set;

[0044] Specifically, this embodiment uses the VoxCeleb dataset as the training set. The VoxCeleb dataset consists of two sub-datasets, VoxCelebl and VoxCeleb2. The VoxCeleb dataset is derived from real scenes in a natural environment. The audio is taken from the English voice on the YouTube website. The voice has a certain amount of real noise, and the noise occurs at irregular times, including sudden environmental noise, background human voices, laughter, voice aliasing, echo, indoor noise, recording equipment noise, etc.

[0045] S2. Preprocess the speech data U in the speech data set and extract the spectrogram X and fractional spectrogram X of the preprocessed speech data U.p ,

[0046] Specifically, the speech data in the training set are divided into several batches of equal size, each batch containing K speech data. For each speech in a batch of data, this embodiment randomly selects a speech segment with a length of 3s, divides the segment into frames with a frame length of 25ms and a frame shift of 10ms, and uses a Hamming window for windowing.

[0047] More specifically, a 512-dimensional short-time Fourier transform (STFT) and a short-time fractional Fourier transform (STFRFT) are calculated for each frame to form a spectrogram X and a fractional spectrogram X, respectively. p ,

[0048] The discrete representation of the spectrum graph X is:

[0049]

[0050] Where i represents the i-th frame of the voice data U, k represents the k-th frequency interval, T is the total number of frames of the voice data, and B represents the number of frequency intervals.

[0051] X(i, k) is expressed as:

[0052]

[0053] In X(i, k), m is the mth sampling point in a frame, and 0≤m<N, N is the number of sampling points in a frame, w(·) is the window function, u i (·) represents the signal of the i-th frame in U, and h represents the interval between two consecutive frames;

[0054] The fractional spectrum X p The discrete representation of is:

[0055]

[0056] Where p represents the fractional order, and 0≤p<1, preferably p is 0.5, X p The representation of (i, k) is:

[0057]

[0058] X p In (i, k), K p (m, k) is the kernel function of the fractional Fourier transform, and the meanings of other symbols are the same as in X(i, k);

[0059] In this embodiment, B=512, N=400, and h=160 are set.

[0060] The relationship between the fractional order p and the rotation angle α in the time-frequency plane can be expressed by the equation When p = 1, This means that FRFT becomes FT, and STFRFT becomes STFT. When p = 1, formula (3) becomes formula (1), formula (4) becomes formula (2), FRFT becomes FT, and STFRFT degenerates into STFT.

[0061] S3, the spectrum graph and the fractional spectrum graph are filtered by the filter bank to obtain the first Fbank feature F x and the second Fbank feature F p , and combined into acoustic features,

[0062] Specifically, in this embodiment, an 80-dimensional filter bank (FBank) is calculated on the spectrum graph and the fractional spectrum graph to obtain the first Fbank feature F x and the second Fbank feature F p , and combined into acoustic features;

[0063] The dimension of acoustic features can be expressed as:

[0064] (2, T, F)

[0065] Where 2 represents the first Fbank feature F x and the second Fbank feature F p The two feature channels are composed of T, F, and the dimension of the acoustic features of each frame, which are set to (2, 301, 80) in this embodiment, and the batch size is 200.

[0066] S4, using feature fusion method MAFF1 to fuse acoustic features;

[0067] Specifically, the feature fusion method MAFF1 uses a convolution module to expand the dimension of acoustic features from (2, T, F) to (C, T′, F′), which means using C convolution kernels to learn the features of two input channels from different perspectives. In this embodiment, C=256, f′=150 and F′=39 are set.

[0068] S5. Input the fused acoustic features into the speaker recognition network model to obtain several voiceprint features.

[0069] Specifically, the fused acoustic features are input into the MFA-Conformer model to obtain the final voiceprint feature e i , i = 1, …, K; where MFA-Conformer is a state-of-the-art speaker recognition method, in which Conformer blocks are used to extract local and global features simultaneously, and 6 Conformer blocks are connected in series to aggregate multi-scale representations;

[0070] S6. Construct a loss function based on the voiceprint features and audio tags;

[0071] Specifically, in this embodiment, the AM-Softmax loss function is used, and the AM-Softmax loss function is expressed as:

[0072]

[0073] Among them, m is the boundary value, which is a scalar hyperparameter, s is the scale factor that helps stabilize the training, and y i is the index of the true label;

[0074] More specifically, in this embodiment, m=0.2, s=30;

[0075] S7, using the loss function to train the speaker recognition network model;

[0076] Specifically, train all data to minimize the loss function L c , get the trained speaker recognition network model;

[0077] After the speaker recognition network model is trained, a speaker recognition method based on multi-domain acoustic feature fusion of the present invention is used to perform speaker recognition. The speaker recognition method comprises the following steps:

[0078] S8, obtaining voice data to be recognized;

[0079] Specifically, two speech sounds to be compared are obtained. In this embodiment, the speech sounds to be compared are from a test set, which is a VoxCeleb1-O and SITW dataset.

[0080] S9, extract the spectrogram X and fractional spectrogram X of each speech data respectively p ;

[0081] S10, the spectrum graph and the fractional spectrum graph are passed through the filter bank to obtain the first Fbank feature F x and the second Fbank feature F p , and the first Fbank feature F x and the second Fbank feature F p Combined into acoustic features;

[0082] S11, using feature fusion method to fuse acoustic features;

[0083] S12, inputting the fused acoustic features into a speaker recognition network model to obtain voiceprint features of the speech of the speaker to be recognized;

[0084] Specifically, the fused acoustic features are input into a pre-trained speaker recognition network model to obtain the voiceprint feature representation of the two voices. 1 , e 2 ;

[0085] S12. Obtain a recognition result according to the voiceprint features of the speech of the speaker to be recognized.

[0086] Specifically, the following formula is used to compare the voiceprint features e of the two speakers' voices: 1 and e 2 , perform similarity scoring, and obtain the judgment result of whether the two voices to be compared are from the same person.

[0087]

[0088] The speaker recognition method provided in this embodiment has the same implementation principle and technical effect as the above training method, and will not be described in detail here.

[0089] According to another aspect of the present invention, there is also provided a speaker recognition device based on multi-domain acoustic feature fusion, comprising:

[0090] An acquisition module, used for acquiring voice data to be recognized;

[0091] An extraction module, used for respectively extracting a spectrogram and a fractional spectrogram of each speech data;

[0092] A first processing module is used for passing the spectrum graph and the fractional spectrum graph through a filter bank to obtain a first Fbank feature and a second Fbank feature, and combining the first Fbank feature and the second Fbank feature into an acoustic feature;

[0093] A fusion module, used to fuse acoustic features using a feature fusion method;

[0094] The second processing module is used to input the fused acoustic features into the speaker recognition network model to obtain the voiceprint features of the speech of the speaker to be recognized;

[0095] The recognition module is used to obtain the recognition result according to the voiceprint features of the speaker to be recognized.

[0096] The implementation principle and technical effect of the speaker recognition device provided by the embodiment of the present invention are similar to those of the above embodiment, and will not be described in detail here.

[0097] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned speaker recognition method based on multi-domain acoustic feature fusion are performed.

[0098] According to another aspect of the present invention, there is also provided a speaker recognition device based on multi-domain acoustic feature fusion, comprising:

[0099] Memory, for storing software applications,

[0100] The processor is used to execute the software application program, and each program of the software application program correspondingly executes the steps in the above-mentioned speaker recognition method based on multi-domain acoustic feature fusion.

[0101] Example 2

[0102] The speaker recognition method of the present invention is used to train a speaker recognition network model, and the training method of the speaker recognition network model includes:

[0103] S1. Obtain a speech data set;

[0104] Specifically, this embodiment uses the VoxCeleb dataset as the training set. The VoxCeleb dataset consists of two sub-datasets, VoxCeleb1 and VoxCeleb2. The VoxCeleb dataset is derived from real scenes in a natural environment. The audio is taken from the English voice on the YouTube website. The voice contains certain real noise, and the noise occurs at irregular times, including sudden environmental noise, background human voices, laughter, voice aliasing, echo, indoor noise, recording equipment noise, etc.

[0105] S2. Preprocess the speech data U in the speech data set and extract the spectrogram X and fractional spectrogram X of the preprocessed speech data U. p ,

[0106] Specifically, the speech data in the training set are divided into several batches of equal size, each batch containing K speech data. For each speech in a batch of data, this embodiment randomly selects a speech segment with a length of 3s, divides the segment into frames with a frame length of 25ms and a frame shift of 10ms, and uses a Hamming window for windowing.

[0107] More specifically, a 512-dimensional short-time Fourier transform (STFT) and a short-time fractional Fourier transform (STFRFT) are calculated for each frame to form a spectrogram X and a fractional spectrogram X, respectively. p ,

[0108] The discrete representation of the spectrum graph X is:

[0109]

[0110] Where i represents the i-th frame of the voice data U, k represents the k-th frequency interval, T is the total number of frames of the voice data, B represents the number of frequency intervals, and X(i, k) is expressed as:

[0111]

[0112] In X(i, k), m is the mth sampling point in a frame, and 0≤m<N, N is the number of sampling points in a frame, w(·) is the window function, u i (·) represents the signal of the i-th frame in U, and h represents the interval between two consecutive frames;

[0113] The fractional spectrum X p The discrete representation of is:

[0114]

[0115] Where p represents the fractional order, and 0≤p<1, preferably p is 0.5, X p The representation of (i, k) is:

[0116]

[0117] X p In (i, k), X p (m, k) is the kernel function of the fractional Fourier transform, and the meanings of other symbols are the same as in X(i, k);

[0118] In this embodiment, B=512, N=400, and h=160 are set.

[0119] The relationship between the fractional order p and the rotation angle α in the time-frequency plane can be expressed by the equation When p = 1, This means that FRFT becomes FT, and STFRFT becomes STFT. When p = 1, formula (3) becomes formula (1), formula (4) becomes formula (2), FRFT becomes FT, and STFRFT degenerates into STFT.

[0120] S3, the spectrum graph and the fractional spectrum graph are filtered by the filter bank to obtain the first Fbank feature F x and the second Fbank feature F p , and combined into acoustic features,

[0121] Specifically, in this embodiment, an 80-dimensional filter bank (FBank) is calculated on the spectrum graph and the fractional spectrum graph to obtain the first Fbank feature F x and the second Fbank feature F p , and combined into acoustic features;

[0122] The dimension of acoustic features can be expressed as:

[0123] (2, T, F)

[0124] Where 2 represents the first Fbank feature F x and the second Fbank feature F p The two feature channels are composed of T, F, and the dimension of the acoustic features of each frame, which are set to (2, 301, 80) in this embodiment, and the batch size is 200.

[0125] S4, using feature fusion method MAFF2 to fuse acoustic features;

[0126] Specifically, the feature fusion method MAFF2 uses a convolution module to compress the dimension of the acoustic feature from (2, T, F) to (1, T′, F′), that is, to fuse the features of the two channels and input them into the subsequent network. In this embodiment, T′=301 and F′=80 are set.

[0127] S5. Input the fused acoustic features into the speaker recognition network model to obtain several voiceprint features.

[0128] Specifically, the fused acoustic features are input into the ECAPA-TDNN model to obtain the final voiceprint feature e i , i = 1, ..., K; where ECAPA-TDNN is a classic speaker recognition method that achieves significant performance improvement by integrating Squeeze-Excitation blocks, multi-scale Res2Net features, additional skip connections, and channel-dependent attention statistics pooling. In this embodiment, ECAPA-TDNN uses two parameters, C = 512 and C = 1024, where C represents the number of channels in the model.

[0129] S6. Construct a loss function based on the voiceprint features and audio tags;

[0130] Specifically, in this embodiment, the AM-Softmax loss function is used, and the AM-Softmax loss function is expressed as:

[0131]

[0132] Among them, m is the boundary value, which is a scalar hyperparameter, s is the scale factor that helps stabilize the training, and y i is the index of the true label;

[0133] More specifically, in this embodiment, m=0.2, s=30;

[0134] S7, using the loss function to train the speaker recognition network model;

[0135] Specifically, train all data to minimize the loss function L c , get the trained speaker recognition network model;

[0136] After the speaker recognition network model is trained, a speaker recognition method based on multi-domain acoustic feature fusion of the present invention is used to perform speaker recognition. The speaker recognition method comprises the following steps:

[0137] S8, obtaining voice data to be recognized;

[0138] Specifically, two speech sounds to be compared are obtained. In this embodiment, the speech sounds to be compared are from a test set, which is a VoxCeleb1-O and SITW dataset.

[0139] S9, extract the spectrogram X and fractional spectrogram X of each speech data respectively p ;

[0140] S10, the spectrum graph and the fractional spectrum graph are passed through the filter bank to obtain the first Fbank feature F x and the second Fbank feature F p , and the first Fbank feature F x and the second Fbank feature F p Combined into acoustic features;

[0141] S11, using feature fusion method to fuse acoustic features;

[0142] S12, inputting the fused acoustic features into a speaker recognition network model to obtain voiceprint features of the speech of the speaker to be recognized;

[0143] Specifically, the fused acoustic features are input into a pre-trained speaker recognition network model to obtain the voiceprint feature representation of the two voices. 1 , e 2 ;

[0144] S12. Obtain a recognition result according to the voiceprint features of the speech of the speaker to be recognized.

[0145] Specifically, the following formula is used to compare the voiceprint features e of the two speakers' voices: 1 and e 2 , perform similarity scoring, and obtain the judgment result of whether the two voices to be compared are from the same person.

[0146]

[0147] The speaker recognition method provided in this embodiment has the same implementation principle and technical effect as the above training method, and will not be described in detail here.

[0148] The difference between Example 1 and Example 2 is that Example 1 uses the feature fusion method MAFF1 to fuse acoustic features. MAFF1 uses a convolution module to expand the feature dimension from (2, T, F) to (C, T′, F′), that is, C convolution kernels are used to learn the features of two input channels from different angles. MAFF1 is suitable for methods that require a new dimension to represent channel C. The baseline model of the speaker recognition network uses MFA-Conformer.

[0149] In Example 2, the feature fusion method MAFF2 is used to fuse acoustic features. MAFF2 uses a convolution module to compress the feature dimension from (2, T, F) to (1, T′, F′), that is, the features of the two channels are fused and then input into the subsequent network. MAFF2 is suitable for methods that directly expand the dimension F to C channels without adding additional dimensions. The baseline model of the speaker recognition network is ECAPA-TDNN.

[0150] The present invention uses equal error rate (Equal Error Rate, EER) and minimum detection cost function (Minimum Detection Cost Function, minDCF) as performance indicators to test the effect of the present invention.

[0151] The formula of minDCF is as follows:

[0152] minDCF= Cmiss ×P targe t×FRR+C a ×(1-P target )×FAR

[0153] Among them C miss and C a They represent the weights of false rejection and false acceptance, both set to 1, P target and 1-P target Represent the prior probabilities of the real speaker and the impostor, P target Set to 0.01.

[0154] In order to verify the effectiveness of the multi-domain fusion method, the present invention compares the following results: using only the FBank feature F extracted from the time-frequency domain x , FBank features F extracted from the score domain p , and the wavelet features F extracted using wavelet transform w , and the results obtained by fusing these features.

[0155] The MFA-Conformer model and MAFF 1 fusion strategy in Example 1 are used for training, and the FRFT order is set to p = 0.5. Table 1 shows the experimental results of VoxCeleb1-O without as-norm. The experimental results show that when the score feature F p When the wavelet feature F is used alone, the performance is poor. p and frequency domain feature F x When combined, the performance is significantly improved. However, the wavelet feature F w Even when fused with the other two features, it does not provide substantial benefits and the performance improvement is not obvious. Therefore, the results show that the method of fusing spectrogram features and fractional spectrogram features achieves the best performance, which demonstrates the necessity and effectiveness of the proposed method.

[0156] Table 1 Results of different transformations on the VoxCeleb1-O dataset

[0157]

[0158] In order to explore the impact of order p on performance, the present invention uses different p values ​​to generate fractional spectrograms, and uses the fusion strategy MAFF1 in Example 1 to fuse the fractional spectrogram features corresponding to order p with the spectrogram features obtained by short-time Fourier transform. Then, the obtained acoustic features are input into the MFA-Conformer model for training.

[0159] The test results without as-norm on VoxCeleb1-O are shown in Table 2. The score spectrogram for p=1 is not listed because it is equivalent to the traditional spectrogram. The test results of the baseline model MFA-Conformer are shown in the first row, and the test results of the MFA-Conformer+MAFF1 model at different p values ​​are shown in the remaining rows. The results show that the performance is best when p=0.5.

[0160] Table 2 Results of different p on the VoxCeleb1-O dataset

[0161]

[0162] In order to explore the impact of fusion strategy on performance, the present invention uses different fusion strategies for multi-domain acoustic features, that is, feature fusion is performed at different positions in the model. This experiment is conducted on the MFA-Conformer model.

[0163] The first fusion position is at the beginning of the model, where the feature with dimension (2,301,80) is input to the convolutional subsampling layer for fusion, and the result is (256,150,39), which is the MAFF1 method. The second position is after the convolutional subsampling layer, and the convolution module is used to convert channel 2 to 1. The third position is after the linear Drop out layer, and channel 2 is fused with 256 feature channels and 39-dimensional features. The fourth position is before the attention statistical pooling layer (ASP), and the convolution module is used to convert channel 2 to 1.

[0164] The test results without using as-norm on VoxCeleb1-O are shown in Table 3. The experiment shows that the fusion effect at the beginning of the model is the best. Therefore, the feature fusion position of the multi-domain feature fusion method of the present invention is at the beginning of the model by default.

[0165] Table 3 Results of different fusion strategies on the VoxCeleb1-O dataset

[0166]

[0167] The present invention applies the MAFF method to different baseline models, where ECAPA+MAFF2 means that the present invention uses MAFF2 on the ECAPA-TDNN model, and MFA+MAFF1 means that the present invention uses MAFF1 on the MFA-Conformer model.

[0168] The test results of each method using as-norm on the VoxCeleb1-O and SITW.Eval datasets are shown in Table 4. The experimental results show that the proposed method outperforms the baseline model and improves the equal error rate (EER) by 9% and 17% on average compared with MFA-Conformer, ECAPA-TDNN (C=512) and ECAPA-TDNN (C=1024) on the VoxCeleb1-O and SITW.Eval datasets, respectively.

[0169] Table 4 Results of all methods on VoxCeleb1-O and SITW.Eval datasets

[0170]

[0171]

[0172] In this paper, we propose a speaker recognition method based on multi-domain acoustic feature fusion, by using fractional Fourier transform to calculate the fractional spectrogram, and the acoustic features extracted from the traditional spectrogram and the fractional spectrogram are fused through two proposed fusion methods to make full use of the information in the time domain, frequency domain and fractional domain. Experimental results show that p = 0.5 is a good order for extracting fractional spectrogram features, and the proposed method outperforms the baseline model.

Claims

1. A speaker recognition method based on multi-domain acoustic feature fusion, characterized in that: The following steps are involved: Acquire voice data to be recognized; Extract the spectrogram and fractional spectrogram of each speech data respectively; The spectrogram and the fractional spectrogram are passed through a filter bank to obtain a first Fbank feature and a second Fbank feature respectively, and the first Fbank feature and the second Fbank feature are combined into an acoustic feature; The acoustic features are fused using feature fusion method; The fused acoustic features are input into the speaker recognition network model to obtain the voiceprint features of the speaker to be recognized; The recognition result is obtained based on the voiceprint features of the speaker to be recognized.

2. The method according to claim 1, characterized in that The extracting the spectrogram and fractional spectrogram of each speech data respectively comprises: extracting the spectrogram and fractional spectrogram of each speech data respectively by short-time Fourier transform and short-time fractional Fourier transform.

3. The method according to claim 1 or 2, characterized in that: The discrete representation of the fractional spectrogram is: X p The representation of (i,k) is: where p represents the fractional order and 0 ≤ p < 1, K p (m, k) is the kernel function of the fractional Fourier transform, i represents the i-th frame of the speech data, k represents the k-th frequency interval, T is the total number of frames of the speech data, B represents the number of frequency intervals, m is the m-th sampling point in a frame, and 0 ≤ m < N, where N is the number of sampling points in a frame, w(·) is the window function, u i (·) represents the signal of the i-th frame in U, and h represents the interval between two consecutive frames.

4. The method according to claim 1, characterized in that: The dimension of the acoustic feature can be expressed as: (2, T, F) Among them, 2 represents two feature channels consisting of the first Fbank feature and the second Fbank feature, T represents the number of frames, and F represents the dimension of the acoustic features of each frame.

5. The method according to claim 4, characterized in that The feature fusion method for fusing acoustic features includes: using a convolution module to expand the dimension of the acoustic features from (2, T, F) to (C, T, F), that is, using C convolution kernels to learn the features of two input channels.

6. The method according to claim 4, characterized in that The feature fusion method for fusing acoustic features includes: using a convolution module to compress the dimension of the acoustic features from (2, T, F) to (1, T, F), that is, fusing the features of two feature channels.

7. The method according to claim 4, characterized in that The speaker recognition network model is an MFA-conformer model or an ECAPA-TDNN model.

8. A speaker recognition device based on multi-domain acoustic feature fusion, characterized in that: include: An acquisition module, used for acquiring voice data to be recognized; An extraction module, used for respectively extracting a spectrogram and a fractional spectrogram of each speech data; A first processing module is used for passing the spectrum graph and the fractional spectrum graph through a filter bank to obtain a first Fbank feature and a second Fbank feature, and combining the first Fbank feature and the second Fbank feature into an acoustic feature; A fusion module, used to fuse acoustic features using a feature fusion method; The second processing module is used to input the fused acoustic features into the speaker recognition network model to obtain the voiceprint features of the speech of the speaker to be recognized; The recognition module is used to obtain the recognition result according to the voiceprint features of the speaker to be recognized.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speaker recognition method based on multi-domain acoustic feature fusion described in any one of claims 1 to 7 are performed.

10. A speaker recognition device based on multi-domain acoustic feature fusion, characterized in that: include: Memory, for storing software applications, A processor is used to execute the software application, each program of the software application correspondingly executes the steps in the speaker recognition method based on multi-domain acoustic feature fusion as described in any one of claims 1 to 7.

Images