A speaker recognition method and system based on time domain residual layer
By introducing time-domain residual layer and transposed convolution in the ResNet model, an improved speaker embedding layer model is constructed, which solves the problems of limited recognition rate and large storage space in the existing technology, and realizes a more efficient speaker Identify.
Patent Information
- Application Number
- CN202411103547.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-13
AI Technical Summary
The existing speaker recognition system based on one-dimensional convolution needs to occupy a large amount of storage space when improving the recognition rate, while the two-dimensional convolution-based model has limited recognition rate due to time domain information loss.
The time domain residual layer is introduced in the ResNet model, and the time domain information is retained through the time domain residual module and transposed convolution, and combined with channel-by-channel convolution and pooling layers, an improved speaker embedding layer model is constructed.
It improves the recognition rate of speaker recognition, reduces storage space usage, and retains time domain information, improving the performance of the recognition system.
Smart Images

Figure CN118629409B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to speaker recognition technology, and in particular to a speaker recognition method and system based on a time domain residual layer. Background Art
[0002] Speaker recognition is a biometric technology. Compared with fingerprint recognition, facial recognition and other methods, it has the characteristics of non-contact and has unique advantages in practical applications.
[0003] Current speaker recognition systems often use speaker embedding layer models based on one-dimensional convolution, such as ECAPA-TDNN. However, in order to achieve a high recognition rate, speaker recognition systems based on this type of model need to retain more intermediate layer results and connect them by channel in the layer before the pooling layer, which takes up more storage space. In contrast, speaker embedding layer models based on two-dimensional convolution, especially ResNet (Residual Network) models, process data by layer and do not require much storage space during operation. However, due to the large loss of time domain information, the recognition rate is limited. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to provide a speaker recognition method and system based on a time-domain residual layer, and to apply the time-domain residual layer in a ResNet model to improve the recognition rate.
[0005] Technical solution: A speaker recognition method based on a time domain residual layer of the present invention comprises the following steps:
[0006] Prepare a training data set, where each item of data in the training data set contains a speech spectrum and a corresponding speaker label;
[0007] Construct a speaker recognition network, including a speaker embedding layer model and a speaker classification layer. After the speech spectrum is input into the speaker embedding layer model, a speaker embedding layer vector is obtained. The speaker embedding layer vector is input into the speaker classification layer to output the speaker classification result. Among them, the speaker embedding layer model includes a convolution layer, a residual layer, three time domain residual layers, a channel-by-channel convolution, a pooling layer, and a fully connected layer in sequence. The first module of the residual layer applies a downsampling residual module, and several residual modules are stacked subsequently. The first module of the time domain residual layer applies a time domain residual module, and several residual modules are stacked subsequently. Channel-by-channel convolution further compresses the frequency domain information, and the time domain size remains unchanged.
[0008] The training data set is used to train the speaker recognition network, and the trained speaker classification layer is removed to obtain the trained speaker embedding layer model;
[0009] The trained speaker embedding layer model is used to perform speaker recognition on real-time audio data to determine whether the speaker corresponding to the real-time audio data is the same as the registered speaker.
[0010] Furthermore, the speaker classification layer is implemented using a fully connected layer, whose input layer node number is consistent with the output node number of the speaker embedding layer model, and the output layer is the number of speakers in the training set.
[0011] Furthermore, the convolutional layers in the speaker embedding model are followed by batch normalization layers and linear rectifier unit layers.
[0012] Furthermore, the output channels of all convolutional layers in the residual layer and the temporal residual layer are the same.
[0013] Furthermore, the downsampling residual module includes two parallel paths formed by three convolutional layers. After the data passes through the downsampling residual module, the sizes of the time domain and frequency domain are reduced; the residual module includes two serial convolutional layers, and the size of the data does not change after passing through the residual module; based on the downsampling residual module, the time domain residual module adds a transposed convolutional layer at the bottom of the two parallel paths. After the data passes through the time domain residual module, the information is only compressed in the frequency domain, and the time domain information is not lost.
[0014] Furthermore, the training data set is used to train the speaker recognition network, and the trained speaker classification layer is removed to obtain the trained speaker embedding layer model; specifically:
[0015] The speech spectrum and speaker labels in the training data set are obtained in batches, the speech spectrum is used as the input of the speaker recognition network, the classification result of the speaker recognition network is calculated by forward propagation, and the cross entropy loss function value after comparison with the speaker label is calculated. The parameters of the speaker recognition network are updated by back propagation using the cross entropy loss function value until the loss function value converges to the preset range. The training is stopped, the trained speaker classification layer is removed, and the trained speaker embedding layer model is obtained.
[0016] Furthermore, the trained speaker embedding layer model is used to perform speaker recognition on the real-time audio data to determine whether the speaker corresponding to the real-time audio data is the same as the registered speaker; specifically, the following steps are performed:
[0017] (1) Obtain the speech spectrum of the registered speaker. The speech spectrum is passed through the trained speaker embedding layer model to obtain the registered speaker embedding layer vector;
[0018] (2) Obtain real-time audio data and extract spectrum features;
[0019] (3) Detect whether there is speech data in the audio data based on the spectral features; when the real-time audio data is not speech data, continue to obtain the real-time audio data and extract the spectral features; when the real-time audio data is speech data, input the spectral features into the trained speaker embedding layer model to obtain the speaker embedding layer vector of the real-time audio data;
[0020] (4) The cosine similarity between the speaker embedding layer vector of the real-time audio data and the speaker embedding layer vector of the registered speaker is calculated to determine whether they are the same speaker; when the cosine similarity exceeds a set threshold, it is considered that the speaker corresponding to the real-time audio data and the registered speaker are the same speaker, otherwise they are not; after the judgment is completed, return to step (2) and execute the step again.
[0021] Based on the same inventive concept, in one embodiment of the present invention, a speaker recognition system based on a time-domain residual layer includes:
[0022] A data set preparation unit, used for preparing a training data set, each data item in the training data set includes a speech spectrum and a corresponding speaker label;
[0023] A speaker recognition network construction unit is used to construct a speaker recognition network, including a speaker embedding layer model and a speaker classification layer. After the speech spectrum is input into the speaker embedding layer model, a speaker embedding layer vector is obtained. The speaker embedding layer vector is input into the speaker classification layer to output a speaker classification result. The speaker embedding layer model includes a convolution layer, a residual layer, three time domain residual layers, a channel-by-channel convolution, a pooling layer, and a fully connected layer in sequence. The first module of the residual layer applies a downsampling residual module, and several residual modules are subsequently stacked. The first module of the time domain residual layer applies a time domain residual module, and several residual modules are subsequently stacked. The channel-by-channel convolution further compresses the frequency domain information, and the time domain size remains unchanged.
[0024] A speaker recognition network training unit is used to train the speaker recognition network using a training data set, remove the trained speaker classification layer, and obtain a trained speaker embedding layer model;
[0025] The speaker recognition unit is used to perform speaker recognition on real-time audio data using a trained speaker embedding layer model to determine whether the speaker corresponding to the real-time audio data is the same as the registered speaker.
[0026] Based on the same inventive concept, in one embodiment of the present invention, an electronic device includes:
[0027] A memory storing executable program code;
[0028] a processor coupled to the memory;
[0029] The processor calls the executable program code stored in the memory to execute the steps of the speaker recognition method based on the time domain residual layer as described above.
[0030] Based on the same inventive concept, in one embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are called, the steps of the speaker recognition method based on the time domain residual layer are executed as described above.
[0031] Beneficial effect: Compared with the prior art, the advantages of the present invention are: the speaker embedding layer network adopted applies the time domain residual layer in the original ResNet structure, and more time domain information is retained. This is because the time domain residual module is applied in the time domain residual layer, and the transposed convolution in the module is used to restore the dimension of the data in the time domain to the dimension before applying the "s=2" convolution; since the time domain information is retained, the improved ResNet structure based on the time domain residual layer can improve the recognition rate of speaker recognition compared with the original ResNet structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a flow chart of the method of the present invention;
[0033] Figure 2 This is a schematic diagram of the speaker recognition network structure;
[0034] Figure 3 This is a schematic diagram of the speaker embedding layer model structure;
[0035] Figure 4 Schematic diagrams of the residual layer structure and the time-domain residual layer structure, where (a) is a schematic diagram of the residual layer structure, and (b) is a schematic diagram of the time-domain residual layer structure;
[0036] Figure 5 Schematic diagram of downsampling residual module;
[0037] Figure 6 Schematic diagram of residual module;
[0038] Figure 7 Schematic diagram of the time domain residual module. DETAILED DESCRIPTION
[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0040] like Figure 1 As shown, the present invention provides a speaker recognition method based on a time domain residual layer, comprising the following steps:
[0041] Step 1: Prepare a training data set. Each item in the training data set contains a speech spectrum and a corresponding speaker label. The speech length corresponding to the speech spectrum is relatively long, which can be more than 3 seconds; the number of speakers should be more than 1,000.
[0042] Step 2: Build a speaker recognition network, including a speaker embedding layer model and a speaker classification layer. The speech spectrum is input into the speaker embedding layer model to obtain the speaker embedding layer vector. The speaker embedding layer vector is input into the speaker classification layer to output the speaker classification result. Figure 2 As shown in the figure. The speaker embedding layer model is an improved ResNet model based on the time domain residual module. The speaker classification layer is implemented using a fully connected layer. The number of nodes in its input layer is consistent with the number of output nodes in the speaker embedding layer model. The output layer is the number of speakers in the training set.
[0043] The speaker embedding layer model is an improved ResNet model structure. Based on the original ResNet model, the residual layer is replaced by the time-domain residual layer. That is, the speaker embedding layer model is obtained by improving the ResNet model based on the time-domain residual module. The speaker embedding layer model structure is as follows: Figure 3 As shown in the figure, from top to bottom are the convolution layer conv3x3, the residual layer, 3 time domain residual layers, channel-by-channel convolution, the pooling layer and the fully connected layer. The convolution layer conv3x3 performs shallow nonlinear mapping of spectral features, and the output feature size obtained is the same as the input speech spectrum feature size; the residual layer and the time domain residual layer further nonlinearly map the features. The difference is that the residual layer compresses the time domain information, while the time domain residual layer does not compress; the channel-by-channel convolution is used to compress the frequency domain information; the pooling layer makes full use of the time domain information; the fully connected layer obtains the speaker embedding layer vector.
[0044] In the convolution layer conv3x3 (+BN+ReLU), c=64, s=1, pad=1, 3x3 means the size of the convolution layer is 3x3, c=64 means the number of output channels is 64, s=1 means the kernel width and height stride are both 1, pad=1 means the width and height are padded with zeros of 1, and (+BN+ReLU) means there are BN (batch normalization) layer and ReLU (Rectified Linear Unit) layer after the convolution.
[0045] The c of the residual layer and the time-domain residual layer is the number of output channels. The output channels of all convolutional layers in the residual layer and the time-domain residual layer are the same. The structure of the residual layer and the time-domain residual layer is as follows: Figure 4 (a) and Figure 4 As shown in (b), the first module of the residual layer applies the downsampling residual module, and several residual modules are stacked subsequently; the first module of the time domain residual layer applies the time domain residual module, and several residual modules are stacked subsequently.
[0046] like Figure 5As shown in the figure, the downsampling residual module consists of three convolutional layers conv3x3. After the data passes through the convolution with s=2 and pad=1, the width and height are reduced to 1 / 2 of the original size. After the convolution with s=1 and pad=1, the width and height remain unchanged. Therefore, after the data passes through the downsampling residual module, the size of the time domain and frequency domain is reduced to 1 / 2 of the original size. If the speaker embedding layer model uses the downsampling residual module, the time domain information will be lost too much. Figure 6 As shown in , the residual module consists of two convolutional layers conv3x3, both with s=1 and pad=1, so the size of the data does not change after passing through the residual module. Figure 7 As shown in the figure, based on the downsampling residual module, the time domain residual module adds a transposed convolution layer convTrans3x3 at the bottom of the two parallel paths, indicating a transposed convolution layer with a kernel size of 3x3, s=(1,2), pad=1, indicating that only the time domain is upsampled. After the data passes through the time domain residual module, the information is only compressed in the frequency domain, the size becomes 1 / 2, and the time domain information is not lost.
[0047] Channel-by-channel convolution further compresses the frequency domain information. After the data undergoes channel-by-channel convolution, the frequency domain size becomes 1, and the time domain size remains unchanged.
[0048] The pooling layer makes full use of the time domain information. After the data passes through the pooling layer, the dimensions of the time domain and frequency domain become 1.
[0049] Finally, the data passes through a fully connected layer and outputs a speaker embedding layer vector.
[0050] The speaker embedding layer vector is a fixed-length feature vector obtained after the speech spectrum passes through the speaker embedding layer model. This vector can represent the unique acoustic characteristics of the speaker; the length can be selected as 512.
[0051] Step 3: Train the speaker recognition network, remove the trained speaker classification layer, and obtain the trained speaker embedding layer model. Obtain the speech spectrum and speaker labels in the training data set in batches. Use the speech spectrum as the input of the speaker recognition network, forward propagate to calculate the classification result of the speaker recognition network, and calculate the cross entropy loss function value after comparing with the speaker label. Use the cross entropy loss function value to back propagate to update the parameters of the speaker recognition network until the loss function value converges to a certain range, stop training, remove the trained speaker classification layer, and obtain the trained speaker embedding layer model.
[0052] Step 4: Use the trained speaker embedding layer model to perform speaker recognition on real-time audio data.
[0053] Step 401: obtain the speech spectrum of the registered speaker, and pass the spectrum through the trained speaker embedding layer model obtained in step 3 to obtain the registered speaker embedding layer vector.
[0054] Step 402: Obtain real-time audio data and extract spectrum features.
[0055] Step 403: Input the spectrum features into an audio activity detector to detect whether there is speech data in the audio data. The detection algorithm may be a double threshold endpoint detection method based on short-time energy and short-time average zero-crossing rate.
[0056] Step 404: When the real-time audio data is not speech data, the system will continue to acquire the real-time audio data and extract spectral features; when the real-time audio data is speech data, the spectral features are input into the trained speaker embedding layer model obtained in step 3 to obtain the speaker embedding layer vector of the real-time audio data.
[0057] Step 405: Calculate the cosine similarity between the speaker embedding layer vector of the real-time audio data and the speaker embedding layer vector of the registered speaker to determine whether they are the same speaker. When the cosine similarity exceeds the set threshold, it is considered that the speaker corresponding to the real-time audio data is the same as the registered speaker, otherwise it is not. After the judgment is completed, return to step 402 and execute the steps again.
[0058] Based on the same inventive concept, the present invention also provides a speaker recognition system based on a time-domain residual layer, comprising:
[0059] A data set preparation unit, used for preparing a training data set, each data item in the training data set includes a speech spectrum and a corresponding speaker label;
[0060] A speaker recognition network construction unit is used to construct a speaker recognition network, including a speaker embedding layer model and a speaker classification layer. After the speech spectrum is input into the speaker embedding layer model, a speaker embedding layer vector is obtained. The speaker embedding layer vector is input into the speaker classification layer to output a speaker classification result. The speaker embedding layer model includes a convolution layer, a residual layer, three time domain residual layers, a channel-by-channel convolution, a pooling layer, and a fully connected layer in sequence. The first module of the residual layer applies a downsampling residual module, and several residual modules are subsequently stacked. The first module of the time domain residual layer applies a time domain residual module, and several residual modules are subsequently stacked. The channel-by-channel convolution further compresses the frequency domain information, and the time domain size remains unchanged.
[0061] A speaker recognition network training unit is used to train the speaker recognition network using a training data set, remove the trained speaker classification layer, and obtain a trained speaker embedding layer model;
[0062] The speaker recognition unit is used to perform speaker recognition on real-time audio data using a trained speaker embedding layer model to determine whether the speaker corresponding to the real-time audio data is the same as the registered speaker.
[0063] Based on the same inventive concept, the present invention further provides an electronic device, the device comprising:
[0064] A memory storing executable program code;
[0065] a processor coupled to the memory;
[0066] The processor calls the executable program code stored in the memory to execute the steps of the above-mentioned speaker recognition method based on the time domain residual layer.
[0067] Based on the same inventive concept, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the steps of the speaker recognition method based on the time domain residual layer as described above.
Claims
1. A speaker recognition method based on time domain residual layer, It is characterized in that The following steps are involved: Prepare a training data set, where each item of data in the training data set contains a speech spectrum and a corresponding speaker label; Construct a speaker recognition network, including a speaker embedding layer model and a speaker classification layer. After the speech spectrum is input into the speaker embedding layer model, a speaker embedding layer vector is obtained. The speaker embedding layer vector is input into the speaker classification layer to output the speaker classification result. The speaker embedding layer model includes a convolution layer, a residual layer, three time-domain residual layers, a channel-by-channel convolution, a pooling layer, and a fully connected layer in sequence. The first module of the residual layer applies a down-sampling residual module, and several residual modules are subsequently stacked. The first module of the time-domain residual layer applies a time-domain residual module, and several residual modules are subsequently stacked. Channel-by-channel convolution further compresses the frequency domain information, and the time domain size remains unchanged. The training data set is used to train the speaker recognition network, and the trained speaker classification layer is removed to obtain the trained speaker embedding layer model; The trained speaker embedding layer model is used to perform speaker recognition on real-time audio data to determine whether the speaker corresponding to the real-time audio data is the same as the registered speaker.
2. The speaker recognition method based on the time domain residual layer according to claim 1, It is characterized in that The speaker classification layer is implemented using a fully connected layer. The number of nodes in its input layer is consistent with the number of output nodes in the speaker embedding layer model, and the output layer is the number of speakers in the training set.
3. The speaker recognition method based on the time domain residual layer according to claim 1, It is characterized in that The convolutional layers in the speaker embedding model are followed by batch normalization and rectified linear unit layers.
4. The speaker recognition method based on the time domain residual layer according to claim 1, It is characterized in that The output channels of all convolutional layers in the residual layer and the temporal residual layer are the same.
5. The speaker recognition method based on the time domain residual layer according to claim 1, It is characterized in that The downsampling residual module includes two parallel paths formed by three convolutional layers. After the data passes through the downsampling residual module, the sizes of the time domain and frequency domain are reduced. The residual module includes two serial convolutional layers. The size of the data does not change after passing through the residual module. Based on the downsampling residual module, the time domain residual module adds a transposed convolutional layer at the bottom of the two parallel paths. After the data passes through the time domain residual module, the information is only compressed in the frequency domain, and the time domain information is not lost.
6. The speaker recognition method based on the time domain residual layer according to claim 1, It is characterized in that The speaker recognition network is trained using the training data set, and the trained speaker classification layer is removed to obtain the trained speaker embedding layer model; specifically: The speech spectrum and speaker labels in the training data set are obtained in batches, the speech spectrum is used as the input of the speaker recognition network, the classification result of the speaker recognition network is calculated by forward propagation, and the cross entropy loss function value after comparison with the speaker label is calculated. The parameters of the speaker recognition network are updated by back propagation using the cross entropy loss function value until the loss function value converges to the preset range. The training is stopped, the trained speaker classification layer is removed, and the trained speaker embedding layer model is obtained.
7. The speaker recognition method based on the time domain residual layer according to claim 1, It is characterized in that Use the trained speaker embedding layer model to perform speaker recognition on real-time audio data to determine whether the speaker corresponding to the real-time audio data is the same as the registered speaker; specifically, it includes: (1) Obtain the speech spectrum of the registered speaker. The speech spectrum is passed through the trained speaker embedding layer model to obtain the registered speaker embedding layer vector; (2) Obtain real-time audio data and extract spectrum features; (3) Detect whether there is speech data in the audio data based on the spectral features; when the real-time audio data is not speech data, continue to obtain the real-time audio data and extract the spectral features; when the real-time audio data is speech data, input the spectral features into the trained speaker embedding layer model to obtain the speaker embedding layer vector of the real-time audio data; (4) The cosine similarity between the speaker embedding layer vector of the real-time audio data and the speaker embedding layer vector of the registered speaker is calculated to determine whether they are the same speaker; when the cosine similarity exceeds a set threshold, it is considered that the speaker corresponding to the real-time audio data and the registered speaker are the same speaker, otherwise they are not; after the judgment is completed, return to step (2) and execute the step again.
8. A speaker recognition system based on time-domain residual layer, It is characterized in that include: A data set preparation unit, used for preparing a training data set, each data item in the training data set includes a speech spectrum and a corresponding speaker label; A speaker recognition network construction unit is used to construct a speaker recognition network, including a speaker embedding layer model and a speaker classification layer. After the speech spectrum is input into the speaker embedding layer model, a speaker embedding layer vector is obtained. The speaker embedding layer vector is input into the speaker classification layer to output a speaker classification result. The speaker embedding layer model includes a convolution layer, a residual layer, three time domain residual layers, a channel-by-channel convolution, a pooling layer, and a fully connected layer in sequence. The first module of the residual layer applies a downsampling residual module, and several residual modules are subsequently stacked. The first module of the time domain residual layer applies a time domain residual module, and several residual modules are subsequently stacked. The channel-by-channel convolution further compresses the frequency domain information, and the time domain size remains unchanged. A speaker recognition network training unit is used to train the speaker recognition network using a training data set, remove the trained speaker classification layer, and obtain a trained speaker embedding layer model; The speaker recognition unit is used to perform speaker recognition on real-time audio data using a trained speaker embedding layer model to determine whether the speaker corresponding to the real-time audio data is the same as the registered speaker.
9. An electronic device, It is characterized in that The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the steps of the speaker recognition method based on the time domain residual layer according to any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the steps of the speaker recognition method based on the time domain residual layer according to any one of claims 1 to 7.
