End-to-end speaker recognition using deep neural network

Abstract

To provide a system that trains a front-end feature extractor using a deep neural network (DNN) with a triplet network architecture.SOLUTION: A DNN includes three feedforward neural networks trained according to a batch processing that utilizes a cohort set of negative training samples. After each batch of training samples is processed, the DNN is trained according to a loss function that utilizes, for example, a cosine measure of similarity between respective samples, along with positive and negative margins, to provide a robust representation of voiceprints.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] This application claims priority to U.S. Non-provisional Patent Application No. 15 / 262,748, filed Sep. 12, 2016, the entire disclosure of which is incorporated herein by reference. The present invention is directed to speech recognition, including speaker verification and / or speaker identification. Additionally, the present invention may be used to perform text-independent speaker recognition. [Background technology]

[0002] Current state-of-the-art approaches to speaker recognition are based on Universal Background Models (UBMs) estimated using either acoustic Gaussian Mixture Models (GMMs) (see Douglas A. Reynolds et al., “Speaker Verification Using Adapted Gaussian Mixture Models,” Digital Signal Processing, 2000, the entire contents of which are incorporated herein by reference), or speech recognition deep neural network architectures (see Y. Lei et al., “A Novel Scheme for Speaker Recognition Using a Phonetically-Aware Deep Neural Network,” Proceedings of ICASSP 2014, the entire contents of which are incorporated herein by reference). The most effective technique consists of fitting a UBM model to every conversational utterance using the total variation paradigm (see N. Dehak et al., “Front-End Factor Analysis for Speaker Verification,” IEEE Transactions on Audio, Speech, and Language Processing, Vol. 19, No. 4, pp. 788-798, May 2011, the entire contents of which are incorporated herein by reference). The total variation paradigm aims to extract a low-dimensional feature vector, known as the “i-vector,” that preserves all information about the speaker and the channel. After the application of channel compensation techniques, the resulting i-vector can be considered as the speaker's voiceprint or voice signature.

[0003] The main drawback of such approaches is that by using only handcrafted features designed to reproduce the human perception system, they tend to discard useful information that is important for recognizing or verifying a speaker. Typically, the above approaches utilize low-order features such as Mel-frequency Cepstral Coefficients (MFCCs) and attempt to fit them to a Gaussian distribution constant (typically 1024 or 2048 Gaussians). This makes it difficult to model complex structures in the feature space where the Gaussian assumption does not necessarily hold. Summary of the Invention

[0004] The present invention is directed to a system that utilizes a deep neural network with a triplet network architecture to train a front-end feature extractor for use in performing the task of speaker identity verification or speaker identification among a closed set of known speakers.

[0005] According to a representative embodiment, the system comprises a memory device and a processor-based device. The memory device stores speech samples, including a dual set of same-speaker speech samples, a cohort set of speech samples not from the dual set and the same speaker, and a speaker model set. The processor-based device is further configured to model a deep neural network having a triplet network architecture. The processor-based device trains the deep neural network according to a batch process, in which the dual set of speech samples is fed through the deep neural network in combination with the cohort set of speech samples.

[0006] According to a further representative embodiment, the deep neural network may include a first feedforward neural network that receives and processes a first input to generate a first output, a second feedforward neural network that receives and processes a second input to generate a second output, and a third feedforward neural network that receives and processes a third input to generate a third output. Also, for each of the multiple speakers, the memory device stores a first set of P speech samples by the speaker. JPEG2025065135000002.jpg736 and a second set of P conversation samples by the speaker JPEG2025065135000003.jpg839 and P is an integer equal to or greater than 1. The deep neural network is trained by the processor-based device, whereby, for each of the multiple speakers, the deep neural network performs a batch processing during which a first set of corresponding speech samples is fed through a first feedforward neural network, a second set of corresponding speech samples is fed through a second feedforward neural network, and a cohort set of speech samples is fed through a third feedforward neural network. Upon completion of the batch processing, a loss function is calculated based on the first network output, the second network output, and the third network output obtained based on the first set of corresponding speech samples, the second set of corresponding speech samples, and the cohort set of speech samples, respectively. The calculated loss function is used to modify the connection weights of each of the first, second, and third feedforward neural networks by a backpropagation method.

[0007] According to a further representative embodiment, the loss function is JPEG2025065135000004.jpg43 and one of the second set of corresponding speech samples. JPEG2025065135000005.jpg55 The similarity S between the second network output and + The corresponding positive distance d + and one from the first set of conversation samples. JPEG2025065135000006.jpg43 The similarity S between the first network output according to and the most similar one of the third network outputs according to each conversation sample of the cohort set - The corresponding negative distance d - Furthermore, the positive distance d + and the negative distance d - is the corresponding similarity S + , S - Each margin M is different. + , M - In particular, the loss function can be determined by applying

number

number

number

number

[0008] According to an alternative representative embodiment, the loss function may be related to an Equal Error Rate (EER) metric. In this case, the loss function is

number

[0009] According to a representative embodiment, each of the feedforward neural networks employed in the triplet network architecture may include at least one convolutional layer, at least one max-pooling layer, and a fully connected layer.

[0010] Furthermore, in one representative embodiment, the present invention may be used to perform a speaker verification task where a user inputs a self-identification, and the recognition speech samples are used to verify that the user's identification is the same as the self-identification. In another representative embodiment, the present invention may be used to perform a speaker identification task where the recognition speech samples are used to identify a user from multiple potential identities stored with each speech sample. The above embodiments are not mutually exclusive, and the same triplet network architecture may be used to perform both tasks. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates a system for performing speaker recognition according to an exemplary embodiment of the present invention. [Figure 2A] 1 illustrates a schematic structure of a deep neural network having a triplet network architecture for use in training, according to an exemplary embodiment of the present invention. [Figure 2B] 1 illustrates a schematic structure of a deep neural network having a triplet network architecture for use in joining and testing for a particular user, according to an exemplary embodiment of the present invention. [Figure 3A] 1 illustrates an example of the structure of a deep neural network having a triplet network architecture designed to receive preprocessed speech samples for use in training. [Figure 3B] Illustrates an example of the structure of a deep neural network architecture for use in subscription and testing for a particular user. [Figure 3C] 1 illustrates another embodiment of a deep neural network structure having a triplet network architecture designed to process raw speech samples for use in training. [Figure 4] 1 illustrates a flow chart of a general process for speaker recognition according to an exemplary embodiment of the present invention. [Diagram 5] 1 illustrates a flowchart of a process for utilizing a deep neural network in a triplet network architecture to perform speaker recognition in accordance with an exemplary embodiment of the present invention. [Figure 6] 1 is a flowchart of a process for training a deep neural network having a triplet network architecture according to an exemplary embodiment of the present invention. [Figure 7] An example of the structure of a deep neural network having a triplet network architecture that incorporates a softmax function and is designed to be pre-trained specifically to perform speaker recognition is illustrated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] A more detailed description of the invention will now be provided with reference to the accompanying drawings.

[0013] Representative embodiments of the present invention are directed to systems and methods that utilize deep neural networks (DNNs) with triplet network architecture to perform text-independent speaker recognition. The terms "deep neural network" and "DNN" refer to neural networks with multiple hidden layers consisting of multiple linear and nonlinear transformation functions. In this application, two subtasks of speaker recognition are considered verification and identification. For the purposes of this application, "verification" refers to the task of detecting whether an actual speaker is who they claim to be, while "identification" refers to the task of obtaining a speaker's identity from a predefined list of speakers. The principles of the present invention described below may be applied to either or both of these subtasks. Also, in this application, the term "user" may be used to refer to a particular speaker that the DNN is trained to recognize.

[0014] The use of DNNs to perform speaker recognition has an advantage over other approaches that utilize Universal Background Models (UBMs) estimated using Gaussian Mixture Models (GMMs) because DNNs are better at representing low-level features and mapping them to high-level features. The low-level features can be as low-level as the raw speech signal. The high-level features are voiceprints that preserve the full variability of the speech signal. Thus, the voiceprints extracted by DNNs can be similar to the i-vectors obtained by UBM / GMMs, but give better results.

[0015] FIG. 1 illustrates a system for performing speaker recognition according to an exemplary embodiment of the present invention. According to FIG. 1, a user or speaker speaks into an input device 10 that includes a microphone for converting spoken sounds into electrical signals. As specifically shown in FIG. 1, the input device 10 can be a device capable of long-distance communication, such as a telephone (either a mobile or landline) or a computer, or other processor-based device capable of Voice over Internet (VoIP) communication. It is believed that the present invention may be specifically utilized in applications that protect against telephone fraud, for example, verifying that a caller is who they claim to be, or detecting a caller's identity as a person on a "blacklist" or "blocked caller list."

[0016] 1, the user's utterances used to perform speaker identification will be referred to herein as "recognition speech samples." The recognition speech samples may be transmitted electronically from the input device 10 to the speaker recognition subsystem 20. It is possible, but not necessary, that the input device 10 through which the recognition speech samples are spoken is a long-range communication device (e.g., a telephone). For example, the input device 10 may simply be a microphone located in close proximity to the speaker recognition subsystem 20.

[0017] The speaker recognition subsystem 20 of FIG. 1 may include a computing system 22, which may be a server or a general-purpose personal computer (PC) programmed to model a deep neural network having a triplet network architecture (the details of which will be described in more detail below). However, it should be noted that the computing system 22 is not strictly limited to a single device, but may instead comprise multiple computers and / or devices working in cooperation to perform the operations described herein (e.g., training of the DNN may occur in one computing device, while the actual verification / identification tasks are performed in another computing device). Where a single or multiple central processing units (CPUs) may be used as a computing device for both training and testing, a graphics processing unit (GPU) may also be used. For example, the use of a GPU in the computing device 22 may help reduce computational costs, especially during training.

[0018] 1, the speaker recognition subsystem 20 also includes a memory device 24 that is used to train the DNN in a representative embodiment. In particular, this memory device 24 may include a plurality of sampled speech signals (or "speech samples") from a plurality of users or speakers, and a plurality of enrolled voiceprints obtained for users who are "enrolled" in the speaker enrollment subsystem 20. In particular, the memory device 24 includes two different data sets corresponding to the respective functions, training and testing, that will be performed on the DNN.

[0019] Regarding the function of training, according to the exemplary embodiment of the present invention, the DNN is trained according to positive samples corresponding to multiple speakers and a number N of negative samples. To perform training, the memory device 24 will preferably include a dataset including at least two speech samples obtained as actual utterances from each of the multiple speakers. These speech samples are called "positive speech samples" for the relevant speaker. In the memory device 24, the dataset for training the DNN will also include a number N of "negative speech samples" for each of the above speakers. These negative speech samples correspond to utterances by people different from the relevant speaker. In a specific example, 1000 negative speech samples (i.e., N=1000) may be used for each of the multiple speakers used to train the DNN. However, the number N may be more or less than 1000. It should also be noted that an unknown original speech sample may be used as one of the negative speech samples, regardless of whether it can be confirmed that such speech sample originates from a speaker i different from the speaker currently used to train the DNN.

[0020] The memory device 24 may include another data set to perform a "test" function, whereby the DNN performs actual speaker recognition by reliably verifying or identifying a user. To perform this function, the data set requires only one positive speech sample of a particular user, which may be obtained as a result of the user's "enrollment" in the speaker recognition subsystem 22 (this will be described in more detail below). Additionally, this data set may include one or more enrolled voiceprints corresponding to each user that may be verified / identified by the system.

[0021] Referring again to FIG. 1, the results of the speaker recognition analysis may be used by an end application 30 that needs to authenticate a caller (i.e., a user), i.e., verify that the caller is who they claim to be. Alternatively, the end application 30 may need to identify any callers that are on a predefined list (e.g., blacklist or blocked callers). This may help detect malicious callers who spoof their phone numbers to evade detection by Calling Terminal Identification (CLID) (sometimes called "Caller ID"). However, while the present invention may be used by applications 30 designed to weed out malicious callers, the present invention is not limited to these types of applications 30. For example, the present invention may be advantageously used in other applications 30 where, for example, voice biometric authentication is used to unlock access to rooms, resources, etc.

[0022] It should be noted that various modifications may be made to the system illustrated in FIG. 1. For example, the input device 10 may send the recognition speech sample directly to the end application 30, which then relays the recognition speech sample to the speaker recognition subsystem 20. In this case, the end application 30 may also receive some form of input from the user representing a self-identification. For example, in the case of a speaker identification task, the end application 30 may request the user to identify the user (either vocally or other form of input) and send both the recognition speech sample and the user's claimed identity to the speech recognition subsystem 20 for authentication. In other cases, the user's self-identification may consist of the user's claimed phone number, as obtained by CLID. Furthermore, there are no limitations regarding the respective locations of the various elements illustrated in FIG. 1. In certain circumstances, the end application 30 may be remote from the user, thus requiring the user to use telecommunications to interact with the end application 30. Alternatively, the user (and input device 10) may be in close proximity to the end application 30 during use, for example if the application 30 controls a voice-activated security gate or the like.

[0023] 2A illustrates a schematic structure of a deep neural network (DNN) having a triplet network architecture for use in training in accordance with a representative embodiment of the present invention, and conceptually illustrates the use of batch processing in which a cohort set of P speech samples, their corresponding P positive samples, and N negative speech samples are used to train first, second, and third feedforward neural networks.

[0024] In FIG. 2A, the DNN consists of three instances of identical feedforward neural networks 212, 222, and 232 that share the same parameters. With shared parameters (as shown in the figure), the three feedforward neural networks 212, 222, 232 are identical in the number and configuration of layers, connection weights between layers, etc. These feedforward neural networks 212, 222, 232 are provided with three different samples via inputs 210, 220, 230, respectively. In particular, when the DNN is being trained, the DNN may perform batch processing, whereby the first feedforward neural network 212 and the second feedforward neural network 222 each receive three different samples via inputs 210 and 220, respectively. JPEG2025065135000020.jpg1052 A set of P conversation samples, denoted as JPEG2025065135000021.jpg55 The two sets of speech samples are supplied together with a corresponding set of P positive speech samples, denoted as P , which are spoken by (and therefore attributed to) the same speaker. Additionally, the third feedforward neural network 232 receives via input 230 during batch processing a common cohort set of negative speech samples, denoted as P . JPEG2025065135000022.jpg831 The negative speech samples in the cohort set are not spoken by (or at least not known to be spoken by) the same speaker as the set of P samples above, and therefore are not attributable to the same speaker. The output 214 of the first feedforward neural network 212 provides the sample JPEG2025065135000023.jpg43 The output 224 of the second feedforward neural network 222 is a set of first P embedding vectors according to the sample JPEG2025065135000024.jpg55 The output 234 of the third feedforward neural network 232 generates a second set of P embedding vectors in response to the negative speech samples in the cohort set. And the output 234 of the third feedforward neural network 232 generates a third set of N embedding vectors in response to the negative speech samples in the cohort set. After a given batch is processed, these embedding vectors are used to calculate a loss (which will be explained in more detail below), and the loss is used to modify the connection weights of the three feedforward neural networks 212, 222, 232 according to the backpropagation method.

[0025] Further, as shown in FIG. 2B, another feedforward neural network 242 is used to perform actual speaker recognition based on the recognition speech samples input by the user (via the input device 10) after the training of the DNN is completed. Since the first, second and third feedforward neural networks 212, 222, 232 incorporate shared parameters and are therefore identical to each other when the training of the DNN is completed, any one of these three feedforward neural networks can be used interchangeably as the feedforward neural network 242 performing speaker recognition. Alternatively, a new instance of a feedforward neural network incorporating parameters shared among the three feedforward neural networks 212, 222, 232 may be used as the feedforward neural network 242 of FIG. 2B for use in user enrollment and / or testing (speaker recognition). The enrollment and testing performed by this feedforward neural network 242 will be described in more detail below.

[0026] Additional details will now be provided for the design of the DNN. As will be discussed below, aspects of a convolutional neural network (CNN) can be combined with aspects of a DNN by incorporating at least one convolutional layer into each of the first, second and third feedforward neural networks 212, 222, 232. Furthermore, each convolutional layer has neurons in three dimensions, namely height, width and depth.

[0027] 3A illustrates a specific example of a DNN structure having a triplet network architecture for use in training the DNN. As shown in FIG. 3A, each of the first, second and third feedforward neural networks 212, 222 and 232 may include a first convolution layer connected to a max pooling layer, a second convolution layer followed by a second max pooling layer, a subsequent fully connected layer, and an output layer including an embedding vector. However, at the end of training, the output layer of each of the feedforward neural networks 212, 222, 232 will be configured to generate a feature representation (i.e., a voiceprint) of the input sample.

[0028] 3A are the inputs 210, 220, 230 of the first, second and third feedforward neural networks 212, 222, 232, respectively. According to an exemplary embodiment, each of the speech samples (speech samples JPEG2025065135000025.jpg43 , Positive conversation sample JPEG2025065135000026.jpg55 , and negative conversation samples. JPEG2025065135000027.jpg34 ) may be pre-processed to generate an "image" of the corresponding utterance. Such pre-processing may include applying voice activity detection to discard non-speech portions of the signal. Pre-processing may also include partitioning the underlying speech signal into a fixed number (W) of overlapping windows and extracting a fixed number (F) of features (e.g., Mel filter bank features) from each of the W overlapping windows. Specific examples of such pre-processing, which are not to be construed as limiting the claims, will be described. In this non-limiting example, an image may be generated for each speech sample by partitioning the underlying speech signal into windows of 20 ms duration with 10 ms of overlap (or window shift). Furthermore, the image for each sample may include 500 of the above overlapping windows (W=500), with 40 Mel filter bank features (F=40) being extracted from each window (thereby resulting in an image of size 40×500 at each of the inputs 210, 220, 230). This would correspond to a 5 second speech sample (40 dimensional feature vectors extracted every 10 ms). However, it should be noted that this is just one example and different speech durations, different numbers of windows, and different numbers and types of features may be used. In other words, different types of speech "images" may be applied to the DNN.

[0029] It should be noted that the above pre-processing may be performed either before or after the associated speech samples are stored in memory device 24. It is also contemplated that device 10 that inputs recognition speech samples from a user, as well as other types of speech transmission / recording devices used to input positive and negative speech samples, may be configured to perform some or all of the above pre-processing operations.

[0030] Referring again to FIG. 3A, assuming that the input signal is a preprocessed image of size F×W as described above, the first convolutional layer of each of the first, second and third feedforward neural networks is configured with an appropriate number of (N C ) convolution filters. Furthermore, each convolution filter in this layer processes the features of the corresponding window, and thus F×w f Each convolution filter can thus be constructed with neural units (or "neurons") of size w , where w is the number of features F. Each convolution filter has a height proportional to the number of features F, and a width (w ) that can be selected based on various considerations (e.g., to take into account conversational dynamics, to meet processing and memory requirements, etc.). f )

[0031] A specific non-limiting example of the configuration of the first convolution layer in the feedforward neural networks 212, 222, 232 of FIG. 3A will now be described. For the purposes of this example, it may be assumed that the input image is preprocessed as described above with respect to the above non-limiting example of preprocessing, with a size (F×W) of 40×500 (corresponding to 5 seconds of speech, including a 40-dimensional feature vector extracted every 10 milliseconds). In this case, the convolution layer of each of the first, second and third feedforward neural networks is composed of 256 convolution filters (N C = 256), with each filter having a height of 40 neurons (corresponding to the 40 features extracted from each window) and a width (w f In this case, the depth of the first overlapping layer of FIG. 3A will be 256, the height of this layer will be 40, and the width of the layer will be w f ≦5. Also, in this non-limiting example, the entry neurons of the first convolution layer may be connected to multiple windows of the input image.

[0032] It should be noted that the above is a non-limiting example, and the dimensions of the depth and height of the first convolution layer may be different from the above 256 and 40, respectively. However, in this case, the depth (i.e., the number of convolution filters N C It is proposed that the n th , , is preferably less than or equal to the number of overlapping windows in the input image, while the height is preferably equal to the number of features (e.g. Mel-filter bank features) extracted from each of the windows during processing of the relevant speech sample.

[0033] Referring again to FIG. 3A, the first convolutional layer in each of the feedforward neural networks 212, 222, 232 is followed by a one-dimensional max pooling layer of size s m The max pooling layer is a layer that calculates the maximum value from a temporary input sequence. In FIG. 3A, the result of the max pooling layer is N C The first convolution layer is then provided to a second convolution layer, consisting of N convolution filters. However, the size of these filters does not have to be the same as that of the first convolution layer. In a non-limiting example, the height and width of the second convolution layer can be 1. Furthermore, according to FIG. 3A, a final global max pooling layer is applied across the entire time axis to make the resulting embedding vector insensitive to the duration of the input image. The next layers are fully connected layers and an output layer, each consisting of, for example, N C The output layer provides embedding vectors during training, but is configured to generate feature representations, i.e., voiceprints or voice signatures, of the speakers from which the samples originate after training. Specifically, the output layers of the respective first, second and third feedforward neural networks 212, 222, 232 are what were referred to above as the first, second and third network outputs 214, 224, 234 of the DNN.

[0034] For example, during training, conversation samples JPEG2025065135000028.jpg55 is fed through the first feedforward neural network 212, the first network output 214 is JPEG2025065135000029.jpg55 Denote the embedding vector for TIFF2025065135000030.tif47 Similarly, the positive speech samples JPEG2025065135000031.jpg55 is fed through the second feedforward neural network 222, the second network output 224 is JPEG2025065135000032.jpg55 Denote the embedding vector for TIFF2025065135000033.tif48 Similarly, the negative speech samples JPEG2025065135000034.jpg34 When the image is fed through a third feedforward neural network 232, where n is any integer from 1 to N, the third network output 234 is JPEG2025065135000035.jpg34 Denote the embedding vector for TIFF2025065135000036.tif48 This produces a result that can be symbolized as:

[0035] As shown in FIG. 3A, after a given batch of training samples has been processed, a loss function may be calculated based on the outputs 214, 224, 234 of the first, second and third feedforward neural networks 212, 222, 232, respectively. The calculated loss function may be used to train each neural network 212, 222, 232 of the DNN using a backpropagation algorithm with a "stochastic gradient descent" optimizer, which aims to calculate the gradient of the loss function with respect to all weights in the DNN. The goal of the optimizer is to update the weights to minimize the loss function. However, it is contemplated that other types of backpropagation algorithms may be used. In the example of FIG. 3A, the loss function may be used to update the connection weights of each of the first convolutional layer, the second convolutional layer and the fully connected layer. More details regarding the training algorithm will be discussed below with respect to FIG. 6.

[0036] Reference will now be made to FIG. 3B, which illustrates a specific example of a DNN architecture for use in the onboarding and testing of a particular user. In particular, FIG. 3B illustrates a feedforward neural network 242 for performing the onboarding and testing functions. As described above with respect to FIG. 2B, the configuration and parameters of the feedforward neural network 242 of FIG. 3B will share the same configuration and parameters with each of the first, second and third feedforward neural networks 212, 222, 232 after the DNN of FIG. 3A is trained. Thus, at the end of training, any one of the first, second and third feedforward neural networks 212, 222, 232 may be used as the feedforward neural network 242 of FIG. 3B, or a new instance of a neural network incorporating the shared parameters and configuration may be used.

[0037] As shown in FIG. 3B, the neural network 242 may receive at its input 240 images generated by pre-processing of speech samples. Depending on whether joining or testing is performed, the input images will correspond to either speech samples entered for the purpose of joining or enrolling a particular user (i.e., "joining speech samples"), or speech samples for which speaker recognition is performed (i.e., "recognition speech samples"). Once the input images are fed through the feedforward neural network 242, the output layer 240 will generate a corresponding feature representation (i.e., a voiceprint). Each feature representation generated by the output layer 240 may be similar to an i-vector, but because they are generated by a DNN, they are designed to better represent the low-level features of a speaker's voice.

[0038] As mentioned above, the example DNN architectures in both Figures 3A and 3B are based on the assumption that each input speech sample is preprocessed as an image. However, this does not have to be the case. For example, the DNN may exhibit a different triplet network architecture, of which Figure 3C illustrates an example of a deep neural network having a triplet network architecture for processing raw speech samples as input. In particular, as shown in Figure 3C, the speech samples received at the respective inputs 210, 220 and 230 of the respective feedforward neural networks 212, 222 and 232 are not preprocessed in the manner described above with respect to Figures 3A and 3B. Instead, each input sample constitutes a signal obtained by sampling a raw speech signal (e.g., input via the input device 10) to obtain a series of samples. Additionally, as shown in the particular example of Figure 3C, each of the first, second and third feedforward neural networks 212, 222, 232 includes an additional convolutional layer and local max pooling, which are positioned before the other layers similar to those described with respect to Figure 3A. The additional convolutional layer of Figure 3C, shown to receive the raw speech samples, consists of a number (F) of convolutional filters corresponding to the number of features to be extracted from the raw speech samples. The additional max pooling layer is a filter of size s m1 These additional layers may be configured to apply transformations similar to the pre-processing described above with respect to Figures 3A and 3B.

[0039] Reference will now be made to FIG. 4, illustrating a flow chart of a schematic process 400 for speaker recognition according to a representative embodiment of the present invention. According to operation S410 of this figure, a DNN having a triplet network architecture is trained. To perform speaker recognition for a particular user, it is conceivable that the DNN will be trained based on at least one speech sample obtained from the user in combination with a number of negative speech samples obtained from other speakers (and possibly unknown speakers). As illustrated in FIG. 4, the speech sample used to train the DNN may be extracted from the storage of the memory device 24. After the DNN is trained, the particular user may be registered or enrolled by the speaker recognition subsystem 20 in operation S420. This means that a "speaker model" may be generated for the particular user for the purpose of future comparison with the feature representation or voiceprint of the same user to perform speaker verification. After that, a recognition speech sample is obtained in operation S430. The recognition speech sample may be obtained, for example, from the input device 10. In operation S440 of FIG. 4, the recognition speech sample is fed through a DNN (i.e., neural network 242), and the output of the DNN responsive to the recognition speech sample is used in operation S450 (together with the speaker model described above) to perform speaker recognition, i.e., to either verify or identify a user based on the DNN output.

[0040] FIG. 5 illustrates a flowchart of a more specific process 500 for performing speaker recognition for a particular user according to an exemplary embodiment of the present invention. In operation S510 of FIG. 5, a DNN consisting of the first, second and third feedforward neural networks 212, 222, 232 is trained by the computer system 22 using speech samples stored in the memory device 24. An exemplary embodiment of this training process will be described in more detail with respect to FIG. 6. It should be noted that the memory device 24 may include a bank of speech samples useful for training the DNN in batches corresponding to multiple different users. Thus, training the DNN according to S510 may include any necessary steps for classifying and extracting speech samples in the memory device 24 by speaker.

[0041] After the DNN is trained as a result of S510, the computer system 22 is operable to use the DNN to perform speaker recognition on a list of registered (or subscribed) users. This means that the DNN can perform verification by determining whether a speaker claiming to be one of the registered (subscribed) users is actually the person he claims to be. It also means that the DNN can identify an anonymous speaker by whether the speaker is actually present in the list of registered (subscribed) users, and indicate that the speaker is unknown if the speaker is not present in this list. Thus, in a representative embodiment of the present invention, a speaker model is generated for each user that is subscribed or registered. A specific example of generating such a speaker model is disclosed with respect to operations S520 and S525 of FIG. 5 for subscribing a particular user.

[0042] In operation S520 of FIG. 5, enrollment of a particular user is initiated by receiving a corresponding enrollment speech sample (i.e., a speech sample actually uttered by the particular user). In operation S525, the enrollment speech sample is fed through a feedforward neural network 242 (as described above with respect to FIGS. 2B and 3B) to output a corresponding voiceprint (or feature representation). The voiceprint generated by the feedforward neural network 242 may then be stored in the memory device 24 and used as a speaker model for the enrolled user. It is also possible that more than one enrollment speech sample from a user may be received and used to enroll the user. In this case, the multiple enrollment samples may be fed through the feedforward neural network 242 to generate multiple voiceprints for the user. In this case, an average vector may be calculated from the multiple voiceprints and used as a speaker model for the user.

[0043] Operations S520 and S525 of Figure 5 illustrate an example of the "join" function of a DNN, according to a representative embodiment. However, it is not a requirement of the present invention that a DNN be used in the manner described above to generate a speaker model for an enrolled user.

[0044] Speaker recognition may be performed for a particular user according to the sequence of operations S530-S560 in Fig. 5. These operations thus correspond to the "test" function of the DNN according to the exemplary embodiment of the present invention.

[0045] A recognition speech sample is received from a user via device 10 in operation S530. The recognition speech sample is then provided to a feedforward neural network 242 in operation S530, which responsively generates a feature representation or voiceprint of the user's recognition speech sample at network output 244.

[0046] In operation S540 of FIG. 5, the voiceprint or feature representation output by the feedforward neural network 242 in response to the recognition speech sample is used to verify or identify the user. In particular, the voiceprint may be compared to one or more speaker models generated by operation S525 and stored in the memory device 24 for one or more enrolled users. As described above, each of these speaker models may be a voiceprint similarly generated by the feedforward neural network 242. According to decision box S550, a determination is made as to whether the voiceprint obtained by the neural network 242 as a result of the recognition speech sample "matches" any of the stored speaker models. Assuming that each speaker model is a similarly generated voiceprint, this match may be determined by a measure of similarity (or distance) between the two voiceprints. On the other hand, if each speaker model includes a different representation of the speech features of the corresponding user, different processes may be used in S540-S550 to compare the voiceprint with the respective features of the speaker models and determine whether a match exists.

[0047] If S550 determines that a match has occurred between the voiceprint of the recognized speech sample and one of the stored speech samples, process 500 proceeds to S560 and indicates that the particular user's identity has been authenticated (in a speaker verification task) or that the particular user's identity is present in a predefined list or known identities (in a speaker identification task). On the other hand, if S550 determines that no match has occurred, process 500 proceeds to S570 and indicates that the particular user is not who they claim to be (and thus is a fake) or that the particular user's identity is unknown to the speaker recognition subsystem 20. These results may be output by computer system 22, for example, on a display and / or audio output, or the results may be sent to another location where they are output or sent to end application 30 and used therein. For example, in both speaker verification and identification tasks, end application 30 may use the results to allow / deny a user from accessing a particular resource or path. For example, end application 30 may be used to block a user from making outgoing calls when the user is identified as being part of a blacklist or list of blocked callers. Application 30 may also be used to automatically flag (or alert authentication regarding) users who are misrepresenting their user identity.

[0048] Referring again to FIG. 5, after one speaker recognition task is performed, the DNN can be used to perform another speaker recognition task on the particular sample based on the current pool of enrolled users, as illustrated in S580 (including the process 500 returning to S530 to receive a new recognition speech sample). It is also conceivable that another speaker needs to be enrolled, i.e., added to the pool of enrolled users, for the purpose of performing future speaker recognition tasks. If a new user needs to be enrolled or enrolled, as illustrated in S590, the process 500 returns to operation S520, whereby a new speaker model is generated.

[0049] Next, a more specific description will be provided regarding the process whereby a DNN is trained for each enrolled user per operation S510 of FIG. 5.

[0050] In particular, Figure 6 is a flow chart of a process 5100 for training a DNN, according to an exemplary embodiment of the present invention. This training of the DNN may be performed by batch processing, whereby the loss function is calculated after a batch of training examples are fed through the DNN. For each batch, we use a set of training samples, X i A set of P speech samples, denoted as (i=1,...,P), is then run through a first feedforward neural network 212. And for each batch, JPEG2025065135000037.jpg43 A set of P corresponding speech samples, all spoken by the same speaker as the sample JPEG2025065135000038.jpg853 is then applied to the second feedforward neural network 222. For the third feedforward neural network 232, the cohort set of negative speech samples JPEG2025065135000039.jpg842 ( TIFF2025065135000040.tif44 and JPEG2025065135000041.jpg55 The negative speech samples (spoken by a different person than the person who spoke the speech sample) are subsequently input into a third feedforward neural network 232 during each batch. By way of a non-limiting example, the number N of negative speech samples in the cohort set (and the number of iterations performed for each batch) may be equal to 1000. However, the cohort set may contain a different number of negative speech samples. It is possible that the same cohort set of negative speech samples may be used for multiple batches, or possibly all batches, during training of the DNN. Also, each negative speech sample may be JPEG2025065135000042.jpg43 and JPEG2025065135000043.jpg55 If a speech sample of unknown origin (i.e., the identity of the speaker is unknown) is thought to be spoken by a different person than the original speaker, it is possible to use the speech sample of unknown origin (i.e., the identity of the speaker is unknown) as one of the negative speech samples in the cohort set.

[0051] Referring to FIG. 6, operation S5110 generates a set of P speech samples from a particular speaker. JPEG2025065135000044.jpg43 is extracted from memory device 24. Similarly, in S5120, a set of P speech samples from the same speaker is also extracted from memory device 24. The number P of positive speech samples should be at least two, and it is possible that P is greater than two. Furthermore, according to operation S5130, a cohort set of N negative speech samples is obtained from memory device 24.

[0052] As mentioned above, a deep neural network (DNN) trains a set of P speech samples in each batch. JPEG2025065135000045.jpg852 and one corresponding set of P positive speech samples JPEG2025065135000046.jpg853 The training is performed in batches such that all P speech samples come from the same speaker and are fed through the first feedforward neural network 212 and the second feedforward neural network 222, respectively. JPEG2025065135000047.jpg753 is subsequently fed through the first neural network 212 in operation S5140, while the set of P positive speech samples JPEG2025065135000048.jpg853 is fed through the second feedforward neural network 222 in operation 5150. The N negative speech samples in the cohort set are subsequently fed through the third feedforward neural network 232 in operation S5160.

[0053] After a given batch has been run through the DNN in this manner, operation S5170 is performed, which runs the speech samples through the DNN. JPEG2025065135000049.jpg43 , Positive conversation sample JPEG2025065135000050.jpg55 , negative conversation sample JPEG2025065135000051.jpg840 Cohort set of TIFF2025065135000052.tif34 Further, according to S5170, the calculated loss is used to modify the connection weights of each of the first, second and third feedforward neural networks 212, 222, 232. In particular, a backpropagation method utilizing a stochastic gradient descent optimizer may be used to modify the weights once the loss function is calculated. The function for calculating the loss will be described in more detail below.

[0054] For each batch (i.e., P conversation samples), JPEG2025065135000053.jpg43 For each set of P and the corresponding positive conversation samples, JPEG2025065135000054.jpg55 The loss function used to calculate the loss on the set of Each conversation sample JPEG2025065135000055.jpg43 The first network output 214 (i.e., the embedding vector JPEG2025065135000056.jpg614 ) and the corresponding positive conversation samples JPEG2025065135000057.jpg55 The second network output 224 (i.e., the embedding vector JPEG2025065135000058.jpg614 ) and the similarity S + Each conversation sample JPEG2025065135000059.jpg43 A first network output 214 (i.e., JPEG2025065135000060.jpg614 ) and the most similar of the third network outputs 234 according to the cohort set (i.e., the feature representation JPEG2025065135000061.jpg752 the most similar one among them) - ·Similarity S + and S - A positive margin M + and a negative margin M - ·Similarity S + and the corresponding margin M + A positive distance d calculated based on + ·Similarity S - and the corresponding margin M - The negative distance d calculated based on + Positive Margin M + and a negative margin M - Use of the conversation sample JPEG2025065135000062.jpg43 and JPEG2025065135000063.jpg55 are reasonably close to each other, and the conversation samples JPEG2025065135000064.jpg43 This helps to avoid the additional cost of the loss function in situations where the nearest neighbor is reasonably far from the negative speech sample.

[0055] According to a representative embodiment, the loss function for each batch may be defined as follows:

number

number

[0056] As shown in equations (3) and (4), the positive and negative distances d + and d - are the similarity S + and S - and the corresponding margin M + and M. - The similarity S + and S - can be calculated by the following equation:

number

number

[0057] Also, the positive and negative margins M + and M. - can be calculated by the following equation:

number

number

[0058] Referring again to Figure 6, when the loss function is calculated for a given batch, the calculated loss is used in operation S5170 to modify the connection weights of each of the first, second and third feedforward neural networks 212, 222, 232, for example, by backpropagation using stochastic gradient descent. For example, if the DNN has the particular configuration illustrated in Figure 3A, the calculated loss may be used to modify weights for the first convolutional layer, the second convolutional layer, and the fully connected layer.

[0059] It is also noted that dropout training may be used in conjunction with the backpropagation method to modify the weights. In particular, in a dropout algorithm, a certain portion or percentage of the neurons being trained are dropped during training to prevent modification of their corresponding weights. For example, a dropout algorithm may be applied such that only 80% (or even just 50%) of the weights are modified as a result of a given batch.

[0060] In process 5100 of FIG. 6, after the connection weights of the DNN are modified by the loss calculated for a given batch by operation S5170, in S5180, the DNN is trained on any more batches (i.e., different sets of speech samples). JPEG2025065135000073.jpg43 and positive ones corresponding to different speakers JPEG2025065135000074.jpg55 ) is determined whether more batches are to be trained. If there are more batches to be processed, processing returns to operation S5110 to extract the necessary samples and begin processing a new batch. Otherwise, in FIG. 6, training of the DNN is considered to be completed.

[0061] It should be noted that the flowcharts of Figures 4-6 are provided for illustrative purposes only and are not intended to limit the present invention as defined by the appended claims. The processes disclosed in these figures may be modified and still remain within the concept and scope of the present invention. For example, although each of these flowcharts illustrates a sequence of operations, the order of these operations may be changed, certain operations illustrated in these flowcharts may be omitted, and other operations not shown may be added.

[0062] 1, 2A, 2B, and 3A-3C are also provided for illustrative purposes only, and the systems and neural network configurations illustrated therein may be modified in any manner consistent with the concept and scope of the present invention.

[0063] Alternative Embodiments In the above embodiment, a DNN with triplet network architecture is trained to perform both verification (detecting whether the actual speaker is who they claim to be) and identification (matching the speaker's identity to someone in a closed set) speaker recognition tasks. In such an embodiment, the DNN model with triplet network architecture may include the structure illustrated in Figure 3A and Figure 3B, and may be further trained by the process shown in Figure 6. In this case, the parameters of the DNN model with triplet network architecture may be randomly initialized before training.

[0064] However, according to alternative embodiments, the DNN may be specifically designed for the task of identification, and different DNN structures and training methods may be adopted. For example, instead of randomly initializing the parameters of a DNN model having a triplet network architecture, the parameters may instead be initialized by the parameters of a DNN specifically trained to perform a speaker identification task on a closed set of speakers, and the output classes are different speaker labels. Furthermore, during such speaker identification training, the structure of the DNN may be based on the structure of the feedforward neural network 212, 222, 232, 242 illustrated in Figures 3A and 3B, but may be modified by at least two additional layers: (1) a softmax layer, and (2) an output layer including multiple units for each speaker in the training set.

[0065] For example, these additional layers may be added to the feedforward neural network structure illustrated in FIG. 3A and FIG. 3B. As described above, each of the feedforward neural networks illustrated in FIG. 3A and FIG. 3B ends with a fully connected layer and a subsequent layer for generating embedding vectors or voiceprints. According to an alternative embodiment, this subsequent layer may be transformed into a second fully connected layer that provides a softmax layer and an output layer. The softmax layer performs a function that is sometimes called a normalized exponential function and is a generalization of the sigmoid function. The softmax function is used in various probabilistic multi-class classification methods, including those implemented by artificial neural networks. With respect to the added output layer, the number of neural units of this layer is equivalent to the number of speakers that the DNN is trained to identify. For example, in a non-limiting example where the DNN is trained to perform speaker identification on 3000 different speakers, the output layer will include 3000 different units.

[0066] After the DNN (with added layers) is trained to perform speaker identification for a closed set of speakers, the trained parameters may then be provided to the first, second and third feedforward neural networks 212, 222, 232 for subsequent training based on a loss function, for example, by the process described above with respect to FIG. 6.

[0067] This alternative embodiment may be based on the structure illustrated in FIG. 3A and FIG. 3B as described above, but it is believed that other modifications may also be made to the structure of the feedforward neural network in this alternative embodiment. For example, FIG. 7 illustrates a specific example of the structure of a DNN used in relation to the alternative embodiment. Using the structure of the feedforward neural network in FIG. 3B as a reference, FIG. 7 illustrates a feedforward neural network 242' having a modified structure incorporating three convolution layers, three max pooling layers, and four fully connected layers. In FIG. 7, the fourth fully connected layer is connected to a softmax layer 710 and an output layer 720 consisting of a number of units (equivalent to the number of speakers the DNN is trained to identify). After speaker identification training is performed, the parameters may then be given to each feedforward neural network with a triplet network architecture and trained based on the loss function described above.

[0068] Second Alternative Embodiment In describing the training process illustrated in Figure 6, it is noted that a specific loss function was described above with respect to equations (1)-(8). However, it is not a requirement that the DNN be trained with the specific loss function described above with respect to Figure 6. In alternative representative embodiments, a different loss function, for example, directly related to an equal error rate (EER) metric, may be used to train the DNN.

[0069] The EER metric is typically used to evaluate the accuracy of a speaker recognition system. In speaker recognition and other biometric systems, the EER is used to predetermine a threshold for equalizing the false accept rate and the false reject rate. The EER is derived under the assumption that the distribution of positive recognition scores (indicating a match) and negative recognition scores (indicating a mismatch) is Gaussian, and may be expressed by the following equation:

number

number

number

number

number

[0070] As shown in equation (9) above, minimizing the value of EER is achieved by the term

number

number

[0071] While specific embodiments have been described above for purposes of example, they are not intended to limit the present invention, the scope of which is defined in the appended claims.

Claims

1. 1. A speaker recognition device, comprising: A memory device for storing speech samples, the speech samples comprising: Dual sets of conversation samples from the same speaker, A cohort set of conversation samples not from the same speaker as the dual set; and a memory device including a set of speaker models; a processor-based device configured to model a deep neural network having a triplet network architecture; the processor-based device trains the deep neural network according to a batch process in which the dual set of speech samples is fed through the deep neural network in combination with the cohort set of speech samples; A speaker recognition device, wherein the processor-based device supplies recognition speech samples through the trained deep neural network, and verifies or identifies a user based on an output of the trained deep neural network in response to the recognition speech samples and at least one of the speaker models.

2. The deep neural network, a first feedforward neural network that receives and processes a first input to generate a first network output; a second feedforward neural network that receives and processes a second input to generate a second network output; and a third feedforward neural network that receives and processes a third input to generate a third network output; For each of a plurality of speakers, the memory device stores a first set of P speech samples by the speaker. and a second set of P speech samples by said speaker P is an integer of 2 or more; The deep neural network is trained by the processor-based device, whereby, for each of the plurality of speakers, the deep neural network performs batch processing during which a first set of the corresponding speech samples is fed through the first feedforward neural network, a second set of the corresponding speech samples is fed through the second feedforward neural network, and a cohort set of the speech samples is fed through the third feedforward neural network; Upon completion of the batch processing, a loss function is calculated based on the first network output, the second network output, and the third network output obtained based on the first set of corresponding speech samples, the second set of corresponding speech samples, and the cohort set of speech samples, respectively; 2. The speaker recognition device of claim 1, wherein the calculated loss function is used to modify connection weights of each of the first, second and third feedforward neural networks by a backpropagation method.

3. The loss function is one of the first set of conversation samples and one of the first network output and the corresponding second set of speech samples according to The similarity S between the second network output ＋ The positive distance d ＋ and, The one of the first set of speech samples. and the most similar one of the third network outputs according to each of the conversation samples of the cohort set. － The negative distance d － The speaker recognition device of claim 2 , based on:

4. The positive distance d ＋ and the negative distance d － is the corresponding similarity S ＋ , S － Each margin M ＋ , M. － The speaker recognition device of claim 3, wherein the speaker is determined by applying

5. The loss function is [0010] wherein: [0025] and d ＋ =2(1-min((S ＋ +M ＋ ), 1), d － =2(1-max((S ＋ +M － −1), 0), [0030] and [0045] and 【number】 is the nth of the N negative speech samples provided during N iterations, 【number】 is the first network output in response to one of the first set of speech samples; 【number】 is the second network output in response to one of the second set of speech samples; 【number】 However, the negative conversation sample 【number】 the third network output being responsive to [0050] and [006] and The speaker recognition device of claim 4 , wherein K is a constant.

6. The speaker recognition device of claim 1 , wherein each of the first, second and third feedforward neural networks includes at least one convolutional layer and a fully connected layer.

7. The speaker recognition device of claim 6 , wherein each of the first, second and third feedforward neural networks further includes at least one max pooling layer and a subsequent fully connected layer.

8. Each speech sample input to a respective one of the first, second and third feedforward neural networks is Partitioning the underlying speech signal into a plurality of overlapping windows; The speaker recognition device of claim 6 , wherein the speech is preprocessed by: extracting a plurality of features from each of the overlapping windows.

9. the first, second and third feedforward neural networks each include a first convolutional layer that receives the preprocessed speech samples; The first superposed layer has a number N Ｃ , including a convolution filter of The N Ｃ Each of the convolution filters is F×w ｆ neurons, F corresponds to the height of the first convolution layer, and w ｆ corresponds to the width of the overlapping layer, F is equal to the number of features extracted from each of the overlapping windows, and w ｆ The speaker recognition device of claim 8 , wherein is less than or equal to 5.

10. 2. The speaker recognition device of claim 1, wherein the device is configured to perform a speaker verification task in which the user inputs a self-identification, and the recognition speech sample is used to verify that the user's identification is the same as the self-identification.

11. 2. The speaker recognition device of claim 1, wherein the device is configured to perform a speaker identification task in which the recognition speech sample is used to identify the user from a plurality of potential identifications stored in the memory device with respective speech samples.

12. The speaker recognition device of claim 1 , further comprising an input device for receiving a speech sample from the user as the recognition speech sample.

13. 1. A method comprising: Training a computer-implemented model of a deep neural network having a triplet network architecture based on a plurality of speech samples stored in a memory device, the plurality of speech samples comprising: Dual sets of conversation samples from the same speaker, a cohort set of conversation samples not from the same speaker as the dual set; a speaker model set; and feeding recognition speech samples through the trained deep neural network, and verifying or identifying a user based on an output of the trained deep neural network in response to the recognition speech samples and at least one of the speaker models; 13. The method of claim 12, wherein the training of the deep neural network is performed according to a batch process in which the dual set of speech samples is fed through the deep neural network in combination with the cohort set of speech samples.

14. The deep neural network, a first feedforward neural network, each iteration of which receives and processes a first input to generate a first network output; a second feedforward neural network, each iteration of which receives and processes a second input to generate a second network output; and a third feedforward neural network, each iteration of which receives and processes a third input to generate a third network output; and For each of a plurality of speakers, the memory device stores a first set of P speech samples by the speaker. and a second set of P speech samples by said speaker P is an integer of 2 or more; The deep neural network is trained to generate, for each of the plurality of speakers, the deep neural network performs batch processing during which a first set of the corresponding speech samples is fed through the first feedforward neural network, a second set of the corresponding speech samples is fed through the second feedforward neural network, and a cohort set of the speech samples is fed through the third feedforward neural network; Upon completion of the batch processing, a loss function is calculated based on the first network output, the second network output, and the third network output obtained based on the first set of corresponding speech samples, the second set of corresponding speech samples, and the cohort set of speech samples, respectively; 14. The method of claim 13, wherein the calculated loss function is used to modify connection weights of each of the first, second and third feedforward neural networks by a backpropagation method.

15. The loss function calculated with respect to one of the first set of conversation samples and one of the first network output and the corresponding second set of speech samples according to The similarity S between the second network output ＋ The positive distance d ＋ and, The one of the first set of speech samples. and the most similar one of the third network outputs according to each of the conversation samples of the cohort set. － The negative distance d － The method of claim 14, based on

16. The positive distance d ＋ and the negative distance d － is the corresponding similarity S ＋ , S － Each margin M ＋ , M. － The method of claim 15, wherein the step is determined by applying

17. The loss function is [0070] wherein: [0080] and d ＋ =2(1-min((S ＋ +M ＋ ), 1), d － =2(1-max((S ＋ +M － −1), 0), [0090] and [0010] and 【number】 is the nth of the N negative speech samples provided during N iterations, 【number】 is the first network output in response to one of the first set of speech samples; 【number】 is the second network output in response to one of the second set of speech samples; 【number】 However, the negative conversation sample 【number】 the third network output being responsive to ##EQU00011## and ##EQU00012## and 17. The method of claim 16, wherein K is a constant.

18. The loss function is ##EQU00013## wherein: μ ＋ and 【number】 are the mean and standard deviation of the positive recognition scores based on a Gaussian distribution, 【number】 and 【number】 The method of claim 14 , wherein x is the mean and standard deviation of the negative recognition scores based on a Gaussian distribution.

19. 14. The method of claim 13, wherein each of the first, second and third feedforward neural networks includes at least one convolutional layer and a fully connected layer.

20. 20. The method of claim 19, wherein each of the first, second and third feedforward neural networks further comprises at least one max pooling layer and a subsequent fully connected layer.

21. each of the first, second and third feedforward neural networks further includes a softmax layer and an output layer; 21. The method of claim 20, wherein parameters of the deep neural network are initialized by parameters of a neural network pre-trained to perform a speaker identification task on a closed set of speakers, and the output layer includes a separate neural unit for each of the speakers.

22. pre-processing each speech sample input to a respective one of the first, second and third feedforward neural networks, comprising: Partitioning the underlying speech signal into a plurality of overlapping windows; The method of claim 19 , further comprising preprocessing by: extracting a plurality of features from each of the overlapping windows.

23. the first, second and third feedforward neural networks each include a first convolutional layer that receives the preprocessed speech samples; The first superposed layer has a number N Ｃ , including a convolution filter of The N Ｃ Each of the convolution filters is F×w ｆ neurons, F corresponds to the height of the first convolution layer, and w ｆ corresponds to the width of the overlapping layer, F is equal to the number of features extracted from each of the overlapping windows, and w ｆ The method of claim 22, wherein is 5 or less.

24. 14. The method of claim 13, wherein a speaker verification task is performed in which the user inputs a self-identification and the recognized speech sample is used to verify that the user's identification is the same as the self-identification.

25. 14. The method of claim 13, wherein a speaker identification task is performed and the recognized speech samples are used to identify the user from a plurality of potential identifications stored in the memory device with respective speech samples.

26. The method of claim 13 , further comprising receiving a speech sample from a user via an input device as the recognition speech sample.

Images