Computer-implemented method, non-transitory computer-readable storage medium, and system for training neural networks (Knowledge Distillation Using Deep Clustering)

Abstract

To provide a computer-implemented method, a non-transitory computer-readable storage medium and a system for training a neural network.SOLUTION: A method for training a neural network includes clustering a full set of training data samples into a special training cluster. A special teacher neural network is trained using respective special training clusters of the special training clusters. A software label is generated for a full set of training data samples using the special teacher neural network. A student model is trained using the full set of training data samples, special training clusters, and software labels.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates generally to machine learning, and more particularly to training compact models that perform the same functions as larger models. [Background technology]

[0002] Knowledge distillation training transfers knowledge from a relatively large model to a relatively small model. For example, a larger model may underutilize its processing power, so that a smaller model can be trained to perform the same function while consuming fewer resources. Summary of the Invention [Problem to be solved by the invention]

[0003] A computer-implemented method, a non-transitory computer-readable storage medium, and a system for training a neural network are provided. [Means for solving the problem]

[0004] A method for training a neural network includes clustering a full set of training data samples into specialized training clusters. An expert teacher neural network is trained using each of the specialized training clusters. Soft labels are generated for the full set of training data samples using the expert teacher neural network. A student model is trained using the full set of training data samples, the specialized training clusters, and the soft labels.

[0005] A system for training a neural network includes a hardware processor and a memory storing computer program code. When executed by the hardware processor, the computer program code implements a student model, a clustering network, multiple expert teacher neural networks, and a model trainer. The clustering network clusters a full set of training data samples into multiple expert training clusters. The expert teacher neural networks jointly generate soft labels for the full set of training data samples. The model trainer trains the multiple expert teacher neural networks using each of the multiple expert training clusters, and trains a student model using the full set of training data samples, the expert training clusters, and the soft labels.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

[0007] The following description provides details of preferred embodiments with reference to the following drawings: [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of training and using a student model based on a general teacher model and an expert teacher model for different acoustic conditions according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block / flow diagram of a method for training a student model based on a general teacher model and an expert teacher model for different acoustic conditions according to one embodiment of the present invention. [Figure 3] FIG. 2 is a block / flow diagram of a method for performing speech recognition and natural language tasks on new utterances using trained student models according to one embodiment of the present invention. [Figure 4]FIG. 1 is a block diagram of a natural language processing system for training a student model based on a general teacher model and an expert teacher model for different acoustic conditions, according to one embodiment of the present invention. [Figure 5] FIG. 1 is a generalized diagram of an artificial neural network illustrating features of a neural network, according to one embodiment of the present invention. [Figure 6] FIG. 1 is a generalized diagram of the architecture of a particular artificial neural network illustrating features of the neural network, according to one embodiment of the present invention. [Figure 7] 1 is a block diagram illustrating an exemplary cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud consumers communicate, according to one embodiment. [Figure 8] FIG. 1 is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Knowledge distillation can take a relatively large "teacher model" and use it to train a relatively small "student model." The student model can incorporate domain-specific knowledge that represents clustered data. In speech recognition and other acoustic processing tasks, such models can incorporate expert teachers for various acoustic conditions, which may include differences in background noise and differences in speaker accents and influences.

[0010] Multiple data clusters are generated and can be used to form expert teacher models using unsupervised deep clustering. A training pipeline is described herein that uses multiple teachers to determine soft labels, which are then used to train student models. Because it is difficult to simultaneously represent multiple acoustic conditions in a single model, using expert teacher models helps capture differences in acoustic conditions when training student models.

[0011] One goal in automatic speech recognition is to provide acoustic modeling that is robust over a wide range of acoustic conditions. While the model is specifically described herein as working with acoustic data, it should be understood that other forms of data may be used instead. Acoustic data may be represented, for example, as a waveform representing the intensity of air pressure over time, although there are many other forms of information that can be represented as similar waveforms. Thus, input data may include speech signals within the human audible range, infrasound or ultrasonic signals outside the human audible range, electromagnetic signals including waveforms of energy within the electromagnetic spectrum, seismic signals recording earth movement, or any other suitable waveform data.

[0012] Although training data may be collected under various conditions, such as different acoustic conditions, which may include speaker age, location, group identity, speaking rate, channel or environmental conditions, or ambient noise level, it is not assumed that the data contains labels with such information. Therefore, a priori information about the acoustic conditions may not be available. Therefore, the training data may be clustered using unsupervised deep clustering to group utterances according to acoustic conditions, and these clusters may be used to train condition-specific teacher networks.

[0013] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0014] 1, an overview of an acoustic processing task is shown. A speaker 102 makes an utterance that is recorded by a microphone 104, which converts the air pressure waves of the utterance into a waveform. The waveform is processed by a trained student model 106, which may, for example, perform speech recognition on the waveform to determine what the speaker 102 said. The information generated by the trained student model 106 is then processed as part of a natural language task 112, for example, to perform a query for requested information or to perform an action in response to a command.

[0015] The trained student model 106 can be trained by a set of teacher models, including a general teacher model 108 and an expert teacher model 110. The general teacher model 108 can be trained with the full set of training data, regardless of the acoustic conditions of each constituent waveform. Exemplary structures of the general teacher model 108 can include a bidirectional long short-term memory network, a visual geometry group (VGG) convolutional neural network, and a residual neural network. The general teacher model 108 can be used to determine how the expert teacher model 110 represents different characteristics from the general environment. The general teacher model 108 is also used to estimate the log-likelihood of context-dependent phonemes as features used in unsupervised clustering so that the characteristics of automatic speech recognition are reflected in the clustering.

[0016] The specialized teacher models 110 may each be trained according to a subset of the training data, for example, after the training data has been clustered according to different acoustic conditions. The specialized teacher models 110 may be implemented as model adaptation, starting from the general teacher model 108 and maintaining the fundamentals of each neuron across different teacher models. Compared to the general teacher model 108 and the specialized teacher model 110, the student model 106 may be a neural network with fewer network parameters. As a result, the student model 106 may operate faster than the teacher model. In one particular example, the student model used to decode a speech signal may have four fully connected layers with 2048 nodes per layer, plus two convolutional layers with 128 and 256 hidden nodes, respectively, to estimate the posterior probabilities of 9300 output targets. The output of the student model 106 may have the same format as the output of the teacher model.

[0017] Convolutional neural networks (CNNs) process information using a sliding "window" across the input, with each neuron in the CNN layer having a respective "filter" applied to each window position. Each filter can be trained, for example, to process a different pattern in the input. CNNs are particularly useful when processing images, where local relationships between individual pixels can be captured by the filter as it passes through different regions of the image. The output of a neuron in a CNN layer includes a set of values ​​representing whether the respective filter matched each set of values ​​in the sliding window.

[0018] Clustering may be performed on training data waveforms represented as speaker vectors, each containing sound features and a sound-level confidence score for that speaker. The speaker vectors including sound-level confidence scores can be determined from the average over a sequence of feature vectors from the same speaker.

[0019] Thus, the general teacher model 108 may be trained on all waveforms in the dataset, or may be trained using a subset that represents the average acoustic conditions of the training data. This subset can be generated, for example, by randomly selecting utterances from the training data pool. The initial acoustic model used to generate the confidence scores may also be trained from the same data.

[0020] After clustering, the training data is divided into N classes and unsupervised deep learning is used to generate the acoustic domain-specific dataset d n (n is one of N classes) can be created. Then, the expert teacher model 110 calculates the n is used to train and adapt the general teacher model 108.

[0021] To train the student model 106, a corresponding expert teacher model 110 is used to generate soft labels. n The student model 106 then generates the corresponding soft label set l n Using all the training data d n The soft labels generated by the expert teacher model 110 can have better quality for the corresponding acoustic condition domain than the general teacher model 108. Therefore, the student model 106 implicitly includes the discriminative capabilities of the various expert teacher models 110.

[0022] 2, a method for training a student model 106 is shown. Block 202 begins with training a general teacher model 108, for example, using the entire training dataset or a randomly sampled subset of the entire training dataset. This training can be performed, for example, using stochastic gradient descent and backpropagation to minimize the cross-entropy loss between the predicted output from the teacher model and the given training data labels.

[0023] Block 204 performs clustering on the training data, for example, using unsupervised deep clustering. Deep clustering may, for example, use a deep neural network to iteratively group features using a clustering process (e.g., K-means) and use subsequent assignments as supervision to update the network weights. Each iteration of clustering is used to provide pseudo-labels for the training data, which are then used in the next iteration of clustering. N clusters d, each containing a respective subset of the training data waveforms, are generated. n A set of , is formed. Although other forms of clustering can be used instead of deep clustering, the deep clustering approach provides superior performance in a framework based on multiple specialized models.

[0024] Block 204 may employ a clustering neural network model that may include a feature mapping layer, a classification layer, and a k-means clustering component. The feature mapping layer converts input signals into intermediate feature representations. The classification layer then classifies the intermediate representations. Once weight parameters are updated, the feature mapping and classification layers are combined.

[0025] TIFF0007808400000001.tif93168

[0026] It can be assumed that each speaker speaks in a specific acoustic condition. All utterances in the training data can be represented using a D-dimensional supervector, which can be formed by concatenating the averaged log-mel frequency features (and their deltas and double deltas) for all context-dependent phonemes. The averaged log-mel frequency features can be calculated after forced alignment is performed on the utterances. The average log-likelihood of each phoneme obtained during the alignment estimation process can also be added to the supervector, resulting in a size of the supervector: D = F × P + S. In this case, F is the dimension of the log-mel frequency features, P is the number of context-dependent phonemes, and S is the number of log-likelihoods used in creating the supervector, which can be equal to P.

[0027] If the phoneme context dependency is, for example, triphones or quinphones, the size of the supervector may be too large to be used as practical input for the clustering network. Therefore, the supervector may be compressed to a more manageable size, for example by principal component analysis (PCA), or context-independent phonemes may be used instead.

[0028] If the utterance is phonetically balanced, the utterance vector may be a good representation of only the acoustic features, since channel-related features are averaged out. In contrast, if the utterance is short or unbalanced, the vector may contain phonetically biased components. However, such information is useful because it represents linguistically meaningful features that can be used by the expert teacher model 110. Using a supervector that includes all utterances from each speaker helps obtain a phonetically balanced representation.

[0029] Thus, to use this system to cluster training data, phone alignment information can be generated, indicating which phoneme each frame belongs to. By averaging over the same phonemes, a supervector can be generated for each speaker. Each phone has a first part of the speaker vector containing the corresponding features, followed by an average confidence score for that phone. The phone-level confidence score can be estimated and added to the speaker vector to incorporate speech recognition information for clustering. This creates an additional set of components for each phone, corresponding to the number of phonemes.

[0030] Block 206 trains the expert teacher models 110. Each expert teacher model 110 trains the expert teacher models 110 for each subset d n as training data to generate models that are trained on waveforms with specific acoustic conditions. These expert teacher models 110 are then used in block 208 to generate soft labels q for each subset of the training data. sp , while using a general supervised model 108 to generate soft labels q for the entire set of training data. bl Generate.

[0031] Next, block 210 uses the soft labels and the clustered training data to train the student model 106. Block 210 can use a bidirectional loss function to train the student model 106. For example, the loss function can be expressed as follows: TIFF0007808400000002.tif42133 where λ is a hyperparameter that weights two different loss components, and q bl (i|x) represents the soft labels from the general teacher model108, and q sp (i|x) represents the soft label from the expert teacher model 110, x is the input feature, and p(i|x) is the output probability of the class from the student model 106 when i denotes the index of the context-dependent phoneme. The hyperparameter can be set to, for example, 0.5.

[0032] Professional Teacher Model 110 Loss L sp First, the classified data d n The predicted output of and the corresponding soft label q sp (i|x). The same data is passed through the generalized training model 108 to obtain the loss L for the generalized training model 108. bl The parameters of the student model 106 are updated to minimize the coupling loss L.

[0033] The two components of the loss function seek to minimize the difference between the distributions of the student model 106 and the general teacher model 108, and between the distributions of the student model 106 and the expert teacher model 110, respectively. With soft labels q, for each training example, competing classes have a small but non-zero probability. The Kullback-Leibler (KL) information criterion, which can be used to train the student model 106, also equivalently minimizes the combined cross-entropy of the soft target labels.

[0034] 3, a method for performing a natural language task is shown. Block 302 trains a student model 106 with a set of training data containing various speech waveforms, for example, using a general teacher model 108 and an expert teacher model 110. Block 304 records new utterances, for example, using a microphone 104.

[0035] Block 306 performs speech recognition on the new utterance, for example, using the trained student model 106. The trained student model 106 may, for example, output a set of phonemes that correspond to different sounds in the utterance. In some cases, the trained student model 106 may further convert the phonemes into complete words and sentences and represent the new utterance as a text string.

[0036] Next, block 308 executes a natural language task 112 on the text. For example, the natural language task 112 may include a chatbot that implements a question-and-answer system. In such an example, the natural language task 112 identifies the user's intent from the spoken words, identifies a request for specific information, and provides the requested information. In another example, the natural language task 112 may include an execution engine that executes the commands encoded in the new utterance.

[0037] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0038] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. Computer-readable storage devices, as used herein, should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0039] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computer / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computer / processing device for storage.

[0040] The computer-readable program instructions for carrying out the operations of the present invention can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions can execute entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry for carrying out aspects of the present invention.

[0041] Aspects of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer-readable program instructions.

[0042] The computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, create means for performing the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner. The computer-readable storage medium having instructions stored thereon thereby constitutes an article of manufacture including instructions for performing aspects of the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0043] Computer-readable program instructions may also be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0044] References herein to "one embodiment" or "an embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with this embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout this specification, and other variations thereof, are not necessarily all referring to the same embodiment.

[0045] It should be understood that the use of any of the following " / ", "and / or", and "at least one of" is intended to encompass selection of only the first listed option (A), or selection of only the second listed option (B), or selection of both options (A and B), for example, in the cases of "A / B", "A and / or B", and "at least one of A and B". As yet another example, for "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selection of only the first listed option (A), selection of only the second listed option (B), selection of only the third listed option (C), selection of only the first and second listed options (A and B), selection of only the first and third listed options (A and C), selection of only the second and third listed options (B and C), or selection of all three options (A, B, and C). This can be expanded as many times as the number of listed items, as would be readily apparent to one of ordinary skill in this and related arts.

[0046] The flowcharts and block diagrams in the figures of this disclosure illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing a specific logical function. In some implementations, the functions depicted in the blocks may be performed in a different order than that depicted in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, executed simultaneously or substantially simultaneously, executed in a partially or fully overlapping manner, or executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs a specific function or operation, or by a combination of dedicated hardware and computer instructions.

[0047] As used herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution units, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, or a separate processor- or computing element-based controller (e.g., logic gates, etc.), or a combination thereof. The hardware processor subsystem can include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, the hardware processor subsystem can include one or more memories that may be on-board or off-board or dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0048] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, including an operating system, one or more applications, or specific code, or combinations thereof, to achieve a particular result.

[0049] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a particular result. Such circuitry may include one or more application-specific integrated circuits (ASICs), FPGAs, or PLAs, or a combination thereof.

[0050] These and other variations of the hardware processor subsystem are contemplated according to embodiments of the present invention.

[0051] 4, there is shown a natural language processing system 400. The system 400 includes a hardware processor 402 and a memory 404. A sensor interface 406 receives waveform information from a sensor, such as the microphone 104. Training data 408 may be stored in the memory 404.

[0052] The clustering network 410 clusters the training data 408 as described above to provide training subsets corresponding to different acoustic conditions. The training data 408 is used by the model trainer 412 to train the general teacher model 108. The clustered subsets of the training data 408 generated by the clustering network 410 are used by the model trainer 412 to train respective specialized teacher models 110. Soft labels are generated for the training data using the general teacher model 108 and the specialized teacher model 110, and the model trainer 412 uses these soft labels to train the student model 106.

[0053] The trained student model 106 receives new utterances from the sensor interface 406. The student model 106 performs, for example, automatic speech recognition on the utterance and generates a text output. The natural language task 112 then accepts the text output and provides an appropriate response.

[0054] The general teacher model 108, the expert teacher model 110, and the student model 106 may each be implemented as artificial neural networks (ANNs). ANNs are information processing systems inspired by biological nervous systems such as the brain. A key element of an ANN is its structure, which includes a large number of highly interconnected processing elements (called "neurons") that operate in parallel to solve a specific problem. ANNs are further trained during use, involving learning to adjust the weights present between neurons. Through this learning process, ANNs are configured for specific applications, such as pattern recognition or data classification.

[0055] Referring now to Figure 5, a generalized diagram of a neural network is shown. ANNs demonstrate the ability to extract meaning from complex or imprecise data and can be used to extract patterns and detect trends that are too complex to be detected by humans or other computer-based systems. The structure of a neural network is generally known as having input neurons 502 that provide information to one or more "hidden" neurons 504. The connections 508 between the input neurons 502 and the hidden neurons 504 are weighted, and these weighted inputs are processed by the hidden neurons 504 through weighted connections 508 between layers according to the function of the hidden neurons 504. There can be any number of layers of hidden neurons 504, as well as neurons that perform different functions. Different neural network structures also exist, such as convolutional neural networks and max-out networks. Finally, a set of output neurons 506 accepts and processes the weighted inputs from the last set of hidden neurons 504.

[0056] This represents a "feedforward" computation in which information is propagated from input neurons 502 to output neurons 506. Once the feedforward computation is complete, the output is compared to the desired output obtained from the training data. Errors with respect to the training data are then processed in a "feedback" computation in which hidden neurons 504 and input neurons 502 receive information about the error propagating backward from output neurons 506. Once the backward error propagation is complete, a weight update is performed in which the weighted connections 508 are updated to account for the received error. This is just one type of ANN.

[0057] Referring now to FIG. 6, an exemplary architecture 600 for an ANN is shown. It should be understood that this architecture is purely exemplary and that other architectures or types of neural networks may be used instead. In particular, while a hardware embodiment of an ANN is described herein, it should be understood that the neural network architecture may be implemented or simulated in software. The hardware embodiment described herein is included with the intent of illustrating the general principles of neural network computation at a high level of generality and should not be construed as limiting in any way.

[0058] Furthermore, the layers of neurons and their connecting weights described below are described in a general manner and can be replaced with any type of neural network layer having any appropriate degree or type of interconnectivity. For example, the layers can include convolutional layers, pooling layers, fully connected layers, softmax layers, or other appropriate types of neural network layers. Furthermore, layers can be added or removed as needed, and weights can be omitted in favor of more complex forms of interconnection.

[0059] During feedforward operation, each of the set of input neurons 602 supplies a parallel input voltage to a respective column of weights 604. In the hardware embodiment described herein, each of the weights 604 has a configurable resistance value such that a current output flows from the weight 604 to a respective hidden neuron 606 to represent the weighted input. In a software embodiment, the weights 604 may simply be represented as a coefficient value that is multiplied against the associated neuron output.

[0060] According to a hardware embodiment, the current output by a given weight 604 is determined as I = v / r, where V is the input voltage from the input neuron 602 and r is the set resistance of the weight 604. The currents from each weight are summed column-wise and flow to the hidden neuron 606. A set of reference weights 607 have fixed resistances and combine their outputs into a reference current supplied to each hidden neuron 606. Because conductance values ​​can only be positive, some reference conductance is necessary to encode both positive and negative values ​​in the matrix. Because the currents generated by the weights 604 are continuously valued and positive, the reference weights 607 are used to provide a reference current; above the reference current, the current is considered to have a positive value, and below the reference current, the current is considered to have a negative value. The use of reference weights 607 is not required in software embodiments, where the output and weight values ​​are obtained accurately and directly. Instead of using reference weights 607, another embodiment could use a separate array of weights 604 to capture negative values.

[0061] Hidden neurons 606 use currents from the array of weights 604 and reference weights 607 to perform some calculation. Hidden neurons 606 then output their voltages to another array of weights 604. This array operates in a similar manner, with columns of weights 604 receiving voltages from each hidden neuron 606 and summing row-wise to produce a weighted current output that is fed to output neuron 608.

[0062] It should be understood that any number of these stages can be implemented by interposing additional layers of arrays and hidden neurons 606. Note also that some neurons may be constant neurons 609 that provide constant outputs to the array. Constant neurons 609 may reside between input neurons 602 or hidden neurons 606 or both, and are used only during feedforward operation.

[0063] During backpropagation, output neurons 608 return voltages to the array of weights 604. The output layer compares the generated network response to training data and calculates an error. The error is applied to the array as a voltage pulse, with the pulse height and / or duration modulated proportionally to the error value. In this example, columns of weights 604 receive voltages in parallel from each output neuron 608 and convert the voltages into currents that, summed column by column, provide inputs to hidden neurons 606. Hidden neurons 606 combine a weighted feedback signal with the derivative of their feedforward calculation, store an error value, and then output a feedback signal voltage to their respective columns of weights 604. This backpropagation propagates throughout the network 600 until all hidden neurons 606 and input neurons 602 store error values.

[0064] During weight updates, input neurons 602 and hidden neurons 606 apply a first weight update voltage in the forward direction, and output neurons 608 and hidden neurons 606 apply a second weight update voltage in the reverse direction through network 600. The combination of these voltages causes a state change in each weight 604, causing the weights 604 to assume new resistance values. In this way, weights 604 can be trained to allow neural network 600 to adapt to errors in its processing. Note that the three modes of operation—feedforward, backpropagation, and weight update—are mutually exclusive.

[0065] As described above, the weights 604 can be implemented in software or hardware, for example, using relatively complex weighting circuits or using resistive cross-point devices. Such resistive devices can have nonlinear switching characteristics that can be used for data processing. Because their nonlinear characteristics are used to perform calculations in the neural network 600, the weights 604 can belong to a class of devices called resistive processing units (RPUs). RPU devices can be implemented with resistive random access memory (RRAM), phase change memory (PCM), programmable metallization cell (PMC) memory, or any other device with nonlinear resistive switching characteristics. Such RPU devices can also be considered memristor systems.

[0066] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0067] The characteristics are as follows:

[0068] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider.

[0069] Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, PDAs).

[0070] Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers generally have no control or knowledge of the exact location of the resources they are provided with, resulting in a sense of location independence. However, consumers may be able to determine location at a higher level of abstraction (e.g., country, state, data center).

[0071] Rapid Elasticity: Computing capacity can be provisioned quickly and elastically, sometimes automatically, to instantly scale out and quickly release to instantly scale in. To the consumer, the computing power available for provisioning often appears unlimited, and can be purchased at any time and in any quantity.

[0072] Metered Services: Cloud systems leverage measurement capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of utilized services.

[0073] The service model is as follows:

[0074] Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure that can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functionality, except for limited user-specific application configuration settings.

[0075] Platform as a Service (PaaS): The capability offered to consumers is to deploy applications they create or acquire using programming languages ​​and tools supported by the provider onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of their hosting environment.

[0076] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provisioning of processors, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, and deployed applications, and in some cases partial control over some network components (e.g., host firewalls).

[0077] The deployment model is as follows:

[0078] Private Cloud: This cloud infrastructure is dedicated to a specific organization and can be managed by that organization or a third party, and can exist on-premise or off-premise.

[0079] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by those organizations or a third party and can exist on-premises or off-premises.

[0080] Public cloud: This cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0081] Hybrid cloud: This cloud infrastructure combines two or more cloud models (private, community, or public), each of which retains its inherent nuances but is bound by standards or specific technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0082] A cloud computing environment is a service-oriented environment that emphasizes statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0083] FIG. 7 illustrates an exemplary cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers (e.g., PDAs or cell phones 54A, desktop computers 54B, laptop computers 54C, or automobile computer systems 54N, or combinations thereof) can communicate. The nodes 10 can communicate with each other. The nodes 10 can be physically or virtually grouped (not shown) in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software as a service, or combinations thereof, for which cloud consumers are not required to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N illustrated in FIG. 7 are merely exemplary, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network-addressable connection (e.g., using a web browser), or both.

[0084] A set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 7) is now shown in FIG. 8. It should be understood in advance that the components, layers, and functions shown in FIG. 8 are merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0085] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage device 65, and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0086] The virtualization layer 70 provides an abstraction layer from which the following virtual entities can be provided, for example: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0087] By way of example, the management layer 80 may provide the following functionality: Resource provisioning 81 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. By way of example, these resources may include application software licenses. Security enables identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 enables advance arrangement and procurement of anticipated future cloud computing resources required in accordance with SLAs.

[0088] The workload layer 90 provides examples of functionality available to a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and model training 96.

[0089] Having described a preferred embodiment of knowledge distillation using deep clustering (intended to be illustrative and not limiting), it should be noted that modifications and variations may occur to those skilled in the art in light of the above teachings. It is therefore understood that changes can be made to the particular embodiments disclosed that are within the scope of the invention, as outlined by the appended claims. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A computer-implemented method for training a neural network, comprising: clustering a full set of training data samples including acoustic waveforms into a plurality of specialized training clusters according to acoustic criteria; training a plurality of expert teacher neural networks using respective expert training clusters of the plurality of expert training clusters; generating soft labels for each subset of the training data samples using the plurality of expert supervised neural networks; training a generalized supervised neural network using the full set of training data samples; generating soft labels for the full set of training data samples using the general supervised neural network; training a student model using the full set of training data samples, the specialized training clusters, soft labels from the specialized teacher neural network, and soft labels from the general teacher neural network; Including, Training the student model includes minimizing a loss function Computer-implemented methods. where λ is a weight hyperparameter, q bl is a set of soft labels from the general teacher neural network, q sp is a set of soft labels from the multiple expert teacher neural networks, and p(i|x) is the class probability from the student model when i denotes the index of a context-dependent phoneme and x denotes the input signal.

2. The computer-implemented method of claim 1 , wherein clustering comprises performing unsupervised deep clustering.

3. The computer-implemented method of claim 1 , wherein the student model is a neural network having fewer parameters than the general teacher neural network.

4. The computer-implemented method of claim 1 , further comprising performing speech recognition on new utterances using the trained student model.

5. The computer-implemented method of claim 4 , further comprising performing a natural language task on the recognized speech from the new utterance.

6. 1. A computer program for training a neural network, the computer program, when executed on a computer, causing the computer to: clustering a full set of training data samples including acoustic waveforms into a plurality of specialized training clusters according to acoustic criteria; training a plurality of expert teacher neural networks using respective expert training clusters of the plurality of expert training clusters; generating soft labels for each subset of the training data samples using the plurality of expert supervised neural networks; training a generalized supervised neural network using the full set of training data samples; generating soft labels for the full set of training data samples using the general supervised neural network; training a student model using the full set of training data samples, the specialized training clusters, soft labels from the specialized teacher neural network, and soft labels from the general teacher neural network; Let Training the student model includes minimizing a loss function Computer program.

7. The computer program of claim 6 , further causing the computer to perform unsupervised deep clustering on the full set of training data samples.

8. The computer program of claim 6 , wherein the student model is a neural network having fewer parameters than the general teacher neural network.

9. The computer program of claim 6 , further causing the computer to perform speech recognition on new utterances using the trained student model.

10. 1. A system for training a neural network, comprising: a hardware processor; and a memory for storing computer program code, the computer program code, when executed by the hardware processor, Student models and a clustering network that clusters the full set of training data samples, including acoustic waveforms, into a plurality of specialized training clusters according to acoustic criteria; a plurality of expert supervised neural networks that jointly generate soft labels for each subset of the training data samples; a general supervised neural network that jointly generates soft labels for the full set of training data samples; a model trainer that trains the plurality of expert teacher neural networks using each of the plurality of specialized training clusters, trains a general teacher neural network using the full set of training data samples, and trains a student model using the full set of training data samples, the expert training clusters, soft labels from the expert teacher neural network, and soft labels from the general teacher neural network; Implement The system, wherein training the student model includes minimizing a loss function: where λ is a weight hyperparameter, q bl is a set of soft labels from the general teacher neural network, q sp is a set of soft labels from the multiple expert teacher neural networks, and p(i|x) is the class probability from the student model when i denotes the index of a context-dependent phoneme and x denotes the input signal.

Images