Hybrid data augmentation for a knowledge distillation framework

Abstract

A method of training a student neural network is provided. The method includes feeding a dataset comprising a plurality of input vectors into a teacher neural network to generate a plurality of output values, and converting two of the plurality of output values from the teacher neural network for two respective input vectors into two respective soft labels. The method also includes combining the two corresponding input vectors to form a synthetic data vector, and forming a masked soft label vector from the two corresponding soft labels. The method further includes feeding the synthetic data vector into a student neural network, using the masked soft label vector to determine an error for modifying weights of the student neural network, and modifying the weights of the student neural network.

Description

BACKGROUND

[0001] The present disclosure relates generally to training neural networks, and more specifically to using mixed augmentation in training neural networks with knowledge distillation for speech recognition.

[0002] In an artificial neural network (ANN), "learning" occurs through changes in weights applied to data inputs to each neuron in the neural network. Depending on the neural network architecture, an artificial neural network can have one or more layers of neurons. Training of a neural network can be performed using training pairs, including input data and an expected output / result (i.e., hard labels). Training the neural network then involves feeding the training pairs into the neural network and generating a prediction (i.e., soft labels) for the output. The resulting or predicted output can be compared to the expected output for each training pair to determine the correctness or incorrectness of the prediction. Feedback from the correctness and incorrectness (e.g., errors) of the prediction compared to the expected output can be used to adjust the weights applied to the data inputs to make subsequent predictions more accurate. The weights can be adjusted continuously using multiple training pairs and / or multiple iterations (e.g., epochs) of training.

[0003] A time series is a collection of observations that proceed sequentially in time, where each piece of data can have its own timestamp. Time problems can be solved by observing past values of a time series to predict future values. Some neural networks can utilize memory to process temporal information of inputs. Recurrent neural networks (RNNs) have feedback connections and internal states that can provide such memory. In speech recognition, a neural network can be trained to translate an audio file that includes phonemes, which can be translated into characters and words that can have contextual meaning. A phoneme is a unit of sound that can distinguish one word from another in a particular language, while a phone is any distinct speech sound.

[0004] Automatic speech recognition (ASR) systems can process user input from different acoustic settings. Speech sequences processed by such systems can include input from environments with stationary and non-stationary noise, such as a restaurant or an exhibition hall, meeting recordings in very reverberant settings, emotional speech, and distorted signals with time-frequency bandwidth loss due to faulty equipment or transmission network conditions.

[0005] However, speech recognition can be computationally too demanding to run on edge devices like smartphones and other embedded devices. Neural models can have millions or even billions of parameters, which can make them too large to deploy on edge devices (e.g., smartphones, tablets, etc.). Knowledge distillation involves model compression by teaching a smaller neural network what a larger trained neural network does. A student network can be trained to learn the behavior of a teacher network by trying to replicate the output at each level / layer instead of just replicating the final value. SUMMARY

[0006] According to an embodiment of the present invention, a method of training a student neural network is provided. The method includes feeding a dataset comprising a plurality of input vectors into a teacher neural network to generate a plurality of output values, and converting two output values of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels. The method also includes combining the two corresponding input vectors to form a synthetic data vector, and forming a masked soft label vector from the two corresponding soft labels. The method further includes feeding the synthetic data vector into a student neural network, using the masked soft label vector to determine an error for modifying weights of the student neural network, and modifying the weights of the student neural network.

[0007] According to another embodiment of the present invention, a processing system for training a student neural network is provided. The processing system includes one or more processor devices, a memory in communication with at least one of the one or more processor devices, and a display screen, wherein the memory includes: a teacher neural network configured to receive a dataset comprising a plurality of input vectors and generate a plurality of output values; a softmax function configured to convert two output values of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels; a hybrid generator configured to combine the two corresponding input vectors to form a synthetic data vector; a masker configured to form a masked soft label vector from the two corresponding soft labels; and the student neural network configured to receive the synthetic data vector and the masked soft labels, and train the student neural network based on an error computed from the masked soft labels.

[0008] According to yet another embodiment of the present invention, there is provided a non-transitory computer readable storage medium comprising a computer readable program for training a student neural network. The computer readable program includes instructions for feeding a dataset comprising a plurality of input vectors into a teacher neural network to produce a plurality of output values, and converting two output values of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels. The computer readable program also includes instructions for combining the two corresponding input vectors to form a synthetic data vector and forming a masked soft label vector from the two corresponding soft labels. The computer readable program also includes instructions for feeding the synthetic data vector into the student neural network, using the masked soft label vector to determine an error for modifying weights of the student neural network, and modifying the weights of the student neural network.

[0009] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0010] The following description will provide details of preferred embodiments referring to the following drawings in which:

[0011] Figure 1 is a block diagram / flowchart illustrating a system / method for training a student neural network using knowledge distillation with soft targets (labels) generated by a teacher network rather than hard targets (one-hot vectors) in accordance with embodiments of the present invention;

[0012] Figure 2 is a block diagram / flowchart illustrating a system / method for training a student neural network using knowledge distillation with soft targets generated by a teacher network in accordance with embodiments of the present invention;

[0013] Figure 3 is a diagram illustrating the application of a softmax function to output values that can be generated by a teacher neural network to generate probabilities that can then be compared to one-hot vectors in accordance with embodiments of the present invention;

[0014] Figure 4 is a diagram illustrating the application of masking to soft labels generated by a teacher neural network in accordance with embodiments of the present invention;

[0015] Figure 5 is a block diagram / flowchart of a method for hybrid augmentation and masking in accordance with embodiments of the present invention;

[0016] Figure 6 is an exemplary processing system to which the present methods and systems can be applied in accordance with embodiments of the present invention;

[0017] Figure 7 is an exemplary processing system configured to implement one or more neural networks for modeling road layouts according to embodiments of the present application;

[0018] Figure 8 is a block diagram illustrating an exemplary neural network 800 according to another embodiment of the present application;

[0019] Figure 9 is a block diagram illustrating a conceptual cloud computing environment according to one embodiment, with the local computing devices used by cloud consumers in communication with one or more cloud computing nodes;

[0020] Figure 10 is a block diagram illustrating a set of functional abstraction layers provided by the cloud computing environment according to one embodiment. DETAILED DESCRIPTION

[0021] Embodiments of the present application relate to methods and systems that provide knowledge distillation training to train a student network by using soft targets produced by a teacher network instead of hard targets (one-hot vectors) when the same training data is input to both the teacher network and the student network. The methods and systems provided can use the error between the predicted output from the student and the predicted output from the teacher network as a soft target and minimize this error when training the student. The teacher network can be trained using hard targets before the knowledge distillation process. If the student network is set to replicate the behavior of a teacher network that has utilized a larger solution space, it can be desirable for the student network to overlap its convergence space with the original teacher network convergence space.

[0022] In various embodiments, a complex deep neural network (i.e., a teacher network) can be trained using a full set of hard targets / labels, which can be done offline. The deep neural network can be a multi-layer perceptron. A correspondence can be established between the intermediate outputs of the teacher network and the student network. The output from the teacher network can be used to backpropagate the computed error values through the student network so that the student network can learn to replicate the behavior of the teacher network instead of learning directly with the hard targets / labels. The teacher network can thereby effectively transfer its knowledge to a smaller size student network.

[0023] In different embodiments, the method constructs virtual training examples by linearly combining both input features and output labels. In various embodiments, the method can combine an arbitrarily chosen pair of training samples and their associated soft labels to generate new training data: x = (1 - l)x i = l x j ; y = (1 - l)y i + l y j ;

[0024] where xi and x j are original input vectors, and y i and y j are one-hot label vectors associated with these original input vectors x i and x j respectively. The blending can be achieved by simple weighted interpolation on both input features and target labels without using any domain-specific knowledge. In different embodiments, this blending augmentation can be applied to a knowledge distillation (teacher-student training) framework for speech recognition.

[0025] Knowledge distillation can utilize a simple student model for test-time deployment to mimic a complex teacher neural network. However, in addition to directly making the outputs of the teacher and student networks similar, many existing methods fail to directly transfer teacher information in other network layers to the student network. General knowledge distillation techniques can supplement missing information or compensate for low-quality acoustic features with privileged information that can only be available during training time.

[0026] Knowledge distillation is a technique to mimic a complex teacher neural network with a simple student neural network, where the training can be performed in two separate steps. In the first step, a complex teacher neural network, such as a bidirectional LSTM, VGG, or ResNet model, can be initially trained using hard targets (i.e., one-hot vectors). The classification can be done by describing each class in one one-hot encoded vector, where all values are zeroed except for one value indicating the true class, which is 1, where the size of this vector corresponds to the number of nodes in the output layer. For example, if the number of phonemes to be classified is 40, then the size of this one-hot vector will also be 40 x 1, containing a single 1 value identifying the phoneme.

[0027] In an ASR system, the output of the neural network is not a single phoneme (mono-phoneme), but a combination of several consecutive phonemes (e.g., tri-phonemes, penta-phonemes, etc.). The value 1 is set to the position in the one-hot vector corresponding to the input phoneme to the neural network. In contrast to the hard target, the soft target (soft label or soft output) is not a one-hot vector, but all elements in the soft target vector can have non-zero values. The size of the hard target vector is the same as the soft target vector. The soft target can be automatically generated by the teacher network.

[0028] A compact acoustic model of the student neural network can then be trained on the soft labels / target outputs of the teacher network using a training criterion that minimizes the difference between the student output distribution and the teacher output distribution. Fast processing (e.g., small turn-around, real-time factors, and minimal latency) is advantageous for ASR processing. However, less complex student neural networks can be less accurate and have poorer ASR performance than more complex teacher networks. Teacher-student learning (knowledge distillation) can leverage a simple architecture (student neural network) to create a more robust neural network while maintaining fast processing speeds by simulating the behavior of the teacher network through the use of soft labels / targets.

[0029] In various embodiments, the student neural network can be a deep neural network (DNN) with fewer parameters than the teacher neural network. The size of the student neural network can be based on the power and accuracy of the device (e.g., smart phone, tablet, etc.) on which the student neural network is intended to run, while the teacher neural network can be complex enough to provide an accurate model.

[0030] Once a large, computationally demanding model has been trained, distillation can be used to transfer knowledge from the large model to a smaller model that is better suited for deployment. The knowledge imparted to the small model is the learned mapping from input vectors to output vectors, not the learned parameter values of the trained large model. The class probabilities produced by the large model can be used as “soft targets” for training the small model.

[0031] In various embodiments, knowledge distillation can be used to train an efficient student network with privileged knowledge to augment the soft labels used to train the student network. The ‘soft’ labels can be restricted to valid probability distributions. The “soft labels” refer to the output feature maps produced by the larger teacher network after each hidden layer.

[0032] Exemplary applications / uses to which the present invention can be applied include, but are not limited to, call center monitoring, car navigation systems, meeting transcription and minute generation, robotic voice control, web browsing through voice commands, etc.

[0033] It should be appreciated that aspects of the present invention will be described in terms of a given illustrative architecture; however, other architectures, structures, components, and process features and steps can be changed, modified, and / or eliminated in various embodiments of the present invention.

[0034] Referring now to the drawings, in which like numerals refer to like elements throughout the several views, and initially to Figure 1 In accordance with embodiments of the present invention, a system / method is shown for training a student neural network using knowledge distillation with soft targets (labels) generated by a teacher network rather than hard targets (one-hot vectors).

[0035] In one or more embodiments, training data 110 can be fed into a system 100 including a teacher neural network 120 and a student neural network 150, wherein the input data can be a set X = {x n y n}, where n is the index on the training sample, and each x n It can be a feature vector, and each output y n It corresponds to the eigenvector x n Ideal output y n (i.e., one-hot vector). In various embodiments, vector x i and x j It can be fed into teacher network 120, where x i and x j This can be the original input vector containing two distinct elements of X, where i and j are indices. In various embodiments, for {x} n y n} is used to train the teacher model, while a pair of x n and the corresponding x generated by the teacher model n The student model is trained using the actual output of x. In hybrid augmentation, the student model is trained by using x. i and x j Weighted interpolation between the samples is used to create new training samples x. m .

[0036] In various embodiments, the teacher neural network 120 may be a recurrent neural network (RNN), including but not limited to long short-term memory (LSTM), bidirectional long short-term memory (BLSTM), and recurrent multilayer perceptron (RMLP), or a non-recurrent neural network, including but not limited to VVGNET (a convolutional neural network architecture), feedforward neural network (FFNN), and time-delay neural network (TDNN). The teacher neural network 120 may have one or more hidden layers that feed to the output layer 130. In addition to the networks listed above, the teacher network 120 may also have fully connected layers.

[0037] In one or more embodiments, nodes 135 of the output layer 130 of the teacher network 120 can generate raw output values, wherein these raw output values ​​may be derived from the input x. n (For example, x) i and x jgenerated by the nodes 135 can be from a series of phonemes, e.g., k-a-s + a + t, k-a-s + u + t, k-a-s + e + t, p-a-t + a + b, with the central phoneme being "s" or "t," and the preceding phonemes being "k," "p," and "a," and the following phonemes being "a," "e," "u," "i," "t," and "b," where "-" denotes preceding phonemes and "+" denotes following phonemes. In other words, the phoneme corresponding to the input training sample x n may be "s," which can be classified into different classes depending on the neighboring phonemes. There can be as many nodes 135 as phonemes that are classified.

[0038] The raw output values can be fed to a Softmax function 140, which converts the raw output values into probabilities that the input x n belongs to the one or more classes and is referred to as the soft label q = {q1, q2,..., q k}, where k is the index of the node 135 in the output layer. The probabilities produced by the softmax sum to one depending on the design. Thus, when using the Softmax function, in order for the probability of one class to increase, the probability of at least one other class decreases by an equal amount. The argmax value of this output layer 130 and Softmax function 140 can correspond to the estimated class y n as associated with the input. n .

[0039] In different embodiments, the teacher network 120 can be trained with a cross-entropy loss.

[0040] In one or more embodiments, the same training data 110 can be fed into the student neural network 150, where the input data can be the set X = {x n , y n}, where n is the index over the training samples, each x n may be a feature vector, and each output y n is the ideal output y n (i.e., a one-hot vector) corresponding to the feature vector x n . In various embodiments, the vectors x i and x j may be fed into the teacher network 120, where x i and x j may be the raw input vectors, and i and j are indices. In various embodiments, the input data can be phoneme sequences, where the one-hot vectors identify the classes of the phonemes.

[0041] In one or more embodiments, the student neural network 150 can be a deep neural network (DNN), including but not limited to a multi-layer perceptron (MLP) and a convolutional neural network (CNN). The student neural network 150 can have one or more hidden layers that feed an output layer 160. For example, the student neural network 150 can be less complex than the teacher neural network 120 by having fewer hidden layers and / or fewer connections between nodes, and / or fewer free parameters / weights.

[0042] In one or more embodiments, the nodes 165 of the output layer 160 of the student network 150 can produce raw output values, where these raw output values can be real numbers generated from the input x n that involve the likelihood of the training data falling into each of the identified classes. These raw output values can be fed to a Softmax function 170 that converts the raw output values into probabilities that the input x n belongs to the one or more classes. The probabilities produced by the softmax sum to one according to design. Thus, when using a Softmax function, in order for the probability of one class to increase, the probability of at least one other class decreases by an equal amount. The argmax value of this output layer 160 and Softmax function 170 can correspond to the estimated class y n for the input as associated with the one-hot input vector x n .

[0043] In one or more embodiments, once the larger, more complex teacher neural network 120 has been trained on the training set X = {x n , y n}, and the soft labels generated by the Softmax function 140, the soft labels q = {q1, q2,..., q k} can be applied 180 to the loss function used to train the student neural network 150.

[0044]

[0045] where q = {q1, q2,..., q k} is the soft label from the teacher network 120 that is also used as a pseudo label; and where p = {p1, p2,..., p kis the output probability of a class generated by the student network 150; and "k" is the index of a node in the output layer of the neural network. The error between the soft labels from the teacher network 120 and the Softmax values of the student network 150 can be fed back into the student network to train the weights of the student network 150. This can train the student network to mimic the knowledge of the teacher network (logist) while simplifying the student network architecture and complexity. The logit function or log-odd is the logarithm of the odds p / 1-p, where p is the probability.

[0046] f k may be considered as the same process as described above for obtaining the predicted output from the student, and g n is a function that produces the posterior probability of node k, f k may be considered as the same process as described above for obtaining the predicted output from the student, and g n is a function that produces the posterior probability of node k, where f k may be considered as the same process as described above for obtaining the predicted output from the student, and g k may be considered as the same process as described above for obtaining the predicted output from the student. p k = f k (x n ); q k = g k (x n ), where p k is the output probability of a class from the student network, and q k is the soft label from the teacher network, which also serves as the pseudo label p.

[0047] Figure 2 is a block diagram / flowchart illustrating a system / method for training a student neural network using knowledge distillation with soft targets generated using a teacher network, according to an embodiment of the present invention.

[0048] At block 210 of the training method 200, training data is input into the teacher neural network, where the training data can be a set of input-output pairs (x n , y n ), where the expected output y n is the ground truth (e.g., one-hot label). The training set X = {x n , y n} can be a discrete time series or digitized time series data, where the time series can be, for example, phonemes. The training set can be fed into the input nodes of the teacher neural network, where the teacher neural network can be a recurrent neural network with one or more hidden layers and an output layer. The teacher neural network can be trained on the training set X = {x n , y n} to recognize the input vector x nof the classes.

[0049] At block 220, the teacher network can generate soft labels from the input data, where the trained teacher network produces output values at the output nodes of the teacher neural network.

[0050] At block 230, the output values at the output nodes can be operated on by a softmax function that converts the raw output values from the output nodes into input x n belonging to one or more predefined classes, where the probabilities can be referred to as soft labels, q = {q1, q2,..., q k The softmax function is a function that takes a vector of K real values and turns it into a vector of K real values that sum to 1. The input values can be positive, negative, zero, or greater than 1, where the softmax function can convert them into values that sum to 1 between 0 and 1, such that they can be interpreted as probabilities.

[0051] At block 240, a mixup augmentation can be applied to the input features / vectors x j and the teacher softmax labels to generate a mixed training pair. The generated mixed training pair can be fed into the student neural network. x = (1 - l)x i + l x j ; q = (1 - l)q i + l q j

[0052] In different embodiments, the knowledge distillation training method can define a loss function as:

[0053]

[0054] where q = {q1, q2,..., q k} is the soft labels from the teacher network that work as pseudo labels; p = {p1, p2,..., p k} is the output probabilities of the classes from the student network; and k is the index of the nodes in the output layer of the neural network. p k is the output probability of the class produced by the student network. p k = f k (x n ); q k = g k (x n ). f k is a function for producing the posterior probability of node k for an input feature x n on the student network. g k is a function for producing the posterior probability of node k for an input feature x n on the teacher network, where x n is used for gk and f k both.

[0055] In different embodiments, instead of using ground truth labels, the knowledge distillation training method can define a loss function with an index of context-dependent phonemes k as:

[0056]

[0057] where q(k \ x) is a so-called soft label from the teacher network for input feature x. p(k \ x) is the output probability of the class from the student network. With soft labels q(k \ x), competing classes can have small but non-zero posterior probabilities for each training example. The KL-divergence criterion for training the student model is equivalently also minimizing the cross-entropy of the soft target labels. Typically, the same acoustic feature input is used to generate the posteriors q(k \ x) and p(k \ x).

[0058] In various embodiments, hybrid augmentation can be applied in the knowledge distillation framework in speech recognition by inputting features x i and x j into the teacher network respectively. Then, soft labels q i and q j can be generated from x i and x j and mixed using an interpolation factor (i.e., weight) λ. The student network is trained with the synthetic pair x and q, where:

[0059] Teacher input: x i → output probability, q i , x j → output probability, q j ;

[0060] Student hybrid: x = (1 - λ)x i + λx j → output probability, p;

[0061] Student soft label: q = (1 - λ)q i + λq j .

[0062] The value λ is an interpolation weight (scalar value) when x i and x j (and q i and q j ) are mixed. λ can be in the range of 0 to 1, where if λ is close to 1, the new x is similar to x j . The optimal interpolation weight λ can be found by experimenting on a development set.

[0063] In various embodiments, the training sample(s) x i and x j are mixed with an interpolation weight 0 < λ < 0.5, so the characteristics of the base sample x i may dominate in the mixed training sample. The soft labels can represent a distribution of likelihoods (probabilities) for each phoneme. Thus, if the component of q i is less than the component of q j , then the component in q i may be unreliable because the mixed training sample primarily has the characteristics of the base training sample x i .

[0064] At block 250, the student network can be trained using a set of synthetic training pairs x, q, where q is a synthetic soft label corresponding to the synthetic mixed vector x. Each of the synthetic input vectors x can be fed into the student network. The student neural network can generate an output value for each input synthetic input vector x m .

[0065] At block 260, a softmax function can be applied to the values output by the output layer of the student network to produce a soft label. In various embodiments, the student network can generate a soft label from a synthetic input vector x that can be compared to a synthetic soft label q corresponding to the input vector x.

[0066] At block 270, the soft label can be compared to the combined soft label q of the synthetic training pairs x, q to determine an error value for the student network. The calculated error can be fed back into the student neural network and used to adjust the weights of the student network. In various embodiments, the soft label created by the teacher network is used to update the weights in the student network.

[0067] Figure 3 is a diagram showing the application of a softmax function to output values that can be generated by a teacher neural network to generate probabilities that can then be compared to a one-hot vector, in accordance with an embodiment of the present application.

[0068] In one or more embodiments, a set of output values generated by a teacher neural network or a student neural network can be real numbers greater than one (i.e., > 1) and negative values. An output vector 310 of M x 1 dimensions can include a dominant value 315 for the most likely class and M-1 smaller values 312 for the less likely classes. A softmax function 320 can be applied to the output vector 310 to produce a soft label vector 330 of also M x 1 dimensions. The soft label vector 330 will also have a dominant value 335 for the most likely (highest probability) class and M-1 smaller values 332 for the less likely (lower probability) classes. The values of the soft label vector 330 will also be between 0 and 1. The soft label vector 330 can be compared 340 to a one-hot vector 350 to train the soft label vector 330 to be closer to one (1) in the one-hot vector 350 and the smaller values 332 to be closer to zero (0) 352. The softmax function 320 can be used to convert output values from a teacher neural network into probabilities, where the soft label vector 330 can be created by the softmax function with the values in the output vector 310.

[0069] In various embodiments, the one-hot vector 350 (i.e., the hard target) is used for comparison as the ideal output of a neural network. A teacher neural network can be trained to minimize the difference between the soft label vector 330 and the hard target 350, where the training corpus can be provided manually to the hard target 350. Instead of using the hard target 350 given from the training corpus, the soft labels created by the teacher neural network can be used as the ideal output for training the student network by the concept of knowledge distillation. When training a neural network on a hard target, the predicted output from the neural network will gradually approach the one-hot vector by updating the weights in the network through training.

[0070] Figure 4 is a diagram illustrating applying masking to soft labels generated by a teacher neural network according to an embodiment of the present invention.

[0071] In different embodiments, soft label pairs (q i and q j ) can be used to generate masked soft labels q m . Let x i and q i be a base training sample pair. Because x i and x j are frames that are usually combined together with different phonemes, soft labels with larger interpolation weights (i.e., q i ) can be used to train a student. Speech signals with larger spectral power are the main target signals for automatic speech recognition (ASR). In the soft label components of the base q i , those q jcomponents of the value are unreliable because the mixed training sample mainly has the base training sample x i When the interpolation weight 0 < λ < 0.5. Thus, the soft labels are masked to keep only the reliable parts as follows:

[0072] Soft labels: where M m = {r1, r2,..., r k}, for example, “k” is an index of a node in an output layer of the neural network, and ε is a constant value 0 < ε < 1, which can be close to 0, or can be set to 0.

[0073] In different embodiments, the interpolation weight can be set to 0 < λ < 0.5 to keep the base training sample x i as the dominant training sample. q ik is the soft label when the base training sample (frame) x i is input to the teacher network, while q jk is the soft label when x j is input to the teacher. If they are greater than those component values q j in other soft labels q jk , (q ik > q jk ), then the soft label components q i in q ik are kept, otherwise those components are set to zero.

[0074] In non-limiting exemplary embodiments, the masking process 400 can generate a mask vector 430, where the soft values q i Vector 410 can include values 411, 412, 413, 414 of varying values, where 414 > 413 > 412 > 411. Soft values q j Vector 420 can include values 421, 422, 423, 424 of varying values, where 424 > 423 > 422 > 421. Soft values q i Vector 410 and soft values q j Vector 420 can be produced by the teacher network. The dark gray / shaded boxes represent large values in the vectors, while the light gray / shaded boxes represent small values as output from the teacher neural network. The values of soft values q i Vector 410 can be masked by comparing each value of soft values q j Vector 410 to the corresponding value of soft values q i Vector 420, and by placing zero values (0) 439 in each location of the mask vector 430 by setting ε to zero (0), where the corresponding value of soft values q j Vector 420 is greater than the corresponding soft value qi The values of the vector 410, and the soft values q i The values 431, 432, 433, 434 of the vector 410 are placed in the corresponding positions of the masking vector 430 where the soft values q j The corresponding values of the vector 420 are less than the corresponding soft values q i The soft labels: The corresponding value 433 equal to the value 413 is placed in the corresponding position of the masking vector 430. The corresponding value 434 equal to the value 414 is placed in the corresponding position of the masking vector 430. The corresponding value 432 equal to the value 412 is placed in the corresponding position of the masking vector 430. The corresponding value 431 equal to the value 411 is placed in the corresponding position of the masking vector 430. The parameter ε is a small constant value which can range from 0 to 1 in order to create a mask for unreliable elements of the soft labels. Using zero (0) can be the default value.

[0075] In various embodiments, the nodes in the output layer of the neural network can correspond to classes of phonemes. But we usually do not consider individual phonemes but rather contiguous phonemes (before and after a few phonemes). For example, if the number of phonemes is 40, then the total number of nodes would be 40 x 40 x 40 = 64000 if the ASR system is constructed based on 3 contiguous phonemes of the output layer. However, these 64000 nodes would also represent semantically meaningless combinations of phonemes, so instead, only meaningful combinations of phonemes can be considered, e.g. 9300. In various embodiments, five-phonemes with 2 phonemes before and after can be considered in the output layer of the neural network.

[0076] The hybrid enhancement can be performed only with speech frames (without using silence frames). The hybrid can also be applied to silence frames, but focusing only on speech frames can be more efficient.

[0077] Figure 5 is a block / flow diagram of a method for hybrid enhancement and masking according to an embodiment of the invention.

[0078] In one or more embodiments, the hybrid generation method 500 can generate hybrid training data with a combined training vector and a masking soft label vector.

[0079] At block 510, the first training vector and the second training vector can be fed into a teacher neural network.

[0080] At block 520, the teacher network can generate soft labels.

[0081] At block 530, the first training vector and the second training vector can be combined to produce a combined training vector x m (also called a synthetic data vector). x m = (1 - λ) x i+ λx j .

[0082] At block 540, the soft-labeled masked vector can be generated by the masking process 400. In different embodiments, the interpolation weight λ can be applied to the soft labels corresponding to the two input vectors to form weighted soft labels, where the soft labels corresponding to the two input vectors are weighted using the interpolation weight λ. These weighted soft labels can be used to generate a masked soft label vector, where q m = (1 - λ)q i + λq j .

[0083] At block 550, the mixed training data can be generated by combining the combined training vector with the soft-labeled masked vector.

[0084] Figure 6 is an exemplary processing system 600 to which the present methods and systems can be applied in accordance with embodiments of the present application.

[0085] The processing system 600 can include at least one processor (CPU) 604 and can have graphics processing (GPU) 605 that can perform vector calculations / manipulations that are operatively coupled to other components via a system bus 602. A cache 606, a read only memory (ROM) 608, a random access memory (RAM) 610, an input / output (I / O) adapter 620, a sound adapter 630, a network adapter 640, a user interface adapter 650, and a display adapter 660 can be operatively coupled to the system bus 602.

[0086] A first storage device 622 and a second storage device 624 are operatively coupled to the system bus 602 by the I / O adapter 620. The storage devices 622 and 624 can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid state device, a magnetic storage device, etc. The storage devices 622 and 624 can be the same type of storage device or different types of storage devices.

[0087] A speaker 632 is operatively coupled to the system bus 602 by the sound adapter 630. A transceiver 642 is operatively coupled to the system bus 602 by the network adapter 640. A display device 662 is operatively coupled to the system bus 602 by the display adapter 660.

[0088] First user input device 652, second user input device 654, and third user input device 656 are operatively coupled to system bus 602 via user interface adapter 650. User input devices 652, 654, and 656 can be any device such as a keyboard, mouse, keypad, image capture device, motion sensing device, microphone, or a device combining the functions of at least two of the aforementioned devices. Of course, other types of input devices can also be used while maintaining the spirit of this principle. User input devices 652, 654, and 656 can be of the same type or different types. User input devices 652, 654, and 656 can be used to input information to and output information to system 600.

[0089] In various embodiments, the processing system 600 may also include other elements (not shown), as readily apparent to those skilled in the art, and certain elements may be omitted. For example, different other input and / or output devices may be included in the processing system 600, depending on the specific implementation of the processing system 600, as readily understood by those skilled in the art. For example, different types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memories, etc., in different configurations may be utilized, as readily understood by those skilled in the art. Given the teachings of the principles provided herein, these and other variations of the processing system 600 will readily occur to those skilled in the art.

[0090] Furthermore, it should be understood that system 600 is a computer system for implementing corresponding embodiments of this method / system. Part or all of the processing system 700 may be... Figures 1-5 The method is implemented in one or more of the components. Furthermore, it should be understood that the processing system 600 can perform at least a portion of the methods described herein, including, for example... Figures 1-5 At least a part of the method.

[0091] Figure 7 An exemplary processing system 700, according to an embodiment of the present invention, is configured to implement one or more neural networks for modeling road layouts.

[0092] In one or more embodiments, the processing system 700 may be a computer system 600 configured to perform a computer-implemented method for training a teacher neural network and a student neural network using mixed training data.

[0093] In one or more embodiments, the processing system 700 can be a computer system 600 having a memory component 760 including, but not limited to, a random access memory (RAM) 610, a hard drive 622, and / or cloud storage of the computer system for storing and implementing the computer-implemented method of understanding road layout from video images. The memory component 760 can also utilize a database for organizing memory storage.

[0094] In various embodiments, the memory component 760 can include a teacher neural network 710 that can be configured to implement a plurality of acoustic models configured to model acoustic inputs and perform automatic speech recognition (ASR). In different embodiments, the teacher neural network 710 can be implemented as a long short-term memory (LSTM) or a bidirectional LSTM (BLSTM). The teacher neural network 710 can also be configured to receive acoustic signals, e.g., phoneme sequences, as input and learn to recognize words and context from the phoneme sequences. The input can be a continuous set of phonemes received by the microphone 656. The teacher neural network 710 can also be configured to generate output values and soft labels.

[0095] In various embodiments, the memory component 760 can include a student neural network 720 that can be configured to learn one or more acoustic models from the teacher neural network 710 and configured to model acoustic inputs and perform automatic speech recognition (ASR). In various embodiments, the student neural network 720 can be implemented as a deep neural network (e.g., a multi-layer perceptron) that is smaller (e.g., fewer nodes and / or layers) and less complex (e.g., fewer free parameters to be taught) than the teacher neural network 710. The student neural network 720 can also be configured to generate output values and soft labels.

[0096] In various embodiments, the memory component 760 can include a hybrid generator 730 that can be configured to combine two separate sets of training data and corresponding soft labels, where the training data can be feature / vectors x i and x j The hybrid generator 730 can be configured to produce student hybrid and student soft label training pairs.

[0097] In different embodiments, the memory component 760 can include a masker 740 that is configured to convert two soft values q i vectors into masked vectors. The masker 740 can be configured to receive two soft values q i vectors and perform a masking process to generate masked vectors.

[0098] In different embodiments, the memory component 760 can include a Softmax function 750 configured to generate soft labels from the output values of the teacher neural network 710 and the student neural network 720. The outputs can be input into an error function to adjust the weights of the teacher neural network 710 and the student neural network 720 and / or displayed on a screen for a user to view and implement.

[0099] Figure 8 is a block diagram illustrating an exemplary neural network according to another embodiment of the present application.

[0100] The neural network 800 can include a plurality of neurons / nodes, and output nodes can communicate using one or more of a plurality of connections 808. The neural network 800 can include a plurality of layers, including, for example, one or more input layers 802, one or more hidden layers 804, and one or more output layers 806. In one embodiment, the nodes at each layer can be used to apply any function (e.g., an input program, input data, etc.) to any previous layer to produce an output, and the hidden layers 804 can be used to transform inputs from an input layer (or any other layer) into outputs for the nodes at a different level.

[0101] Figure 9 is a block diagram illustrating an illustrative cloud computing environment with one or more cloud computing nodes that are in communication with a local computing device used by a cloud consumer and with one another.

[0102] It should be appreciated that while the present disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present application are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

[0103] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0104] The characteristics are as follows:

[0105] On-demand self-service: cloud consumers can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.

[0106] Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0107] Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but can be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).

[0108] Rapid elasticity: capabilities can be provisioned and released in a very short period of time (e.g., within minutes), in some cases automatically, to quickly scale out and to rapidly release to quickly scale in. Ability to rapidly provision resources is typically not limited by the location of the resources but rather by the availability of resources on the provider's network.

[0109] Measured service: cloud systems automatically control and optimize resource use by leveraging utilization of resources in an efficient manner, for example, in response to changes in demand. Consumers can have different levels of control or knowledge over the manner in which resources are used.

[0110] Service models are as follows:

[0111] Software as a Service (SaaS): the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0112] Platform as a Service (PaaS): the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

[0113] Infrastructure as a Service (laaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0114] Deployment models are as follows:

[0115] Private cloud: the cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0116] Community cloud: the cloud infrastructure is shared by several organizations and supports mission-critical enterprise resources. It can be managed by the organizations or a third party and can exist on-premises or off-premises.

[0117] Public cloud: the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.

[0118] Hybrid cloud: the cloud infrastructure is a composition of two or more types of cloud (private, community, or public) that remain unique clouds but are bound together using standard or proprietary technologies that enable data and application portability.

[0119] A cloud computing environment is service-oriented, centrally focused on stateless, loosely-coupled, modular, and semantic interoperability. At the core of cloud computing is an infrastructure comprising a network of interconnected nodes.

[0120] Referring now to the drawing Figure 9 , illustrative cloud computing environment 950 is depicted. As shown, cloud computing environment 950 includes one or more cloud computing nodes 910 with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone 951, desktop computer 952, laptop computer 953, and / or automobile computer system 954 can communicate. Nodes 910 can communicate with one another. They can be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment 950 to offer infrastructure, platforms and / or software as services with Figure 9 The types of computing devices 951, 952, 953, 954 shown in FIG. 11 are intended to be illustrative only and computing nodes 110 and cloud computing environment 950 can communicate with any type of computerized devices over any type of network and / or network addressable connection (e.g., using a web browser).

[0121] Figure 10 is a block diagram illustrating a set of functional abstraction layers provided by cloud computing environment 950.

[0122] Referring now to the drawing Figure 10 , a set of functional abstraction layers provided by cloud computing environment 950 (Figure 9 ) a set of functional abstraction layers provided. It should be appreciated that, Figure 10 The components, layers and functions shown in FIG. 10 are intended to be illustrative, and that embodiments of the application are not limited thereto. As described, the following layers and corresponding functions are provided:

[0123] Hardware and software layer 1060 includes hardware and software components. Examples of hardware components include: mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63, blade servers 64; storage devices 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0124] Virtualization layer 1070 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0125] In one example, management layer 1080 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

[0126] Workloads layer 1090 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and teacher neural network 96, where the teacher neural network can be a recurrent neural network configured to learn automatic speech recognition and prepare student neural networks.

[0127] As used herein, the term "hardware processor subsystem" or "hardware processor" can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more particular tasks. In useful embodiments, a hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and / or a separate processor or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem can include one or more on-board memories (e.g., cache, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem can include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that can be on-board or off-board, or can be dedicated for use by the hardware processor subsystem.

[0128] In some embodiments, a hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0129] In other embodiments, a hardware processor subsystem can include specialized, dedicated circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry can include one or more application specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0130] These and other variations of a hardware processor subsystem are also contemplated in accordance with embodiments of the present application.

[0131] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0132] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0133] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0134] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0135] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0136] These computer readable program instructions can be provided to a processor of a computer, or other programmable data processing apparatus, to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include, without limitation, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage technology. When the computer readable program instructions are executed by the computer, other programmable data processing apparatus, or other device, a series of operational steps are performed.

[0137] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include, without limitation, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage technology. When the computer readable program instructions are executed by the computer, other programmable data processing apparatus, or other device, a series of operational steps are performed.

[0138] Reference throughout this specification to “one embodiment” or “an embodiment” and

[0139] It should be understood that any further “ / ”, “and / or”, and “at least one of” as used herein refers to a selection of one or more of the items it follows, disjunctively. As an example, in the phrases “A / B,” “A and / or B,” and “at least one of A and B,” such phrasing should be understood to mean only the selection of the first listed item (A) or the selection of the second listed item (B) or the selection of both items (A and B). As another example, in the phrases “A, B, and / or C” and “at least one of A, B, and C,” such phrasing should be understood to mean the selection of only the first listed item (A), or the selection of only the second listed item (B), or the selection of only the third listed item (C), or the selection of the first and the second listed items (A and B), or the selection of the first and third listed items (A and C), or the selection of the second and third listed items (B and C), or the selection of all three items (A and B and C). This can be extended, as is apparent to one of ordinary skill in this art, for as many items as are listed.

[0140] The computer program product of the present application can be implemented by a hardware device, for example, a microprocessor, Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Programmable Logic Device (PLD) or Field-Programmable Gate Array (FPGA), whether separately-stored software, memory or produced into the hardware device, which can execute the operations, functions and / or methods of the present application.

[0141] Having described preferred embodiments of systems and methods (which are intended to be illustrative and not limiting), it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes can be made in the particular embodiments of the disclosure disclosed which are within the scope and spirit of the disclosure as outlined by the appended claims. Having thus described the aspects of the present application, with the details and particularity required by the patent laws, what is claimed and desired protected by the patent laws is set forth in the following claims.

Claims

1. A computer-implemented method for training a student neural network, comprising: A dataset comprising multiple input vectors is fed into a trained teacher neural network to produce multiple output values ​​corresponding to the multiple input vectors; A pair of output values ​​from the plurality of output values ​​from the trained teacher neural network for a pair of corresponding input vectors is converted into a pair of corresponding soft labels; Combine the pair of corresponding input vectors to form a synthetic data vector; A masking soft-label vector is formed from the pair of corresponding soft labels to mask the components in the pair of corresponding soft labels from the synthetic data vector, wherein the interpolated weight value of the component is less than the interpolated weight value of other components from the pair of corresponding soft labels that have characteristics from the underlying training samples of the dataset; The student neural network is trained to perform speech-to-text conversion using the following methods: The synthesized data vector is fed into the student neural network; The masked soft-label vector is used to determine the error used to modify the weights of the student neural network; and Modify the weights of the student neural network.

2. The method according to claim 1, wherein, The trained teacher neural network is a recurrent neural network and the student neural network is a multilayer perceptron (MLP).

3. The method according to claim 1, wherein, Each of the plurality of input vectors represents a phoneme sequence.

4. The method according to claim 1, further comprising: The masking soft-label vector is formed by comparing the output value of the first input vector in the pair of corresponding input vectors from the trained teacher neural network with the corresponding output value of the second input vector in the pair of corresponding input vectors from the trained teacher neural network.

5. The method according to claim 4, wherein, The output values ​​from the trained teacher neural network are converted into soft tags using the softmax function.

6. The method according to claim 5, wherein, use The input vector is combined with the interpolation weights , where, .

7. The method according to claim 6, further comprising: Interpolation weights are applied to the soft labels corresponding to the pair of corresponding input vectors to form weighted soft labels, wherein the interpolation weights are used to weight the soft labels corresponding to the pair of corresponding input vectors.

8. A processing system for training a student neural network, comprising: One or more processor devices; The memory communicates with at least one of the one or more processor devices; as well as Display screen; The memory includes: A teacher neural network is configured to receive a dataset comprising multiple input vectors and is trained to generate multiple output values ​​corresponding to the multiple input vectors; The softmax function is configured to transform one pair of output values ​​from a plurality of output values ​​from a teacher neural network for a pair of corresponding input vectors into a pair of corresponding soft labels; A hybrid generator is configured to combine the pair of corresponding input vectors to form a synthetic data vector; A masker is configured to form a masking soft-label vector to mask components in the pair of corresponding soft labels from the synthetic data vector, wherein the interpolated weight values ​​of the components are less than the interpolated weight values ​​of other components from the pair of corresponding soft labels that have characteristics from the underlying training samples of the dataset; and The student neural network is configured to receive the synthetic data vector and masking soft tags, and to train the student neural network based on the error calculated from the masking soft tags to perform speech-to-text conversion.

9. The processing system according to claim 8, wherein, The teacher neural network is a recurrent neural network and the student neural network is a multilayer perceptron (MLP).

10. The processing system according to claim 8, wherein, Each of the plurality of input vectors represents a phoneme sequence.

11. The processing system according to claim 8, wherein, The mask is configured to form the masking soft-label vector by comparing the output value of the first input vector in the pair of corresponding input vectors from the teacher neural network with the corresponding output value of the second input vector in the pair of corresponding input vectors from the teacher neural network.

12. The processing system according to claim 11, wherein, use The input vector is combined with the interpolation weights , where, .

13. The processing system according to claim 12, wherein, The hybrid generator is further configured to apply interpolation weights to the soft tags corresponding to the pair of corresponding input vectors to form weighted soft tags, wherein the soft tags corresponding to the pair of corresponding input vectors are weighted using the interpolation weights 𝜆.

14. A non-transitory computer-readable storage medium comprising a computer-readable program for training a student neural network, wherein when executed on a computer, the computer-readable program causes the computer to perform: A dataset comprising multiple input vectors is fed into a trained teacher neural network to produce multiple output values ​​corresponding to the multiple input vectors; A pair of output values ​​from the plurality of output values ​​from the trained teacher neural network for a pair of corresponding input vectors is converted into a pair of corresponding soft labels; Combine the pair of corresponding input vectors to form a synthetic data vector; A masking soft label vector is formed to mask the components in the pair of corresponding soft labels from the synthetic data vector, wherein the interpolated weight values ​​of the components are less than the interpolated weight values ​​of other components from the pair of corresponding soft labels that have characteristics from the underlying training samples of the dataset; The student neural network is trained to perform speech-to-text conversion using the following methods: The synthesized data vector is fed into the student neural network; The masked soft-label vector is used to determine the error used to modify the weights of the student neural network; and Modify the weights of the student neural network.

15. The non-transitory computer-readable storage medium according to claim 14, wherein, The trained teacher neural network is a recurrent neural network and the student neural network is a multilayer perceptron (MLP).

16. The non-transitory computer-readable storage medium according to claim 14, wherein, Each of the plurality of input vectors represents a phoneme sequence.

17. The non-transitory computer-readable storage medium of claim 14, further comprising: The masking soft-label vector is formed by comparing the output value of the first input vector in the pair of corresponding input vectors from the trained teacher neural network with the corresponding output value of the second input vector in the pair of corresponding input vectors from the trained teacher neural network.

18. The non-transitory computer-readable storage medium according to claim 17, wherein, The output values ​​from the trained teacher neural network are converted into soft tags using the softmax function.

19. The non-transitory computer-readable storage medium according to claim 18, wherein, use The input vector is combined with the interpolation weights , where, .

20. The non-transitory computer-readable storage medium of claim 19, further comprising: Interpolation weights are applied to the soft labels corresponding to the pair of corresponding input vectors to form weighted soft labels, wherein the interpolation weights are used to weight the soft labels corresponding to the pair of corresponding input vectors.

Images