Model training methods and equipment

Abstract

A model training method and device are disclosed, wherein the model training method obtains recognition results of a teacher model and recognition results of a student model for an input sequence, and trains the student model so that the recognition results of the teacher model and the recognition results of the student model are indistinguishable from each other.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0053131, filed on May 7, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes. Technical Field

[0002] The following description relates to a model training method and apparatus. Background Art

[0003] Research is actively underway to classify input patterns in a group so that effective pattern recognition can be performed on computers. This includes research into artificial neural networks (ANNs), which are obtained by modeling pattern recognition properties using mathematical expressions using a neural network model implemented via a processor as a dedicated computing architecture. After extensive training, the neural network model can provide a computationally intuitive mapping between input patterns and output patterns. ANNs use algorithms to generate mappings between input patterns and output patterns, and the ability to generate mappings is referred to as the learning ability of the ANN. ANNs can employ algorithms that mimic learning abilities. In addition, ANNs have the ability to generate relatively accurate outputs for input patterns that have not yet been used for training based on the results of previous training. However, because such operations or applications are performed using a dedicated computing architecture and in an automated manner that differs from the automated manner in which they are performed in non-computer-implemented or non-automated methods, they also raise problems or shortcomings that arise simply because of the automation and dedicated computing architecture on which they are implemented.

[0004] Furthermore, research is being conducted to maximize the recognition rate of ANNs while miniaturizing the size of ANNs. Summary of the Invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a method for training a model is provided, the method comprising: obtaining a recognition result of a teacher model and a recognition result of a student model for an input sequence; and training the student model so that the recognition result of the teacher model and the recognition result of the student model cannot be distinguished from each other.

[0007] The step of training the student model may include: determining an adversarial loss based on the extent to which the recognition result of the teacher model and the recognition result of the student model are distinguished from each other; and training the student model to reduce the adversarial loss.

[0008] The determining of the adversarial loss may include determining the adversarial loss based on a degree to which an output sequence of the teacher model and an output sequence of the student model, which are respectively output as recognition results with respect to the input sequence, are distinguished from each other.

[0009] The determining of the adversarial loss may include determining the adversarial loss by applying Gumbel-max based on probabilities of elements included in the output sequence.

[0010] The step of determining the adversarial loss may include determining the adversarial loss by applying Gumbel-max based on a probability of the output sequence.

[0011] The step of determining the adversarial loss may include determining the adversarial loss by further applying Gumbel-max based on probabilities of a plurality of candidate sequences that may correspond to the output sequence.

[0012] The step of determining the adversarial loss may include determining the adversarial loss based on the extent to which elements in an output sequence of the teacher model and an output sequence of the student model, which are respectively output as recognition results for the input sequence, are distinguished from each other.

[0013] The determining of the adversarial loss may include determining the adversarial loss by applying Gumbel-max based on probabilities of elements included in the output sequence.

[0014] The step of training the student model may include: training the student model so that the recognition result of the teacher model and the recognition result of the student model cannot be distinguished from each other by the discriminator model; and the discriminator model may be trained to distinguish between the recognition result of the teacher model and the recognition result of the student model

[0015] The step of training the student model may include: using the recognition result of the teacher model to train the student model, so that the recognition result of the teacher model is output from the student model.

[0016] The input sequence may include sentence data including words and speech data including frames.

[0017] In another general aspect, a device for training a model is provided, the device comprising: a processor configured to: obtain, for an input sequence, a recognition result of a teacher model and a recognition result of a student model; and train the student model so that the recognition result of the teacher model and the recognition result of the student model are indistinguishable from each other.

[0018] The processor may be configured to: determine an adversarial loss based on a degree to which a recognition result of the teacher model and a recognition result of the student model are distinguished from each other; and train the student model to reduce the adversarial loss.

[0019] The processor may be configured to determine the adversarial loss based on the extent to which an output sequence of the teacher model and an output sequence of the student model, which are respectively output as recognition results with respect to the input sequence, are distinguished from each other.

[0020] The processor may be configured to determine the adversarial loss by applying Gumbel-max based on probabilities of elements included in the output sequence.

[0021] The processor may be configured to determine the adversarial loss by applying Gumbel-max based on the probability of the output sequence.

[0022] The processor may be configured to determine the adversarial loss based on the extent to which elements in an output sequence of the teacher model and an output sequence of the student model, which are respectively output as recognition results for the input sequence, are distinguished from each other.

[0023] The processor can be configured to: train the student model so that the recognition results of the teacher model and the recognition results of the student model cannot be distinguished from each other by the discriminator model; and the discriminator model can be trained to distinguish between the recognition results of the teacher model and the recognition results of the student model.

[0024] The processor may be configured to train the student model using the recognition result of the teacher model, so that the recognition result of the teacher model may be output from the student model.

[0025] The device may include a memory configured to store parameters of a student model and to store instructions that, when executed, configure a processor to obtain recognition results of a teacher model and a student model and to train the student model.

[0026] In another general aspect, a device for training a model is provided, the device comprising: a processor configured to: obtain, for an input sequence, a recognition result of a teacher model and a recognition result of a student model; and train the student model to minimize the difference between the recognition result of the teacher model and the recognition result of the student model in a discriminator model.

[0027] The processor may be configured to train a discriminator model to distinguish between recognition results of the teacher model and recognition results of the student model.

[0028] The processor may be configured to train the student model to reduce the adversarial loss by applying Gumbel-max based on the probability of elements included in the output sequence.

[0029] The processor may be configured to train the student model to reduce the adversarial loss by applying Gumbel-max based on the probability of the output sequence.

[0030] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 and Figure 2 Shows examples of teacher models and student models.

[0032] Figure 3 An example of a process of training a student model is shown.

[0033] Figure 4 An example of a process of performing training in element units is shown.

[0034] Figure 5 An example of a process of performing training in units of sequences is shown.

[0035] Figure 6 An example of a model training method is shown.

[0036] Figure 7 An example of a data identification method is shown.

[0037] Figure 8 An example of a device for processing data based on a neural network is shown.

[0038] Throughout the drawings and detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0039] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, except for operations that must occur in a specific order, the order of operations described herein is merely an example and is not limited to the order set forth herein, but may be changed as will be clear after understanding the disclosure of the present application. In addition, for increased clarity and brevity, descriptions of features known in the art may be omitted.

[0040] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein that will become apparent upon understanding the disclosure of this application.

[0041] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Rather, these terms are used only to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, a first member, first component, first region, first layer, or first portion mentioned in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

[0042] Throughout the specification, when an element such as a layer, region, or substrate is described as being “on,” “connected to,” or “coupled to” another element, it may be directly “on,” “connected to,” or “coupled to” the other element, or one or more other elements may be present between them. Conversely, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, no other elements may be present between them.

[0043] The terms used herein are intended only to describe various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, singular terms are intended to include the plural form. The terms "include," "comprising," and "having" indicate the presence of the recited features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0044] Use of the term "may" herein with respect to an example or embodiment (eg, with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such feature, but all examples and embodiments are not limited thereto.

[0045] Figure 1 and Figure 2 Shows examples of teacher models and student models.

[0046] Figure 1 A teacher model 110 and a student model 120 are shown.

[0047] The teacher model 110 and the student model 120 are neural networks having different sizes and the same target data to be recognized. A neural network is a recognition model using a large number of artificial neurons connected by edges.

[0048] The teacher model 110 is a model that extracts sufficient features from the target data to be recognized to recognize the target data with high accuracy. The teacher model 110 is larger in size than the student model 120. For example, when compared with the student model 120, the teacher model 110 may include more layers, more nodes, or a combination thereof.

[0049] Student model 120 is a neural network with a smaller size than teacher model 110. Due to its smaller size, student model 120 uses less memory and has a faster recognition speed than teacher model 110. Student model 120 is trained so that, for specific input data, it outputs the same output data as that output by teacher model 110. For example, the output data may be probability values ​​and logical values ​​output by the corresponding model.

[0050] The input sequence X is input to the teacher model 110 and the student model 120. The input sequence X includes a plurality of elements x1, ..., x M The data may include, for example, sentence data including a plurality of words or voice data including a plurality of frames. Here, sentence data is also referred to as text data including a plurality of words, and voice data is also referred to as data obtained by sampling voice data with a certain length (for example, in units of frames). In one example, the length of a frame may be predetermined.

[0051] exist Figure 1 In the example, the probability distribution p(Y=y1...y L |X;θ T ) is output from the teacher model 110, and the probability distribution q(Y=y1...y L |X;θ S ) is output from the student model 120. Here, θ T represents the parameters of the teacher model 110, and θ S represents the parameters of the student model 120. In addition, Y is an output sequence output from each of the teacher model 110 and the student model 120, and includes a plurality of elements y1, ..., y L .

[0052] The same output data as that of the teacher model 110 can be obtained at a high recognition rate by using the student model 120 trained using the teacher model 110. This training scheme is called model compression, and a related description will be provided later.

[0053] The teacher model 110 and the student model 120 can be trained to perform desired operations by mapping input data and output data via deep learning to perform tasks such as, for example, object classification, object recognition, audio or speech recognition, and image recognition, wherein there is a nonlinear relationship between the input data and the output data. Deep learning is a machine learning method for solving problems given from large data sets. Deep learning can also be interpreted as an optimized problem-solving process for finding a point where energy is minimized while training a neural network using provided training data. Through deep learning (e.g., supervised or unsupervised learning), weights corresponding to the architecture or model of the neural network can be obtained, and the input data and output data can be mapped to each other based on the obtained weights.

[0054] In one example, the teacher model 110 and the student model 120 may be implemented as an architecture having multiple layers including an input image, a feature map, and an output. In the teacher model 110 and the student model 120, a convolution operation between the input image and a filter called a kernel is performed, and as a result of the convolution operation, a feature map is output. Here, the output feature map is the input feature map, and the convolution operation between the output feature map and the kernel is performed again, and as a result, a new feature map is output. Based on the convolution operation repeatedly performed in this way, a recognition result of the characteristics of the input image via the neural network can be output.

[0055] In another example, the teacher model 110 and the student model 120 may include an input source sentence (e.g., speech input) instead of an input image. In such an example, a convolution operation is performed on the input source sentence with a kernel, and as a result, a feature map is output. The output feature map is again convolved with the kernel as the input feature map, and a new feature map is output. When the convolution operation is repeatedly performed in this way, the recognition result of the feature of the input source sentence can be ultimately output by the neural network. The input data for the teacher model 110 and the student model 120 may include image data, speech data, and text data. However, they are provided only as examples, and other types of data will be considered to be within the scope of the present disclosure.

[0056] Figure 2 An example of using the teacher model 110 to train the student model 120 is shown.

[0057] The teacher model 110 and the student model 120 are networks of different sizes. A method and apparatus for recognizing data based on a neural network are proposed, and a method and apparatus for training a neural network are proposed. In this specification, the term "recognition" is used as a concept including verification and identification. Verification is an operation to determine whether input data is true or false. Identification is an operation to determine a label indicated by input data from a plurality of labels. For example, a neural network is a model that receives a sequence and performs an operation such as, for example, conversion, interpretation, and speech recognition.

[0058] In one example, the student model 120 and the teacher model 110 may correspond to a recurrent neural network (RNN) or a convolutional neural network (CNN). In one example, the CNN may be a deep neural network (DNN). In one example, the DNN may include a region proposal network (RPN), a classification network, a reinforcement learning network, a fully connected network (FCN), a deep convolutional network (DCN), a long short-term memory (LSTM) network, and a gated recurrent unit (GRU). The DNN may include multiple layers. The multiple layers may include an input layer, at least one hidden layer, and an output layer. In one example, in addition to the convolutional layer, the neural network may also include a subsampling layer, a pooling layer, a fully connected layer, and the like.

[0059] The neural network includes a plurality of layers, each layer including a plurality of nodes. In addition, the neural network includes connection weights connecting the plurality of nodes included in the plurality of layers to nodes included in another layer.

[0060] For example, a neural network includes an input layer, at least one hidden layer, and an output layer. The input layer receives input for training or recognition and transmits the input to the hidden layer. The output layer generates the output of the neural network based on the signals received from the hidden layer. The hidden layer is inserted between the input and output layers and changes the data transmitted by the input layer into a value that is easier to predict.

[0061] The input nodes included in the input layer and the hidden nodes included in the hidden layer are connected through edges having connection weights. The hidden nodes included in the hidden layer and the output nodes included in the output layer are connected through edges having connection weights.

[0062] A neural network may include multiple hidden layers. A neural network including multiple hidden layers is called a deep neural network. Training a deep neural network is called deep learning. For example, the teacher model 110, which is larger in size than the student model 120, may include a greater number of hidden layers.

[0063] The model training device determines the parameters of the nodes using a gradient descent scheme based on the output values ​​of the nodes included in the neural network and the losses backpropagated to the neural network. For example, the model training device updates the connection weights between the nodes by loss backpropagation learning. Loss backpropagation learning is a method in which the loss is estimated by performing forward calculations on given training data, the estimated loss is propagated in the reverse direction from the output layer toward the hidden layer and the input layer, and the connection weights are updated to reduce the loss. The processing of the neural network is performed in the direction from the input layer toward the hidden layer and the output layer. In loss backpropagation training, the update of the connection weights is performed in the direction from the output layer toward the hidden layer and the input layer. One or more processors may use a buffer memory that stores a layer or a series of calculation data for processing the neural network in the desired direction.

[0064] In one example, the model training device defines an objective function for measuring the degree to which the currently set connection weights are close to the optimal, continuously changes the connection weights based on the result of the objective function, and repeatedly performs training. For example, the objective function is a loss function for calculating the loss between the actual output value output by the neural network based on the training input of the training data and the expected expected value to be output (e.g., the training output). The model training device updates the connection weights to reduce the value of the loss function. The loss function will be described in detail as follows.

[0065] Based on knowledge distillation for knowledge propagation between two different neural networks, the student model 120 is trained from the teacher model 110. Knowledge distillation is a type of model compression. In this example, the Kullback-Leibler divergence (KLD) loss expressed by the following equation 1 is used.

[0066] [Equation 1]

[0067]

[0068] In Equation 1, H(pY|X;θ T ),q(Y|X;θ S )) represents the cross entropy between the teacher model 110 and the student model 120, H(p(Y|X;θ T )) represents the entropy of the teacher model 110. KLD-based knowledge distillation is a scheme for training the student model 120 using the probability distribution of the teacher model 110 as a soft target.

[0069] The student model 120 is trained to output the recognition result of the teacher model 110 so that the difference between the recognition result of the teacher model 110 and the recognition result of the student model 120 is reduced. Here, for example, the recognition result includes the probability distribution output from each model or the class sampled with the highest probability in the probability distribution.

[0070] Figure 3 An example of a process of training a student model is shown.

[0071] Figure 3 An example of training the student model 120 using the discriminator model 310 and the teacher model 110 is shown.

[0072] The discriminator model 310 is a neural network that distinguishes between the recognition results of the teacher model 110 and the recognition results of the student model 120, and may include, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), and self-attention. The discriminator model 310 is trained to distinguish the recognition results of the teacher model 110 as true and the recognition results of the student model 120 as false. The student model 120 is trained so that the recognition results of the teacher model 110 and the student model 120 cannot be distinguished from each other by the discriminator model 310. As such, training in which two models are trained while competing with each other is called adversarial training. Adversarial loss used in training It is expressed by the following equation 2.

[0073] [Equation 2]

[0074]

[0075] In Equation 2, d(Y|θ D ) represents a probability distribution for distinguishing whether the sequence Y input to the discriminator model 310 is generated in the teacher model 110 or the student model 120. ~p indicates that the sequence Y input to the discriminator model 310 is input from the teacher model 110, ~q indicates that the sequence Y input to the discriminator model 310 is input from the student model 120, and Express expectations.

[0076] By training the student model 120 to reduce the adversarial loss And train the discriminator model 310 to increase the adversarial loss In this way, the student model 120 is trained to output the same recognition results as the teacher model 110 to a certain extent, so that the discriminator model 310 cannot distinguish between the two.

[0077] In addition, based on the KLD loss described above The student model 120 is further trained. According to Equation 3, the student model 120 is trained to reduce the KLD loss and combat loss In addition, the discriminator model 310 is trained to increase the adversarial loss In one example, the teacher model 110 is fixed and not trained.

[0078] [Equation 3]

[0079]

[0080] The discriminator model 310 receives a sequence or an element included in a sequence from each of the teacher model 110 and the student model 120, so that training is performed in units of sequences or in units of elements included in the sequence. Figure 4 and Figure 5 The training process using the discriminator model 310 is described in detail.

[0081] Figure 4 An example of a process of training performed in element units is shown.

[0082] Reference Figure 4 , elements included in the sequences output from the teacher model 110 and the student model 120 are input to the discriminator model 310, so that training is performed in units of elements.

[0083] The j-th element in the sequence output from the teacher model 110 and the j-th element in the sequence output from the student model 120 are transmitted to the discriminator model 310. The discriminator model 310 distinguishes the models from which each of the j-th elements is input. Figure 4 In the equation, p(y1|X,Y ：1 θ T ) represents the probability distribution corresponding to the first element in the sequence output from the teacher model 110, q(y1|X,Y ：1 θ S ) represents the probability distribution corresponding to the first element in the sequence output from the student model 120, p(y L |X,Y ：L θ T ) represents the probability distribution corresponding to the Lth element in the sequence output from the teacher model 110, q(y L |X,Y ：L θ S ) represents the probability distribution corresponding to the Lth element in the sequence output from the student model 120.

[0084] In this example, the adversarial loss It is expressed by the following equation 4.

[0085] [Equation 4]

[0086]

[0087] In Equation 4, represents the class of the jth element in the sequence output from the teacher model 110, and G represents the class of the jth element in the sequence output from the student model 120. P (y j ) is to enable the student model 120 to use adversarial loss The Gumbel-max is trained, which is expressed by the following equation 5.

[0088] [Equation 5]

[0089]

[0090] g~-log(-log(uniform(0,1)))

[0091] In Equation 5, p represents the probability distribution output from the teacher model 110, and q represents the probability distribution output from the student model 120. As τ converges to 0, the maximum value of the probability distribution approaches 1 and the remaining value approaches 0. Through Gumbel-max, information about the discriminator model 310 is transmitted to the student model 120 and used for training the student model 120. In Equation 5, softmax represents the maximum soft function, and uniform represents uniform distribution.

[0092] The probability distribution of the elements output from the teacher model 110 and the probability distribution of the elements output from the student model 120 Classes sampled in and class Fighting losses when they are indistinguishable from each other This situation is expressed by the following equation 6.

[0093] [Equation 6]

[0094]

[0095] Adversarial loss for Equation 4 The gradients of the student model 120 and the discriminator model 310 are expressed by Equation 7, from which it can be known that the training is performed normally.

[0096] [Equation 7]

[0097]

[0098]

[0099] Figure 5 An example of a process of training performed in units of sequences is shown.

[0100] Reference Figure 5 , the sequences output from the teacher model 110 and the student model 120 are input to the discriminator model 310, so that training is performed in sequence units.

[0101] The sequence output from the teacher model 110 and the sequence output from the student model 120 are transmitted to the discriminator model 310. The discriminator model 310 distinguishes the models from which each of the sequences is input. Figure 5 In the equation, p(Y=y1...y L |X;θ T ) represents the probability distribution corresponding to the sequence output by the teacher model 110, q(Y=y1…y L |X;θ S ) represents the probability distribution corresponding to the sequence output by the student model 120.

[0102] In this example, the adversarial loss It is expressed by the following equation 8.

[0103] [Equation 8]

[0104]

[0105] The Gumbel-max applied to Equation 8 is determined based on a combination of probabilities of elements, or is determined based on the probability of a sequence.

[0106] The Gumbel-max based combination of element-wise probabilities is shown in Equation 9.

[0107] [Equation 9]

[0108]

[0109] As such, the Gumbel-max is determined based on the multiplication of the probabilities of the elements included in the output sequence.

[0110] The Gumbel-max based on sequence probability is shown in Equation 10.

[0111] [Equation 10]

[0112]

[0113] In Equation 10, Y' represents a plurality of candidate sequences that may correspond to the output sequence and may be, for example, the k-best among the candidate sequences. As such, by limiting the number of candidate sequences, Gumbel-max is determined based on the probability of the output sequence.

[0114] Applied to adversarial loss Z {p,q} represents the class of the sequence sampled based on the probability distribution output from the teacher model 110 or the student model 120. For example, Z {p,q} represents the class of sequences that are selected as one-best using various schemes such as, for example, beam search and gradient search. {p,q} One of the schemes is expressed by the following equation 11.

[0115] [Equation 11]

[0116]

[0117]

[0118] In Equation 11, represents the jth element in the ground truth sequence, U(0,1) represents uniform distribution, and ω represents the threshold. Onehot represents a vector representation of elements with the size of the element set to the size of the vector, assigning the value 1 to the index of the element you want to represent and 0 to the other indices.

[0119] Adversarial loss for Equation 8 The gradients of the student model 120 and the discriminator model 310 are expressed by Equation 12, from which it can be known that the training is performed normally.

[0120] [Equation 12]

[0121]

[0122]

[0123] Figure 6 An example of a model training method is shown. Although it can be performed in the order and manner shown Figure 6 However, the order of some operations may be changed or some of the operations may be omitted without departing from the spirit and scope of the illustrative examples described. Figure 6 Many of the illustrated operations may be performed in parallel or concurrently. Figure 6 One or more blocks and combinations of blocks may be implemented by a computer (such as a processor) based on dedicated hardware that performs specific functions or a combination of dedicated hardware and computer instructions. Figure 5 In addition to the description, Figures 1 to 5 The description also applies to Figure 6 , and is incorporated herein by reference. Therefore, the above description may not be repeated here.

[0124] The model training method is executed by a processor of the model training device.

[0125] In operation 610 , the model training apparatus obtains a recognition result of a teacher model and a recognition result of a student model with respect to an input sequence.

[0126] In operation 620, the model training device trains the student model so that the recognition results of the teacher model and the student model are indistinguishable from each other. In other words, the recognition results of the teacher model and the student model converge as a result of the training. The model training device determines an adversarial loss based on the degree to which the recognition results of the teacher model and the student model are distinguishable from each other, and trains the student model so that the adversarial loss is reduced.

[0127] In one example, the model training device determines the adversarial loss based on the degree to which the output sequence of the teacher model and the output sequence of the student model, which are output as recognition results of the input sequence, are distinguished from each other. For example, the model training device determines the adversarial loss by applying Gumbel-max based on the probability of elements included in the output sequence. In addition, the model training device determines the adversarial loss by applying Gumbel-max based on the probability of the output sequence.

[0128] In another example, the model training device determines the adversarial loss based on the degree to which elements included in an output sequence of a teacher model and an output sequence of a student model, each output as a recognition result of the input sequence, are distinguished from each other. For example, the model training device determines the adversarial loss by applying Gumbel-max based on the probability of the elements included in the output sequence.

[0129] The model training device trains the student model so that the recognition results of the teacher model and the recognition results of the student model cannot be distinguished from each other by the discriminator model. In this example, the discriminator model is trained to distinguish between the recognition results of the teacher model and the recognition results of the student model.

[0130] The model training device trains the student model using the recognition result of the teacher model, so that the recognition result of the teacher model is output from the student model.

[0131] Figure 7 An example of a data identification method is shown.

[0132] The data identification method is executed by a processor of the data identification device.

[0133] In operation 710, the data recognition device receives data to be recognized. In operation 720, the data recognition device recognizes the target data using a pre-trained model. In one example, the pre-trained model is the student model described above. Since the training method of the student model is the same as described above, Figures 1 to 6 The description also applies to Figure 7 , and are incorporated herein by reference. Therefore, the above description may not be repeated here and is omitted for brevity.

[0134] Figure 8 An example of a device for processing data based on a neural network is shown.

[0135] Reference Figure 8 , the data processing device 800 includes a memory 810, a processor 820, and an input / output interface 830. The memory 810 and the processor 820 communicate with each other via a bus 840.

[0136] The data processing device 800 is a device for processing input data and outputting processed data, and may be one of the model training device and the data recognition device described herein.

[0137] The memory 810 includes instructions read by a computer. The processor 820 performs the aforementioned operations in response to the instructions stored in the memory 810 being executed in the processor 820. The processor 820 can be a data processing device configured as a hardware of a circuit having a physical structure that implements the desired operation. For example, the desired operation may include a code or instruction contained in a program. For example, the data processing device configured as hardware may include a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a reconfigurable processor, a multiprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processor unit (GPU) or any other type of multi-processor or single processor configuration. Further details about the processor 820 are provided below.

[0138] The memory 810 is a volatile memory or a non-volatile memory. In addition, the data processing device 800 processes the operations described herein. Further details about the input / output interface 830 are provided below.

[0139] In one example, the input / output interface 830 may be a display that receives input from a user or provides output. In one example, the input / output interface 830 may function as an input device and receive input from a user through traditional input methods (e.g., a keyboard and mouse) and new input methods (e.g., touch input, voice input, and image input). Therefore, the input / output interface 830 may include, for example, a keyboard, a mouse, a touch screen, a microphone, and other devices that can detect input from a user and send the detected input to the data processing device 800.

[0140] In one example, the input / output interface 830 may function as an output device and provide the output of the data processing device 800 to the user through visual, auditory, or tactile channels. The input / output interface 830 may include, for example, a display, a touch screen, a speaker, a vibration generator, and other devices that can provide output to the user.

[0141] However, the input / output interface 830 is not limited to the examples described above, and any other display (e.g., a computer monitor and an eyewear display (EGD) operably connected to the data processing device 800) may be used without departing from the spirit and scope of the illustrative examples described. In one example, the input / output interface 830 is a physical structure that includes one or more hardware components that provide the capability to render a user interface, render a display, and / or receive user input.

[0142] The data processing device 800 may be implemented in various electronic devices such as, for example, a mobile phone, a smart phone, a wearable smart device (such as a ring, a watch, a pair of glasses, a glasses-like device, a bracelet, an anklet, a belt, a necklace, an earring, a hairband, a helmet, a device embedded in clothing, or an eyeglass display (EGD)), a computing device (such as a server, a laptop computer, a notebook computer, a small notebook computer, a netbook, an ultra mobile PC (UMPC), a tablet personal computer (tablet), a tablet phone, a mobile internet device (MID), a personal digital assistant (PAD), an enterprise digital assistant (EDA), an ultra mobile personal computer (UMPC), a portable laptop PC), an electronic Sub-products (e.g., robots, digital cameras, digital video cameras, portable game consoles, MP3 players, portable / personal multimedia players (PMPs), handheld e-books, global positioning system (GPS) navigators, personal navigation devices, portable navigation devices (PNDs), head-up displays (HUDs), handheld game consoles, e-books, televisions (TVs), high-definition televisions (HDTVs), smart TVs, smart home appliances, smart home devices or security devices for door control, various Internet of Things (IoT) devices, autonomous vehicles, automatic or autonomous driving systems, smart vehicles, advanced driver assistance systems (ADAS)), or any other device capable of wireless or network communication consistent with the devices disclosed herein. In one example, the data processing device 800 uses a pre-trained model to identify target data. In one example, the pre-trained model is the student model described above.

[0143] In one example, the data processing apparatus 800 may be connected to an external device via an input / output device of the external device (eg, a personal computer (PC) or a network) to exchange data with the external device.

[0144] The equipment described herein, unit, module, device and other components are realized by hardware components. Examples of hardware components that can be used for performing the operation described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators and any other electronic components configured to perform the operation described in this application. In other examples, one or more of the hardware components for performing the operation described in this application are realized by computing hardware (e.g., by one or more processors or computers). Processors or computers can be realized by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to achieve desired results). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operation described in this application. Hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor or two or more processors or a processor and a controller. One or more hardware components can be implemented by one or more processors or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors or other processors and other controllers. One or more processors or a processor and a controller can implement a single hardware component or two or more hardware components. A hardware component can have any one or more of different processing configurations, wherein the example of a processing configuration includes: a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, a multiple instruction single data (MISD) multiprocessing and a multiple instruction multiple data (MIMD) multiprocessing.

[0145] The method for performing the operations described in this application is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as described above as execution instructions or software to perform the operations performed by the method described in this application. For example, a single operation or two or more operations can be performed by a single processor or two or more processors or a processor and a controller. One or more operations can be performed by one or more processors or a processor and a controller, and one or more other operations can be performed by one or more other processors or other processors and other controllers. One or more processors or a processor and a controller can perform a single operation or two or more operations.

[0146] The instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure a processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include at least one of a small program, a dynamic link library (DLL), middleware, firmware, a device driver, and an application program that stores a method based on a neural network training model or a method for training a model. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by a processor or computer. In another example, the instructions or software include high-level code that is executed by a processor or computer using an interpreter. Programmers of ordinary skill in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0147] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RW, , BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card-type storage (such as, MultiMediaCard, Secure Digital (SD) card or Extreme Digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to: store instructions or software and any associated data, data files and data structures in a non-transitory manner, and provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0148] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered illustrative only and not for purposes of limitation. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. Appropriate results can be achieved if the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the present disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the present disclosure.

Claims

1. A method for training a model, the method comprising: For the input sequence, obtain the recognition results of the teacher model and the recognition results of the student model; and Train the student model so that the recognition results of the teacher model and the student model are indistinguishable from each other. The steps of training the student model include: Based on the degree to which the recognition results of the teacher model and the recognition results of the student model are distinguished from each other, an adversarial loss is determined, and the student model is trained to reduce the adversarial loss, The step of determining the adversarial loss includes: determining the adversarial loss based on the degree to which the output sequence of the teacher model and the output sequence of the student model, which are respectively output as recognition results for the input sequence, are distinguished from each other, The steps for determining the adversarial loss include: The adversarial loss is determined by applying Gumbel-max based on the probability of the elements included in the output sequence, or by applying Gumbel-max based on the probability of the output sequence, The input sequence includes: sentence data containing words and / or speech data including frames.

2. The method according to claim 1, wherein The steps to determine adversarial losses include: The adversarial loss is determined by further applying Gumbel-max based on the probabilities of multiple candidate sequences that may correspond to the output sequence.

3. The method according to claim 1, wherein The steps to determine adversarial losses include: The adversarial loss is determined based on the extent to which elements in an output sequence of the teacher model and elements in an output sequence of the student model, which are respectively output as recognition results for the input sequence, are distinguished from each other.

4. The method according to claim 1, wherein The steps to train the student model include: Train the student model so that the recognition results of the teacher model and the student model cannot be distinguished from each other by the discriminator model, and The discriminator model is trained to distinguish between the recognition results of the teacher model and the recognition results of the student model.

5. The method according to claim 1, wherein The steps to train the student model include: The student model is trained using the recognition results of the teacher model so that the recognition results of the teacher model are output from the student model.

6. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method of claim 1.

7. A device for training a model, the device comprising: The processor is configured to: obtain, for an input sequence, a recognition result of a teacher model and a recognition result of a student model; And train the student model so that the recognition results of the teacher model and the student model are indistinguishable from each other, The processor is further configured to: determine an adversarial loss based on the degree to which the recognition results of the teacher model and the recognition results of the student model are distinguished from each other, and train the student model to reduce the adversarial loss, The processor is further configured to determine the adversarial loss based on the degree to which the output sequence of the teacher model and the output sequence of the student model, which are respectively output as recognition results for the input sequence, are distinguished from each other, The processor is further configured to determine the adversarial loss by applying Gumbel-max based on the probability of elements included in the output sequence, or by applying Gumbel-max based on the probability of the output sequence, The input sequence includes: sentence data containing words and / or speech data including frames.

8. The apparatus according to claim 7, wherein The processor is further configured to determine an adversarial loss based on the extent to which elements in an output sequence of the teacher model and an output sequence of the student model, which are respectively output as recognition results for the input sequence, are distinguished from each other.

9. The apparatus according to claim 7, wherein The processor is further configured to: train the student model so that the recognition results of the teacher model and the recognition results of the student model cannot be distinguished from each other by the discriminator model, and The discriminator model is trained to distinguish between the recognition results of the teacher model and the recognition results of the student model.

10. The apparatus according to claim 7, wherein The processor is further configured to: train the student model using the recognition result of the teacher model so that the recognition result of the teacher model is output from the student model.

11. The apparatus according to claim 7, further comprising: The memory is configured to store parameters of the student model and store instructions that, when executed, configure the processor to obtain recognition results of the teacher model and the student model, and train the student model.

12. A device for training a model, the device comprising: The processor is configured to: For the input sequence, obtain the recognition results of the teacher model and the recognition results of the student model; and The student model is trained to minimize the difference between the teacher model’s recognition results and the student model’s recognition results in the discriminator model. The processor is further configured to: determine an adversarial loss based on the degree to which the recognition results of the teacher model and the recognition results of the student model are distinguished from each other, and train the student model to reduce the adversarial loss, The processor is further configured to determine the adversarial loss based on the degree to which the output sequence of the teacher model and the output sequence of the student model, which are respectively output as recognition results for the input sequence, are distinguished from each other, The processor is further configured to determine the adversarial loss by applying Gumbel-max based on the probability of elements included in the output sequence, or by applying Gumbel-max based on the probability of the output sequence, The input sequence includes: sentence data containing words and / or speech data including frames.

Images