Streaming sequence transducer learning device using knowledge distillation and learning control method thereof

Abstract

The present invention relates to a technique for training an automatic speech recognition model using a knowledge distillation learning method. More specifically, the present invention relates to a knowledge distillation learning method in which, when a first and second model receive training data, the first model outputs a first probability distribution based on the input training data, the second model outputs a second probability distribution based on the input training data, and the second model is trained based on the first and second probability distributions.

Description

Technology Field

[0001] The present invention relates to a method for learning a streaming sequence converter, and more specifically, to a technique for learning a fast and high-accuracy streaming sequence converter through knowledge distillation based on an offline model. Background Technology

[0003] Deep neural networks approximate complex functions with a large number of learnable parameters, and the number of these parameters determines the network's performance.

[0004] Methods are being researched to achieve high performance with a relatively small number of parameters, among which the knowledge distillation method is proposed as a representative example. Learning methods utilizing this knowledge distillation learn by transferring knowledge from a pre-trained teacher model to a student model.

[0005] Streaming Automatic Speech Recognition (Streaming ASR) has the core objective of outputting each assumed word as quickly and accurately as possible. However, as measured by the word error rate (WER), outputting quickly without quality degradation is a very difficult task. Sequence transducers exist as models for such streaming automatic speech recognition. Sequence transducers have been the most popular streaming speech recognition models in recent years.

[0006] As mentioned above, in streaming scenarios, both accuracy and latency are important, but these two factors are in a trade-off relationship. For example, increasing accuracy typically leads to increased latency, while reducing latency generally results in lower accuracy.

[0007] Accordingly, there is a need for research on sequence converters that can improve both accuracy and latency in streaming scenarios. The problem to be solved

[0009] The problem that the present invention aims to solve is to provide a sequence converter that can increase accuracy while simultaneously reducing delay time.

[0010] Another problem that the present invention aims to solve is to provide a method for producing a high-performance streaming model by utilizing a knowledge distillation learning technique.

[0011] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0013] According to one aspect of the present invention to solve the above or other problems, a knowledge distillation learning method is provided, comprising the steps of: receiving training data for a first and a second model; outputting a first probability distribution for the first model based on the input training data; outputting a second probability distribution for the second model based on the input training data; and training the second model based on the first and second probability distributions.

[0014] The above first model and second model may be end-to-end streaming automatic speech recognition models.

[0015] The above end-to-end streaming automatic speech recognition model may be a sequence converter model.

[0016] The first model above is an offline model for automatic speech recognition, and the second model above may be a streaming model for automatic speech recognition.

[0017] It may further include a step of performing alignment between the first model and the second model.

[0018] The step of performing the alignment above can be performed by controlling the delay time of at least one of the first model and the second model.

[0019] The control of the above delay time can be achieved by adjusting the issuance probability of the label token.

[0020] The control of the above delay time can adjust the issuance probability of the label token while maintaining the issuance probability of the blank token as is.

[0022] According to another aspect of the present invention to solve the above or other problems, a knowledge distillation learning device comprises: a memory for storing first and second models and instructions; and a processor configured to execute the stored instructions, wherein the processor comprises: a first model that outputs a first probability distribution based on input training data; a second model that outputs a second probability distribution based on the input training data; and a learning unit that trains the second model based on the first and second probability distributions.

[0023] The above first model and second model may be end-to-end streaming automatic speech recognition models.

[0024] The above end-to-end streaming automatic speech recognition model may be a sequence converter model.

[0025] The first model above is an offline model for automatic speech recognition, and the second model above may be a streaming model for automatic speech recognition.

[0026] It may further include a model alignment unit that performs alignment between the first model and the second model.

[0027] The above model alignment unit can perform the alignment by controlling the delay time of at least one of the first model and the second model.

[0028] The above model alignment unit may be implemented by controlling the delay time in a manner that adjusts the issuance probability of the label token.

[0029] In controlling the delay time, the above model alignment unit can adjust the issuance probability of the label token while maintaining the issuance probability of the blank token as is. Effects of the invention

[0031] The effects of the streaming sequence converter according to the present invention are described as follows.

[0032] According to at least one of the embodiments of the present invention, there is an advantage in that a sequence converter can be provided that can increase accuracy while simultaneously reducing delay time.

[0033] In addition, according to at least one of the embodiments of the present invention, there is an advantage in that a method for producing a high-performance streaming model by utilizing a knowledge distillation learning technique can be provided.

[0034] Further scopes of the applicability of the present invention will become apparent from the following detailed description. However, since various changes and modifications within the spirit and scope of the present invention are clearly understood by those skilled in the art, specific embodiments, such as the detailed description and preferred embodiments of the present invention, should be understood as being given merely as examples. Brief explanation of the drawing

[0036] FIG. 1 illustrates a conceptual diagram of a knowledge distillation learning method of a knowledge distillation learning device (10) according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a conceptual diagram of a sequence converter (200) according to an embodiment of the present invention. FIG. 3 is a diagram illustrating alignment according to the "FastEmit" hyperparameter λ of a streaming model according to an embodiment of the present invention. FIG. 4 is a diagram illustrating alignment according to the "FastEmit" hyperparameter λ in an offline model according to an embodiment of the present invention. Figure 5 is a diagram comparing the alignment between the offline model and the streaming reference model. Figure 6 is a diagram illustrating the alignment interval between the offline model and the "FastEmit" applied streaming model. FIG. 7 is a diagram illustrating the control flowchart of a knowledge distillation learning device (10) according to an embodiment of the present invention. FIG. 8 is a diagram illustrating a delay time control flowchart for knowledge distillation learning according to an embodiment of the present invention. FIG. 9 shows a table of experimental results for offline and streaming models for the hyperparameter λ of "FastEmit" according to an embodiment of the present invention. FIG. 10 shows a table comparing the results of WER and delay time for the distillation loss type and the teacher model of S1 according to an embodiment of the present invention. FIG. 11 is a diagram illustrating the result of the alignment match between an offline model and a streaming model according to an embodiment of the present invention. Figure 12 illustrates the experimental results showing the effect of knowledge distillation weight (KD weight) on performance. Figure 13 illustrates experimental results comparing performance for distillation loss type. FIG. 14 is a diagram illustrating the streaming baseline, "FastEmit", and the WER (%) and delay time (ms) of the method according to an embodiment of the present invention. FIG. 15 is a diagram illustrating the configuration of a knowledge distillation learning device (100) according to one embodiment. Specific details for implementing the invention

[0037] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0038] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0039] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0040] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0041] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] Streaming-type automatic speech recognition must output each assumed word as quickly and accurately as possible. To this end, end-to-end (E2E) streaming automatic speech recognition (ASR) models have recently been gaining attention.

[0044] These streaming automatic speech recognition models can enhance the usefulness of ASR because they perform decoding simultaneously with the streaming audio input to provide a transcript on the fly. Sequence converter models are representative end-to-end models suitable for such streaming ASR.

[0045] Accordingly, various neural network-based sequence converter models have been proposed. These sequence converter models operate efficiently as batch or frame-by-frame models depending on the encoder structure.

[0047] In streaming scenarios for providing ASR results in real time, both model accuracy and latency become important.

[0048] Therefore, in one embodiment of the present invention, a knowledge distillation learning technique is proposed to improve both the accuracy and latency of the model. That is, it is proposed to reduce latency by using a small number of parameters while obtaining high accuracy by receiving knowledge from a model with a large number of parameters.

[0049] In one embodiment of the present invention, partial latency may refer to the difference between the end point of a voice part (speaker's utterance) (not the end point of the audio signal) and the point where the full text script appears when a voice audio signal is received.

[0051] In the following detailed description of the present invention, a model with a high number of parameters is referred to as a "teacher model," and a model with a low number of parameters is referred to as a "student model."

[0052] According to a knowledge distillation learning method according to one embodiment of the present invention, the same voice is input to a teacher model and a learning model, and the student model's prediction distribution can be learned to be as similar as possible to the teacher model's prediction distribution.

[0054] In one embodiment of the present invention, the teacher model may be an offline model, and the student model may be a streaming model.

[0055] Offline models predict output tokens using full contextual information, whereas streaming models predict output tokens using only past and present contextual information. Generally, high-latency streaming models are more accurate because they utilize additional prediction information, while low-latency streaming models operate faster but at the cost of lower accuracy. Therefore, to implement a highly usable streaming model, it is necessary to appropriately balance accuracy and latency.

[0057] FIG. 1 illustrates a conceptual diagram of a knowledge distillation learning method of a knowledge distillation learning device (10) according to an embodiment of the present invention.

[0058] Knowledge Distillation (KD) is a learning method that maximizes the performance of a student model by utilizing a high-performing teacher model.

[0059] In particular, as will be explained in more detail below, the knowledge distillation learning method in the sequence converter model of one embodiment of the present invention generally aims to minimize the difference between the frame-level output probability distributions for the teacher model and the student model. Here, the frame-level output probability distribution refers to the probability distribution when voice input is input in frame units.

[0061] In one embodiment of the present invention, to address the performance degradation of a streaming model that is relatively inferior to an offline model in a streaming scenario for providing ASR results in real time, a learning method is proposed through knowledge distillation between an offline model acting as a teacher model and a streaming model acting as a student model.

[0063] Deep neural networks approximate complex functions with a large number of learnable parameters, and the number of these parameters determines the network's performance.

[0064] Learning methods utilizing knowledge distillation learn by transferring knowledge from a pre-trained teacher model to a student model. Specifically, this knowledge transfer is achieved by training the student model against the output probability distribution (soft labels) of the teacher model.

[0065] More specifically, knowledge distillation in the sequence transducer model is performed by minimizing the Kullback-Leibler (KL) divergence between the posterior distributions of the offline and streaming models at each node (t, u). The specific learning method is explained with reference to the figure below.

[0067] A knowledge distillation learning system (10) according to one embodiment of the present invention may be configured to include a teacher model (102), a student model (103), a KL divergence calculation unit (104), a learning unit (106), and a model alignment unit (107).

[0068] The components illustrated in FIG. 1 are not essential for implementing the knowledge distillation learning system (10), so the knowledge distillation learning system (10) described herein may have more or fewer components than the components listed above.

[0070] The teacher model (102) may be a photo-trained sequence converter model for transferring knowledge to the student model (103). The teacher model (102) may have a greater number of parameters than the student model (103) and may be a model capable of performing automatic speech recognition with high accuracy on its own.

[0071] When a voice input (101) is received, the teacher model (102) outputs a first probability distribution (105-1) as a voice recognition result for the voice input (101).

[0073] The student model (103) may be a sequence converter model that performs learning to receive knowledge from the teacher model (102). The knowledge distillation learning illustrated in FIG. 1 may be learning targeting the student model (103).

[0074] When a voice input (101) is received, the student model (103) outputs a second probability distribution (105-2) as a voice recognition result for the voice input (101).

[0076] The KL divergence calculation unit (104) calculates the KL divergence value between the first probability distribution (105-1), which is the output of the teacher model (102), and the second probability distribution (105-2), which is the output of the student model (103), when the same voice input (101) is input to the teacher model (102) and the student model (103), respectively.

[0078] The learning unit (106) performs learning by adjusting the parameters of the student model (103). At this time, the parameters of the student model (103) can be adjusted in a direction that minimizes the KL divergence value calculated by the KL divergence calculation unit (104).

[0080] The model alignment unit (107) is configured to align two models to reduce the alignment gap between the teacher model (102) and the student model (103). More specifically, the model alignment unit (107) according to one embodiment of the present invention can control the latency of the teacher model (102) and the student model (103). This is to reduce the alignment gap that occurs due to the latency difference between the teacher model and the student model.

[0082] The teacher model (102) and student model (103) according to one embodiment of the present invention may be composed of a sequence transducer. The sequence transducer is a representative end-to-end (E2E) streaming automatic speech recognition (ASR) model, which has been the most popular streaming speech recognition model in recent years.

[0084] First, the same voice input (101) is provided to the teacher model (102) and the student model (103). The voice input (101) provided as input may be provided in fixed frame units as described above, but is not necessarily limited thereto.

[0086] The voice input (101) provided according to an embodiment of the present invention may be training data for supervised learning. Training data refers to data in which an input set is paired with a specific output (correct label) for learning purposes. Then, the algorithm to be learned can search for patterns within the input set and gradually learn the correlation with the output.

[0087] The teacher model (102) of the knowledge distillation learning system (10) according to one embodiment of the present invention may be a pre-trained model. That is, in order to properly transfer knowledge to the student model (103), it must be a model that already possesses high accuracy.

[0089] In addition, the parameters of the teacher model (102) according to one embodiment of the present invention may be greater than the parameters of the student model (103). This is because the objective of the present invention is for the student model (103) having low parameters to have high accuracy and short latency.

[0090] In one embodiment of the present invention, the teacher model (102) may be an offline model, and the student model (103) may be a streaming model.

[0092] The output of the teacher model (102) that receives the voice input (101) is the first probability distribution (105-1), and the output of the student model (103) that receives the voice input (101) is the second probability distribution (105-2).

[0094] At this time, the parameters for the student model (103) may be adjusted by the learning unit (106) in a direction in which the first and second probability distributions (105-1, 105-2) become similar to each other. In particular, when the voice input (101) is composed of training data, the parameters of the student model (103) may be adjusted in a direction in which the correct label (106) paired with the input and the second probability distribution (105-2) become closer (matching).

[0096] As described above, the first and second probability distributions (105-1, 105-2) may be in the form of posterior distributions. And, the direction in which the first and second speech recognition results (105-1, 105-2) become closer to each other may be such that the Kullback-Leibler (KL) divergence between the first and second speech recognition results (105-1, 105-2) is minimized.

[0098] When parameter adjustments are made to the student model (103) in this way, the knowledge possessed by the teacher model (102), which has already been trained to have high accuracy, can be transferred to the student model (103).

[0100] As seen in Figure 1, the knowledge distillation learning method in the sequence converter model generally aims to minimize the difference between the frame-level output probability distributions for the teacher model and the student model (105-1 and 105-2 in Figure 1). However, there is a problem in that the alignment between the offline model (teacher model) and the streaming model (student model) is generally not well matched because the latency is different.

[0101] In other words, the knowledge distillation learning method in sequence converter models has a problem in that the teacher model must be fine-tuned to the student model to match the alignment gap of the latency between the teacher model and the student model.

[0103] Ultimately, it was recognized that when training is performed using the knowledge distillation method as shown in Figure 1, the larger the latency gap between the offline model and the streaming model, the greater the performance degradation.

[0105] The following detailed description of the present invention explains that if the delay time interval between the offline model, which is the teacher model, and the streaming model, which is the student model, is small, training to improve accuracy for the offline model can simultaneously reduce the delay time of the streaming model.

[0106] Furthermore, the detailed description of the present invention shows that in the knowledge distillation learning process, performance degradation occurs in the student model when frame-by-frame alignment is not well matched between the offline model, which is the teacher model, and the streaming model, which is the student model.

[0108] Additionally, in the following embodiment of the present invention, an effective knowledge distillation method is proposed for performing matching alignments between an offline model, which is a teacher model, and a streaming model, which is a student model.

[0110] First, the structure of the sequence converter will be explained with reference to Figure 2.

[0111] FIG. 2 is a diagram illustrating a conceptual diagram of a sequence converter (200) according to an embodiment of the present invention.

[0112] A sequence converter (200) according to one embodiment of the present invention may be configured to include an encoder (201), a prediction network (202, predict network), and a joint network (203, joint network).

[0113] The components illustrated in FIG. 2 are not essential for implementing the sequence converter (200), so the sequence converter (200) described herein may have more or fewer components than those listed above.

[0114] The encoder (201) is the voice input (101) Receive speech representation vector Prints. Here represents the input vector at time t.

[0115] Next, the prediction network (202) labels the sequence Receives as input, and is text representation vectors Prints.

[0116] The joint network (203) is a speech representation vector and It takes as input and outputs the most probable label token or a blank token. The output token(s) represent the lattice alignment a. To learn the probabilistic alignment, the vocabulary is a whitespace token including It is expanded to. This can be expressed mathematically as follows.

[0118]

[0119] At this time, the loss of the sequence converter (200) is aligned as in the following mathematical formula 2. It is defined as the sum of all possible paths of the conditional distribution.

[0121]

[0122] At this time is a function that removes empty tokens from each sorted a, and is the true label sequence. The goal of the sequence converter (200) loss is to maximize the log probability of the conditional distribution of Equation 3.

[0124]

[0126] The loss of the sequence converter (200) described above attempts to maximize the log probability of all possible alignments, so label tokens and empty tokens are treated identically. And in this case, the streaming sequence converter model is insufficient in information because it calculates the emission probability using only causal context information.

[0127] As a result, there is a problem where the emission of label tokens is delayed because the model continues to generate empty tokens until sufficient information is collected.

[0129] To solve such problems, one embodiment of the present invention proposes adjusting the delay time of at least one of the teacher model (102) and the student model (103).

[0130] In particular, it is proposed that delay time control according to one embodiment of the present invention be performed based on "FastEmit".

[0132] "FastEmit" is an algorithm that increases the gradient of label output by adjusting the emission of label tokens to be faster. It is a latency control method proposed in the paper "FASTEMIT: LOW-LATENCY STREAMING ASR WITH SEQUENCE-LEVEL EMISSION REGULARIZATION," published by Google LLC on February 3, 2021. To explain it more simply, "FastEmit" is a method to reduce the latency of streaming models through emission regularization.

[0133] Emission normalization refers to the process of calculating gradients for label tokens and blank tokens during the gradient calculation for the transducer's loss function; in order to apply a stronger gradient that emits labels, in Equation 4 below It refers to a normalization method that increases the probability of occurrence by that amount.

[0134] "FastEmit" proposes a learning method that controls the trade-off relationship between accuracy and latency by adjusting the issuance probability of label tokens.

[0135] The slope of the label token in "FastEmit" ( ) is modified as shown in the following mathematical formula 4.

[0137]

[0138] is the loss of the sequence converter (200) to which "FastEmit" is applied, and is the loss of the sequence converter (200) in outputting the label token It is a hyperparameter for boosting.

[0139] In the mathematical equation 4 above, the equation on the left is for the label token, and since the lambda value is added and x >= 1, the equation for calculating the gradient for the blank token on the right is maintained so that the emission probability for the label token can be increased.

[0140] In conclusion, referring to Equation 4, it can be seen that the gradient for blank tokens remains unchanged, while only the output of label tokens is accelerated.

[0142] Returning to Figure 1, we will provide further explanation regarding the remaining components.

[0144] As described above, the first and second probability distributions (105-1, 105-2), which are the outputs of the teacher model (102) and the student model (103) respectively, may be in the form of posterior distributions. And, the direction in which the first and second speech recognition results (105-1, 105-2) become closer to each other may be such that the Kullback-Leibler (KL) divergence between the first and second speech recognition results (105-1, 105-2) is minimized.

[0146] Knowledge distillation in the sequence converter model is performed in a direction that minimizes the Kullback-Leibler (KL) divergence between the posterior distributions of the offline model and the streaming model at each node (t, u). To this end, the KL divergence calculation unit (104) calculates the KL divergence value between the first probability distribution (105-1) and the second probability distribution (105-2) as shown in Equation 5 below.

[0148]

[0149] The loss function in Equation 5 above minimizes the posterior distance between the teacher model and the student model.

[0150] Another method uses the ℓ2-norm as a distance measure and can calculate the KD loss that minimizes the distance between the unnormalized model output (logit) values ​​as shown in Equation 6 below.

[0152]

[0153] Here, T is the teacher model (102), S is the student model (103), and h is the output of the joint network (203).

[0155] Performance degradation occurs during the knowledge distillation learning process due to the difference in latency between the teacher model (102) and the student model (103). In an embodiment of the present invention described below, a method is proposed to improve the performance of the student model (103), which is a streaming model, by minimizing the latency interval.

[0157] In streaming scenarios, it is common to use neural networks with causal structures, such as sequence converter encoders. Representative examples of causal structured neural networks include unidirectional recurrent neural networks and causal conformers. These models tend to delay the emission of output tokens until sufficient information is accumulated. This delay causes a time lag between the offline model and the streaming model, which hinders stable knowledge transfer during knowledge distillation learning.

[0159] In one embodiment of the present invention, a contextual block-wise conformer model is used to mitigate this problem.

[0160] According to one embodiment of the present invention, a frame unit refers to a unit of voice signal having a preset window size (e.g., 25ms unit), and a block unit may refer to a set of a predetermined number of frames.

[0161] The block-by-block conformer model processes the voice input (101) sequence block by block to prevent excessive delay when the model emits label tokens.

[0162] Furthermore, since block-level information processing allows for the consideration of some future context when inferring the current time step, streaming models can mimic the output of offline models with relatively more information. This is because existing streaming models predict output tokens using only past and present context information.

[0164] "FastEmit" is a method to reduce the latency of streaming models through emission normalization.

[0165] FIG. 3 is a diagram illustrating alignment according to the "FastEmit" hyperparameter λ of a streaming model according to an embodiment of the present invention.

[0166] FIG. 4 is a diagram illustrating alignment according to the "FastEmit" hyperparameter λ in an offline model according to an embodiment of the present invention.

[0167] Referring to Figure 3, it can be seen that performance degradation occurs because the streaming model is forced to predict labels that are not sufficiently informative.

[0168] On the other hand, referring to Figure 4, since the offline model predicts the label with the entire context, it can be seen that encouraging the release of label tokens in the offline model can significantly reduce latency by releasing label tokens faster.

[0170] Accordingly, one embodiment of the present invention proposes applying "FastEmit" to an offline model to obtain a teacher model that is better and faster than the student model. That is, "FastEmit" is applied to reduce the latency of the offline model. According to one embodiment of the present invention, a streaming model can be trained with higher accuracy and lower latency.

[0172] Figure 5 is a diagram comparing the alignment between the offline model and the streaming reference model. Figure 6 is a diagram illustrating the alignment interval between the offline model and the "FastEmit" applied streaming model.

[0173] The latency of the streaming block unit conformer is similar to the latency of the offline conformer in Fig. 5. However, when "FastEmit" is applied to reduce the latency of the streaming model, it can be seen that the latency gap between the streaming model and the offline model widens as shown in Fig. 6.

[0175] Meanwhile, in one embodiment of the present invention, it is proposed to reduce the delay time interval by using "FastEmit" simultaneously in the offline model and the streaming model. That is, a knowledge distillation learning method from the teacher model (102) to the student model (103) is proposed in a way that minimizes the delay time interval between the two models.

[0176] The proposed method can not only improve model accuracy by reducing the alignment gap between the two models but also reduce the latency of the streaming model. The final training loss is calculated as a linear combination of the sequence converter loss and the knowledge distillation loss, as shown in Equation 7 below.

[0178]

[0179] is the loss of the sequence converter (200) to which FastEmit is applied (calculated by Equations 1 to 3), and α is the knowledge distillation loss weight. Is (Mathematical Formula 5) or (Mathematical Formula 6)

[0181] FIG. 7 is a diagram illustrating the control flowchart of a knowledge distillation learning device (10) according to an embodiment of the present invention.

[0183] First, the knowledge distillation learning device (10) pre-trains the teacher model (102) (S701). Then, the same voice input (101) is input as the nth training data (S702) to each of the teacher model (102) and the student model (103).

[0184] Each of the teacher model (102) and the student model (103) can be composed of a sequence converter (200).

[0185] In one embodiment of the present invention, the teacher model may be an offline model, and the student model may be a streaming model.

[0187] As seen in the structure of the sequence converter (200) of FIG. 2 described above, in addition to the voice input (101) of the nth training data, in step S702, the label sequence corresponding to the correct label (106) It can be input together into the teacher model (102) and the student model (103), respectively.

[0189] The teacher model (102) can output a first probability distribution (105-1) (S703) based on the received voice input (101) and the correct answer label (106). Then, the student model (103) can output a second probability distribution (105-2) (S704) based on the received voice input (101) and the correct answer label (106).

[0190] In the illustrated drawings, steps S703 and S704 are in a fixed order, but embodiments of the present invention will not be limited to this order.

[0192] Next, the learning unit (106) can calculate the loss (S705) and adjust the parameters (S706).

[0193] The loss according to one embodiment of the present invention can be calculated by Equation 7 for calculating the final training loss described above.

[0194] Mathematical Formula 7 Loss In order to calculate KL divergence, the KL divergence calculation unit (104) can calculate KL divergence based on the first and second probability distributions (105-1, 105-2) as in Equation 5.

[0195] is the loss value for knowledge distillation.

[0196] The learning unit (106) can perform parameter adjustment (S706) to minimize the final training loss or maximize the log probability.

[0198] In step S707, the learning unit (106) determines whether sufficient learning has been performed. If it is not sufficient, the process proceeds to step S708, increases the value of n by one, and then steps S702 through S706 can be repeated.

[0199] If it is determined that learning is sufficient, the learning unit (106) can terminate the knowledge distillation learning method.

[0201] FIG. 8 is a diagram illustrating a delay time control flowchart for knowledge distillation learning according to an embodiment of the present invention.

[0202] Step S701 is the same as the step of Fig. 7 described above.

[0203] Next, in step S801, the model alignment unit (107) controls the delay time of the teacher model (102).

[0204] And in step S802, the model alignment unit (107) controls the delay time of the student model (103).

[0205] In the illustrated drawings, steps S801 and S802 are in a fixed order, but embodiments of the present invention will not be limited to this order.

[0206] Next, the model alignment unit (107) performs alignment to reduce the alignment gap between the teacher model and the student model (performs knowledge distillation learning in step S700).

[0207] The knowledge distillation learning of step S700 refers to the iterative learning of S702 to S708 in Fig. 7.

[0208] That is, alignment can be achieved through a process of efficiently optimizing the parameters of the student model by inducing them to generate an optimal approximate distribution using the teacher model having multiple high-dimensional parameters described in Figure 7.

[0211] Conducting experiments to prove performance

[0212] Experiments to verify the performance of the embodiments of the present invention used the public "LibriSpeech" dataset, which consists of 960 hours of English audiobook data with correct answer labels (training data). As input, 80-channel filter bank functions calculated in a 25ms window with a stride of 10ms were extracted, and "SpecAugment" was applied for data augmentation.

[0213] Next, a Conformer Transformer (Conformer-T) was used on an offline model consisting of 12 layers, 512 DIMs, and 8 attention head conformer-based encoders, 1 layer, 512 LSTM (Long Short-Term Memory) based prediction networks, and 640 DIMs.

[0214] The context-block-specific conformer transformer used in the streaming model is a collaborative network. It consists of 12 layers, 256 dims, 4 attention heads, 40 block sizes, 16 hop sizes, and 16 preview context encoders.

[0215] In both models, the prediction network and the co-network were designed identically to the offline model to accurately measure the encoder latency.

[0216] The knowledge distillation loss weight was set to α = 0.3, and all models were trained to predict a wordpiece model of size 1,024. The Adam optimizer was used with a warm-up learning rate scheduler with a 25k warm-up phase. Training was performed using the ESPnet tool.

[0217] Accuracy and latency were measured using Word Error Rate (WER) and Partial Latency (PR). Here, PR represents the difference between the time the last token was emitted and the time the user finished speaking. The 50th and 90th percentile values ​​both represent the percentage of all utterances in the test set. Speech latency means that the model generated a hypothesis (predicted result) before the speaker spoke.

[0219] FIG. 9 shows a table of experimental results for offline and streaming models for the hyperparameter λ of "FastEmit" according to an embodiment of the present invention.

[0220] During the experiment, we estimated the performance of the offline conformer converter and the streaming context block-unit conformer converter baseline models. The results showed that the offline baseline O1 was 1.0% smaller in WER than the streaming baseline S1, and the latency was 20ms slower at PR50 and PR90. As mentioned above, the block-unit structure showed almost no difference in latency between the offline and streaming models.

[0221] Looking at the results in Figure 9, it can be seen that for both the offline and streaming models, latency decreases and WER increases as the hyperparameter λ of "FastEmit" increases. As previously mentioned, it can be seen that the offline model with "FastEmit" applied shows a significant decrease in latency as regulation becomes stricter. Among them, O2, O3, and O4 had WERs of 3.0 / 7.3%, 3.0 / 7.3%, and 3.3 / 8.0% (Clear / Other), and latency was 50ms, 130ms, and 490ms, which is lower than S1.

[0223] FIG. 10 shows a table comparing the results of WER and delay time for the distillation loss type and the teacher model of S1 according to an embodiment of the present invention.

[0224] Looking at the experimental results in Figure 10, we can see how the student model is affected by the accuracy and latency of the teacher model. As a result, there was almost no change in performance when the KD loss was the MSE loss. On the other hand, in the case of the KL loss, performance improved in O1, O2, and O3 when the latency gap between the teacher model and the student model was small. Additionally, it can be seen that performance deteriorated in O4 and O5 as the latency gap increased again.

[0226] FIG. 11 is a diagram illustrating the result of the alignment match between an offline model and a streaming model according to an embodiment of the present invention.

[0227] Referring to Figure 11, it illustrates how alignment matching affects performance. Although C1 and C2 were trained on the same teacher model, C2 showed degraded performance due to a larger alignment gap with O2. C4 was trained using O4 and S5, which have smaller alignment gaps, resulting in improved WER while maintaining latency similar to S5. Furthermore, it can be observed that C3 achieved the greatest balance between accuracy and latency.

[0229] Figure 12 illustrates the experimental results showing the effect of knowledge distillation weight (KD weight) on performance.

[0230] Referring to Figure 12, experiments were conducted on the same teacher-student pairs with various KD weights, and it was confirmed that the results converged to specific weight values.

[0231] In particular, since both the word error rate (WER) and latency were improved in experiment A3, a subsequent experiment was performed in one embodiment of the present invention using a KD weight of 0.3.

[0233] Figure 13 illustrates experimental results comparing performance for distillation loss type.

[0234] In relation to Figure 13, experiments were conducted to evaluate the effects of various types of loss, including mean squared error (MSE), target distillation, and KL divergence (KL), on knowledge distillation training.

[0235] The results showed that the scale difference due to transducer loss was minimal, so MSE and target distillation remained stable even in teacher-student pairs with large alignment intervals. However, in KL, the scale difference with transducer loss increased rapidly with pairs having large alignment intervals, leading to a deterioration in learning performance.

[0236] In addition, unlike MSE and target distillation trained with hard labels, KL achieved better performance when trained with soft labels.

[0238] FIG. 14 is a diagram illustrating the streaming baseline, "FastEmit", and the WER (%) and delay time (ms) of the method according to an embodiment of the present invention.

[0239] It was suggested that the smaller the latency interval, the better the offline model trains the streaming model during knowledge distillation learning. Additionally, based on these conditions, it was confirmed that if the offline model is better and faster, the streaming model learns accuracy and latency.

[0240] Referring to Fig. 14, it can be seen that the method proposed in the embodiment of the present invention reduces the WER from 4.0% to 3.7% and the delay time from 20ms to -110ms compared to the streaming conformer converter (conformer-T) baseline.

[0242] FIG. 15 is a diagram illustrating the configuration of a knowledge distillation learning device (100) according to one embodiment.

[0243] Referring to FIG. 15, a knowledge distillation learning device (100) includes a processor (1501) and a memory (1502). The memory (1502) stores one or more instructions, a teacher model, and a student model that are executable by the processor (1501). The processor (1501) executes one or more instructions stored in the memory (1502), and can execute the teacher model and the student model by executing the instructions. By executing the instructions, the processor (1501) can execute one or more operations described above in relation to FIGS. 1 through 12. Additionally, the configuration of the present invention described above together with FIGS. 1 and 2 may be a configuration implemented by instructions executed by the processor (1501).

[0245] Although embodiments of the knowledge distillation learning device and the control method thereof according to the present invention have been described above, they are described as at least one embodiment, and the technical concept, configuration, and operation of the present invention are not limited thereby. Furthermore, the scope of the technical concept of the present invention is not limited or restricted by the drawings or the description with reference to the drawings. In addition, the concept and embodiments of the invention presented in this invention may be used by those skilled in the art to serve as a basis for modifying or designing other structures to perform the same purpose of the present invention. An equivalent structure modified or altered by those skilled in the art to serve as a basis is bound by the technical scope of the present invention as described in the claims, and various changes, substitutions, and modifications are possible within the limits of not departing from the concept or scope of the invention as described in the claims.

Claims

Claim 1 A knowledge distillation learning method performed by first and second models executed by a processor, comprising: a step in which the processor receives training data; a step in which the first model outputs a first probability distribution based on the input training data; a step in which the second model outputs a second probability distribution based on the input training data; a step in which the second model is trained based on the first and second probability distributions; and a step in which alignment is performed between the first model and the second model by controlling a delay time; wherein the delay time control is performed in a manner that adjusts only the issuance probability of label tokens and maintains the issuance probability of blank tokens. Claim 2 In claim 1, the first model and the second model are end-to-end streaming automatic speech recognition models, a knowledge distillation learning method. Claim 3 In claim 2, the end-to-end streaming automatic speech recognition model is a knowledge distillation learning method, which is a sequence converter model. Claim 4 A knowledge distillation learning method according to claim 1, wherein the first model is an offline model for automatic speech recognition and the second model is a streaming model for automatic speech recognition. Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 A knowledge distillation learning device comprising: a memory storing first and second models and instructions; and a processor configured to execute the stored instructions, wherein the processor comprises: the first model outputting a first probability distribution based on input training data; the second model outputting a second probability distribution based on the input training data; and a learning unit for training the second model based on the first and second probability distributions, wherein the processor performs alignment by controlling a delay time of at least one of the first model and the second model, and wherein, in controlling the delay time, the processor adjusts the issuance probability of a label token while maintaining the issuance probability of a blank token as is. Claim 10 In claim 9, the first model and the second model are knowledge distillation learning devices, which are end-to-end streaming automatic speech recognition models. Claim 11 In claim 10, the end-to-end streaming automatic speech recognition model is a knowledge distillation learning device, which is a sequence converter model. Claim 12 A knowledge distillation learning device according to claim 9, wherein the first model is an offline model for automatic speech recognition and the second model is a streaming model for automatic speech recognition. Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete

Images