Method and apparatus for improving the performance of attention-based sequence-to-sequence models

Abstract

To provide a method and apparatus for improving performance of an attention-based sequence-to-sequence model.SOLUTION: A method for improving performance of an attention-based sequence-to-sequence model includes the steps of: determining an output sequence corresponding to an input sequence on the basis of the attention-based sequence-to-sequence model; selecting one or more target attention heads among a plurality of attention heads; detecting one or more error output tokens among output tokens constituting the output sequence on the basis of the target attention head; and correcting the output sequence on the basis of the error output token.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The following embodiments relate to a method and apparatus for improving the performance of an attention-based sequence-to-sequence model. [Background technology]

[0002] An encoder-decoder artificial neural network model, which is a sequence-to-sequence model, has been introduced as a method for solving problems that require receiving an input sequence of an indefinite length and generating an output of an indefinite length using an artificial neural network.

[0003] The sequence-to-sequence model, which takes a variable length input and generates a variable length, is an artificial neural network model used in various tasks such as translation, speech recognition, conversation, and document summarization, and consists of an encoder that converts a variable input into a context vector, which is a higher-level representation, and a decoder that generates a variable output based on the context vector. However, the sequence-to-sequence model has the disadvantage that performance decreases as the context vector becomes longer. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a method and apparatus for improving the performance of an attention-based sequence-to-sequence model. [Means for solving the problem]

[0005] A method for improving performance of an attention-based sequence-to-sequence model according to one embodiment includes a step of determining an output sequence corresponding to an input sequence based on the attention-based sequence-to-sequence model, a step of selecting one or more target attention heads from a plurality of attention heads, a step of detecting one or more erroneous output tokens from output tokens constituting the output sequence based on the target attention head, and a step of correcting the output sequence based on the one or more erroneous output tokens.

[0006] The selecting step may include selecting as the target attention head an attention head that produces a particular attention weight matrix that is trained to be a target attention weight matrix corresponding to the target attention head.

[0007] The particular attention weight matrix may be learned based on a guide weight matrix having a predetermined form.

[0008] The guide weight matrix may be determined based on at least one of an output sequence length, an input frame length, a start shift, an end shift, and a diffusion ratio.

[0009] The particular attention weight matrix may be trained such that the distribution of attention weights differs at each step.

[0010] The particular attention weight matrix can be trained such that the attention weights of a current step are determined based on a cumulative sum of attention weights of previous steps.

[0011] The selecting step may include selecting as the target attention head an attention head that generates an attention weight matrix that best suits a predetermined purpose.

[0012] The selecting step may include selecting the target attention head based on a guide weight matrix having a predetermined shape according to a predetermined objective.

[0013] The selecting step may include, when the attention-based sequence-to-sequence model has a monotonic property, selecting the target attention head by performing a monotonic regression analysis on an attention weight matrix generated by the multiple attention heads.

[0014] The selecting step may include selecting the target attention head based on the entropy of an attention weight matrix produced by the multiple attention heads.

[0015] The step of selecting the target attention head based on the entropy may include a step of selecting, as the target attention head, an attention head that produces an attention weight matrix among the attention weight matrices with the largest entropy.

[0016] Selecting the target attention head based on the entropy may include selecting the target attention head based on Kullback-Leibler divergence.

[0017] The selecting step may include selecting as the target attention head an attention head that produces an attention weight matrix in which the distance between the distributions of each row of the attention weight matrix is ​​the furthest.

[0018] The detecting step may include a step of detecting one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, the difference between the attention weight of the target attention head and a guide weight matrix being greater than or equal to a threshold, and a step of determining output tokens corresponding to the one or more erroneous attention weights as the one or more erroneous output tokens.

[0019] The detecting step may include a step of detecting one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, the erroneous attention weights having a similarity to the attention weights of the previous step that is greater than or equal to a threshold value, and a step of determining output tokens corresponding to the one or more erroneous attention weights as the one or more erroneous output tokens.

[0020] The correcting step may include removing the one or more erroneous output tokens from the output sequence.

[0021] The step of correcting may include a step of determining a next input token from among other output token candidates excluding the one or more erroneous output tokens.

[0022] According to an embodiment, the method for improving performance of an attention-based sequence-to-sequence model may further include determining an input token of a step in which the one or more error output tokens are output as the next input token.

[0023] The number of the plurality of attention heads may correspond to the product of the number of attention layers and the number of decoder layers of the attention-based sequence-to-sequence model.

[0024] An electronic device for an attention-based sequence-to-sequence model according to one embodiment includes a processor that determines an output sequence corresponding to an input sequence based on the attention-based sequence-to-sequence model, selects one or more target attention heads from among a plurality of attention heads, detects one or more erroneous output tokens from among output tokens constituting the output sequence based on the target attention head, and corrects the output sequence based on the one or more erroneous output tokens.

[0025] The processor may select as the target attention head an attention head that produces a particular attention weight matrix that is trained to be a target attention weight matrix corresponding to the target attention head.

[0026] The particular attention weight matrix may be learned based on a guide weight matrix having a predetermined form.

[0027] The particular attention weight matrix may be trained such that the distribution of attention weights differs at each step.

[0028] The processor may select as the target attention head an attention head that generates an attention weight matrix that best suits a predetermined purpose.

[0029] The processor may select the target attention head based on a guide weight matrix having a predetermined shape according to a predetermined objective.

[0030] When the attention-based sequence-to-sequence model has a monotonic property, the processor may select the target attention head by performing a monotonic regression analysis on an attention weight matrix generated by the plurality of attention heads.

[0031] The processor may select the target attention head based on the entropy of an attention weight matrix produced by the multiple attention heads.

[0032] The processor may select an attention head that produces an attention weight matrix among the attention weight matrices with the largest entropy as the target attention head.

[0033] The processor may select as the target attention head an attention head that produces an attention weight matrix in which the distance between the distributions of each row of the attention weight matrix is ​​the furthest.

[0034] The processor may detect one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, where the difference between the attention weight of the target attention head and a guide weight matrix is ​​greater than or equal to a first threshold, and determine output tokens corresponding to the one or more erroneous attention weights as the one or more erroneous output tokens.

[0035] The processor may detect one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, which have a similarity to the attention weights of the previous step that is greater than or equal to a second threshold, and determine the output tokens corresponding to the erroneous attention weights as the one or more erroneous output tokens.

[0036] The processor may remove the one or more erroneous output tokens from the output sequence.

[0037] The processor may determine a next input token from among other output token candidates after removing the one or more erroneous output tokens.

[0038] The processor may determine, as the next input token, the input token of the step that output the one or more error output tokens.

[0039] An electronic device according to one embodiment includes an encoder and a decoder that receive an input sequence and output an output sequence based on the input sequence, and at least one processor, wherein the at least one processor selects a target attention head from among a plurality of attention heads included in an attention-based sequence-to-sequence model of the decoder, detects an error output token included in the output sequence based on the target attention head, corrects the output sequence based on the error output token, and outputs the corrected output sequence.

[0040] The encoder and the decoder may be comprised in an artificial neural network.

[0041] The at least one processor may select the target attention head from among a plurality of attention weight matrices stored in the attention-based sequence-to-sequence model.

[0042] The attention-based sequence-to-sequence model stores multiple attention weight matrices corresponding to multiple attention heads by inputting example inputs that produce correct results to the encoder and the decoder, and the at least one processor may select as the target attention head an attention head that generates an attention weight matrix that is most suitable for a predetermined objective of the sequence-to-sequence model.

[0043] The at least one processor may learn to generate an attention weight matrix in a desired form for a specific attention head among the plurality of attention heads, and may determine the specific attention head as the specific attention head. Effect of the Invention

[0044] According to the present invention, a method and apparatus for improving the performance of an attention-based sequence-to-sequence model can be provided. [Brief description of the drawings]

[0045] [Figure 1] FIG. 1 is a configuration diagram of a system in which a sequence-to-sequence model is used according to one embodiment. [Figure 2A] FIG. 2 is a diagram for explaining the operation of an electronic device having an end-to-end artificial neural network of an encoder-decoder structure according to an embodiment. [Figure 2B] FIG. 13 is a diagram for explaining a method for determining an attention weight according to an embodiment. [Figure 2C] FIG. 13 is a diagram for explaining a method for determining an attention weight according to an embodiment. [Figure 3A] FIG. 1 is a diagram for explaining an attention weight matrix according to an embodiment. [Figure 3B] FIG. 1 is a diagram for explaining an attention weight matrix according to an embodiment. [Figure 4A] FIG. 1 is a block diagram of an electronic device according to an embodiment. [Figure 4B] FIG. 13 is a block diagram of an electronic device according to another embodiment. [Diagram 5] 1 is a flowchart illustrating a method for improving performance of an attention-based sequence-to-sequence model according to an embodiment. [Figure 6] FIG. 13 is a diagram illustrating a guide weight matrix having a predetermined form according to an embodiment. [Figure 7] 11 is a diagram illustrating a guide weight matrix configured to have different attention weights when generating each output token according to an embodiment; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] The specific structural or functional description of the embodiment is disclosed for the purpose of illustration only, and may be modified in various forms. Therefore, the embodiment is not limited to the specific disclosed form, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0047] Although terms such as first or second may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be named as a second component, and similarly, a second component may be named as a first component.

[0048] In addition, in describing components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are merely for distinguishing the components from other components, and do not limit the nature, order, etc. of the components. When a component is referred to as being "coupled," "coupled," or "connected" to another component, it should be understood that the component is directly coupled or connected to the other component, but that additional components may be "coupled," "coupled," or "connected" between each component.

[0049] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the words "comprise" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0050] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used and predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as ideal or overly formal unless expressly defined in this specification.

[0051] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings, in which the same reference numerals in each drawing denote the same elements.

[0052] FIG. 1 is a configuration diagram of a system in which a sequence-to-sequence model according to an embodiment is used.

[0053] Referring to Fig. 1, a system using a sequence-to-sequence model includes a user terminal 110 and an electronic device 120. Each component of the system using the sequence-to-sequence model shown in Fig. 1 indicates a functional element that is functionally divided, and at least one component may be realized in a form integrated with each other in an actual physical environment. For example, although the electronic device 120 is shown as a kind of server device in Fig. 1, each function of the electronic device 120 may be realized in a form built into the user terminal 110 according to an embodiment.

[0054] The sequence-to-sequence model may be used in various fields that output an output sequence in another domain from an input sequence. For example, the sequence-to-sequence model may be used in a speech recognition system, a machine translation system, a chatbot system, etc. A speech recognition system may receive a speech and output a speech recognition sentence, a machine translation system may receive a first language sentence and output a corresponding second language sentence, and a chatbot system may receive a question and output a reply corresponding to the question. For convenience of explanation, the following description will be given based on a sequence-to-sequence model in a speech recognition system, but the embodiment may be applied to various fields in which the sequence-to-sequence model may be used.

[0055] In a speech recognition system, the user terminal 110 is a terminal that receives and transmits an input sequence 130 to the electronic device 120 and provides a user with an output sequence 140 received from the electronic device 120. In Fig. 1, the user terminal 110 is shown as a smartphone, but the user terminal 110 can be one of the electronic devices such as a computer, an Ultra Mobile PC (UMPC), a workstation, a net-book, a Personal Digital Assistant (PDA), a portable computer, a web tablet, a wireless phone, a mobile phone, an e-book, a portable multimedia player (PMP), a portable game console, a navigation device, a black box, or a digital camera.

[0056] The electronic device 120 is a computer device that receives a speaker's input sequence 130 and provides an output sequence 140, which is a recognition result, and is an electronic device based on a sequence-to-sequence model. The input sequence 130 includes input data and an input feature vector extracted from the input data. For example, in a speech recognition system, the input sequence 130 includes an input voice and an input voice feature vector extracted from the input voice. The computer device may be a mobile device such as a notebook computer, a desktop computer, a laptop computer, or a smart phone, but is not limited thereto, and may include all types of devices equipped with a computing device.

[0057] The electronic device 120 constructs a sequence-to-sequence model composed of an artificial neural network, and uses the constructed sequence-to-sequence model to provide an output sequence 140, which is a recognition result for an input sequence 130. The artificial neural network may be, for example, a recurrent neural network (RNN), a convolutional neural network (CNN), a self-attention neural network (SANN), a bi-directional RNN (BRNN), a long short term memory (LSTM), a bi-directional LSTM (BLSTM), a gated recurrent unit (GRU), a bi-directional GRU (BGRU), etc., but is not limited thereto.

[0058] The electronic device 120 also includes a correction device for improving the accuracy of the output sequence 140. The correction device indicates a functional element functionally separated from the electronic device 120, and in an actual physical environment, at least one component of the correction device may be integrated with the electronic device 120. The electronic device 120 can use the correction device to detect errors in the output sequence 140 and output a corrected output sequence.

[0059] In the following, with reference to Figures 2A to 2C, we will explain how the multi-head attention-based sequence-to-sequence model operates, with reference to Figures 3A to 3B, we will explain the attention weighting matrix, and with reference to Figures 4A to 7, we will explain how to improve the performance of the attention-based sequence-to-sequence model.

[0060] FIG. 2A is a diagram illustrating the operation of an electronic device having an end-to-end artificial neural network in an encoder-decoder structure according to an embodiment.

[0061] Referring to FIG. 2A, an artificial neural network according to one embodiment includes an encoder 220 and a decoder 230.

[0062] The artificial neural network is an end-to-end artificial neural network with an encoder 220-decoder 230 structure, and since a route is possible where all parameters of the model are simultaneously trained for one loss function, the encoder 220 and the decoder 230 can be trained simultaneously. The artificial neural network receives input from one end and generates output at the other end, and the network weights can be optimized by directly considering the input and output. The nodes of the layers in the artificial neural network are in a nonlinear and mutually influencing relationship, and the parameters of the artificial neural network, such as the values ​​output from each node and the relationship between nodes, can be optimized by learning.

[0063] The end-to-end artificial neural network with the encoder 220-decoder 230 structure is a network structure in which the encoder 220 and the decoder 230 are integrated, and the encoder 220 and the decoder 230 directly generate an output sequence corresponding to the input sequence from the input sequence.

[0064] For example, the electronic device may extract an input feature vector 215 from input data 210, and the encoder 220 may encode the input feature vector 215 to generate encoded features 225. The encoder 220 may transform the dimensions of the input feature vector 215 to generate encoded features 225. That is, the encoder 220 serves to smoothly summarize the meaning of the input sequence.

[0065] The encoded features 225 may be applied to an artificial neural network decoder 230. The decoder 230 may determine output tokens based on the token-wise encoded features 225 and previously determined recognition results.

[0066] Specifically, the decoder 230 that receives the encoded features 225 outputs the recognition result in units of tokens, and a token is a word, subword, character, or a unit that constitutes a single character (initial, neutral, final, etc. in Hangul). Hereinafter, the entire recognition result output by the decoder 230 is referred to as an output sequence. The output sequence includes a plurality of output tokens.

[0067] The decoder 230 uses a special token to indicate the end of the sequence in order to compute the output of the artificial neural network of undetermined length. <eos>Until the next token is output, the next token can be predicted one by one by inputting the tokens estimated so far. The decoder 230 obtains the output token based on the information calculated from the encoder 220 for each step, and here, the output token can be obtained by belonging to the output token of the previous step. This method is called auto-regressive decoding.

[0068] That is, the token selected as the output token from the end-to-end artificial neural network model performing autoregressive decoding can be determined as the next input token. For example, when recognizing "me" in the sentence "tell me a joke", the output token "tell" of the previous step is considered.

[0069] However, since the end-to-end artificial neural network with the encoder 220-decoder 230 structure processes the entire input data into a single vector, if the input data is long, the information of the earlier part of the data is diluted, resulting in a problem of degraded quality of the output sequence. To solve this problem, the entire input data in the encoder 220 can be referred to again for each step of predicting the output token in the decoder 230. However, instead of referring to the entire input data in the same proportion, more focus can be placed on the input speech part related to the output token to be predicted in the corresponding step.

[0070] More specifically, an attention weight is assigned to each input frame constituting the input sequence according to the degree to which it is related to the output token that must be predicted at the corresponding step, and the decoder 230 can predict the output token for each step based on the assigned attention weight.

[0071] The encoder 220 and the decoder 230 may be pre-trained to generate a sequence of recognition results from an input feature vector. For example, the model parameters (e.g., synaptic connection weights and neuronal biases) of the encoder 220 and the decoder 230 may be pre-trained to generate a sequence of recognition results from a sequence of ground-truth data pairs corresponding to the input data. Furthermore, the attention weights of the encoder 220 and the decoder 230 may also be pre-trained to generate a sequence of recognition results from a sequence of ground-truth text pairs corresponding to the input data.

[0072] 2B to 2C are diagrams for explaining a method for determining attention weights according to an embodiment.

[0073] Referring to FIG. 2B, the encoder 240 receives an input frame (e.g., i 1 ~i N , N is the number of input frames) and the encoder’s hidden state vector (e.g., h 1 ~h N ) can be generated.

[0074] The attention module 251 selects the input frames (e.g., i 1 ~i N ) and calculate the attention weights for the context vector (e.g., c 1 ~c M , M is the number of output tokens).

[0075] In the case of a multi-head attention-based sequence-to-sequence model, the attention module 251 may be a multi-head attention module, and the attention layer may include multiple attention layers (e.g., the first attention layer to the kth attention layer), each of which may determine attention weights in parallel and generate a context vector based on the determined attention weights. The attention module 251 may use multiple attention layers that are implicitly trained to play different roles.

[0076] The decoder 250 receives the hidden state vector s t-1 , the output token y of the previous step t-1 , and the context vector c t is input, and the hidden state vector s t and the output token y of the current step t The current step context vector c is used when the decoder extracts the output tokens in the current step. t can be calculated using the following formula (1).

number

[0077] Referring to FIG. 2C, α t i is the attention weight for the i-th input frame for step t, and the sum of the attention weights for all input frames (e.g., from 1st to Nth) is 1. If α t 1 Or α t N Of which, α t 2 If the value of is the largest, the decoder 250 can extract output tokens with the most focus on the second input frame at the current step t.

[0078] Although the multi-head attention model has improved performance over the existing single-attention model, the chronic shortcomings of sequence-to-sequence models with variable-length outputs, such as jumping some outputs or repeating the contents of previous outputs, remain.

[0079] For example, when a voice corresponding to the sentence "tell me a joke" is input to the multi-head based sequence-to-sequence model, a phenomenon may occur in which "me a" is jumped after "tell" and "joke" is output, depending on the situation.

[0080] In addition, when a rare sound or a sound corresponding to a previously unseen word (essendi rerum) is input, the multi-head based sequence-to-sequence model may repeat a phrase or a single word as shown in Table 1 below. [Table 1] In this regard, a correction device according to one embodiment can select a target attention head from a plurality of attention heads, detect one or more erroneous output tokens based on the target attention head, and correct the output sequence based on the erroneous output tokens to improve the accuracy of the final output sequence of the electronic device.

[0081] 3A and 3B are diagrams for explaining an attention weight matrix according to an embodiment.

[0082] In one embodiment, the attention weight matrix may have, as elements, attention weights of each input frame constituting an input sequence for each output token constituting an output sequence. For example, when an electronic device outputs an output sequence such as "tell me a joke", the attention weight matrix is ​​as shown in Table 2 below. [Table 2] In the example of Table 2, the output tokens are words, and each input frame may have a time unit of 10 ms. 1 The attention weight of i is the largest at 0.9, 2 The attention weight of i is the second largest at 0.07, 3 The attention weight of i is 0.02, which is the third largest. 1 ~i 3 The sum of all the attention weights is 0.99, and the output token “tell” is 1 ~i 3 It can be seen that the extract is concentrated in

[0083] In a multi-head based sequence-to-sequence model, one attention weight matrix is ​​created per attention head, and each attention weight matrix can be implicitly trained to play its own role. For example, in a multi-head based sequence-to-sequence model with n attention heads, n attention weight matrices can be created.

[0084] Referring to FIG. 3A, the attention weights of the attention weight matrix may be expressed using colors. For example, in the two attention weight matrices 310 and 320, the larger the attention weight, the closer to white it is represented, and conversely, the smaller the attention weight, the closer to black it is represented. The horizontal axis of the two attention weight matrices 310 and 320 is the order of input frames over time from left to right, and the vertical axis is the order of output tokens (e.g., words) over time from top to bottom.

[0085] When an output sequence is predicted using multiple attention weight matrices, the roles of each attention weight matrix may be different. Although they are learned implicitly, the two attention weight matrices 310 and 320 can be interpreted as playing different roles. For example, the attention weight matrix 310 has a large attention weight in silent sections where there is no actual speech, which helps to ignore background noise when constructing an output, and the attention weight matrix 320 has a high attention weight in positions where each word is actually spoken, which allows focusing on the speech part of the word to be actually recognized.

[0086] As mentioned above, although the multi-head attention model has improved performance over the existing single attention model, it has a chronic shortcoming of sequence-to-sequence models that output variable lengths, such as jumping some outputs or repeating previous output content.

[0087] For example, the attention weight matrix 330 shown in FIG. 3B is an attention weight matrix when recognition cannot be performed normally and a repeated error is output. The attention weight matrix 330 may be an attention weight matrix created with an attention head like the attention weight matrix 320, but the attention weight matrix 320 may be an attention weight matrix when an output token is normally generated, and the attention weight matrix 330 may be an attention weight matrix when an error that repeats the previous output content is output. Referring to the attention weight matrix 330, it can be seen that the result is normally output up to about 1 / 3 of the way point based on the x-axis, which is the output token axis, but several output tokens (e.g., words) thereafter are repeatedly output.

[0088] In order to correct the above-mentioned errors, the correction device according to an embodiment can improve the performance of the multi-head attention sequence-to-sequence model by selecting a specific attention head, analyzing the attention weight of the attention head (hereinafter, referred to as a target attention head), and suppressing the error output. A specific method for improving the performance of the attention-based sequence-to-sequence model will be described with reference to the following Figures 4A to 7.

[0089] FIG. 4A is a block diagram of an electronic device according to one embodiment. Referring to FIG. 4A, an electronic device 401 according to an embodiment includes an encoder 420, a decoder 430, a target attention head selection module 460, and an error output token detection and correction module 470. However, FIG. 4A shows only components according to an embodiment, and other general components may be further included. In addition, it should be noted that each component of the electronic device 401 shown in FIG. 4A is shown separately to show that it can be functionally and logically separated, and does not necessarily mean that it is realized in a physically separate component or separate code, but may be realized in a form integrated with each other in an actual physical environment. Each component of the electronic device 401 will be described below.

[0090] The electronic device 401 further includes a feature vector extraction module (not shown). The feature vector extraction module extracts a feature vector from an input signal (e.g., an audio signal). The input signal is a signal including information for each of a plurality of frames, and the feature vector is a sequence of information extracted in units of at least one frame, which may be expressed as a multi-dimensional vector. Hereinafter, the input sequence may include the input signal and the feature vector extracted from the input signal, or may include information in units of input frames.

[0091] The encoder 420 and the decoder 430 shown in FIG. 4A correspond to the encoder 220 and the decoder 230 described with reference to FIG. 2A. Therefore, the description overlapping with FIG. 2A will be omitted. The decoder 430 includes an attention module 440 that determines which input frame should be more focused on each time an output token is estimated for each step. Specifically, the attention module 440 is used when the decoder 430 estimates an output token for each step, and the decoder 430 that determines the attention weight of each input frame constituting an input sequence at the corresponding step can estimate an output token for each step based on the set attention weight. The attention module 440 may be a multi-head attention module. The multi-head attention module can apply attention multiple times by dividing the entire dimension into multiple parts using multiple attention layers trained to play various roles, instead of applying attention only once to the entire dimension. Also, the attention module 440 can store the attention weight matrix of all attention heads.

[0092] The electronic device 401 uses an encoder 420 and a decoder 430 to output an output sequence 450 that corresponds to the input sequence 410. However, as previously mentioned, the output sequence 450 may be different from what was expected.

[0093] An electronic device 401 according to one embodiment can select a target attention head from among multiple attention heads using a target attention head selection module 460, detect erroneous output tokens among output tokens included in an output sequence based on the selected target attention head using an erroneous output token detection and correction module 470, and output a corrected output sequence 480 as a final result.

[0094] More specifically, the target attention head selection module 460 may select a target attention head from among a plurality of attention weight matrices stored in the attention module 440. The attention module 440 may input examples that produce correct results to the encoder 420 and the decoder 430 and store attention weight matrices for all attention heads, and the target attention head selection module 460 may select an attention head that produces the most appropriate attention weight matrix according to the purpose of the sequence-to-sequence model from among all the stored attention heads as a target attention head. Here, the operation of storing the attention weight matrix and selecting the target attention head may be performed in advance before the input sequence 410 to be judged for error is input.

[0095] As previously mentioned, the electronic device 401 includes a correction device, which includes a target attention head selection module 460 and an error output token detection and correction module 470 .

[0096] FIG. 4B is a block diagram of an electronic device according to another embodiment.

[0097] The electronic device 402 shown in Fig. 4B may include an attention guide module 490 instead of the target attention head selection module 460 of the electronic device 401 shown in Fig. 4A. The explanations shown in Figs. 1 to 4A are also applicable to Fig. 4B, so the explanation of the overlapping contents will be omitted.

[0098] Unlike the electronic device 401 shown in FIG. 4A which selects a target attention head from the attention weight matrix stored in the attention module 440, the electronic device 402 shown in FIG. 4B can learn to explicitly utilize a specific attention head as the target attention head.

[0099] More specifically, the attention guide module 490 can learn to generate an attention weight matrix of a desired shape based on an arbitrarily determined attention head, and can later inform the error output token detection and correction module 470 of the attention head learned in the target attention layer when determining whether the output sequence is erroneous.

[0100] As previously mentioned, the electronic device 402 may include a correction device, which may include an attention guide module 490 and an error output token detection and correction module 470 .

[0101] FIG. 5 is a flowchart illustrating a method for improving the performance of an attention-based sequence-to-sequence model according to an embodiment.

[0102] Referring to Fig. 5, steps S510 to S540 can be performed by the electronic device described above with reference to Figs. 1 to 4B. The electronic device may be implemented by one or more hardware modules, one or more software modules, or various combinations thereof, and as described above, the electronic device includes a correction device. The description of Figs. 1 to 4B may also be applied to Fig. 5, so the description of the overlapping contents will be omitted.

[0103] In step S510, the electronic device determines an output sequence corresponding to the input sequence based on the attention-based sequence-to-sequence model. For example, the electronic device may receive an input speech or an input speech feature vector corresponding to the input speech.

[0104] In step S520, the electronic device selects one or more target attention heads from among the multiple attention heads. The electronic device based on the multi-head attention-based sequence-to-sequence model may have as many attention heads as the integral of the number of attention layers and the number of decoder layers. The attention weight matrix generated by each attention head may have different roles. The electronic device may select, as the target attention head, one or more attention heads that generate an attention weight matrix most suitable for the purpose performed by the corresponding multi-head attention-based sequence-to-sequence model from among the attention heads that generate attention weight matrices that perform different roles.

[0105] There are two main methods for selecting a target attention head. The first is to input an example input that produces a correct result into an electronic device, pre-store attention weight matrices for all attention heads, and select an attention head that produces the most appropriate attention weight matrix according to the purpose of the sequence-to-sequence model from among all the stored attention heads. The second is to train an arbitrarily determined attention head to generate an attention weight matrix of a desired form, and select the attention weight matrix thus trained as the target attention layer.

[0106] According to the first method, the electronic device selects, from among a plurality of attention heads, an attention head that generates an attention weight matrix that is most suitable for a predetermined purpose as a target attention head.

[0107] According to an embodiment, the electronic device may select a target attention head based on a guide weight matrix having a predetermined form according to a predetermined purpose. For example, speech recognition has a monotonic property if an input sequence (e.g., an input voice) and an output sequence (e.g., a recognized word) are mapped according to time. In this case, the electronic device performs a monotonic regression analysis on the attention weight matrix generated by a plurality of attention heads, and selects the attention head that most satisfies the monotonic property as the target attention head. In this case, the guide weight matrix may be a monotonic matrix.

[0108] FIG. 6 is a diagram illustrating a guide weight matrix having a predetermined form according to an embodiment.

[0109] 6, the guide weight matrix 600 may have attention weights that induce the attention-based sequence-to-sequence model to have a monotonic property. The attention weights of the guide weight matrix 600 are determined as shown in Equation (2) below.

number

[0110] Referring to FIG. 5, as another example of the first method, in the case of a task that requires various inputs to be viewed, such as translation and summarization, the guide attention weights of the guide weight matrix do not have a predetermined form, and are configured to have different attention weights each time an output token is generated. In this case, the electronic device may select a target attention head based on the entropy of an attention weight matrix generated by a plurality of attention heads. The electronic device may select an attention head that generates an attention weight matrix with the largest entropy among the attention weight matrices as the target attention head. For example, the electronic device may select an attention head that generates an attention weight matrix with the greatest distance between the distributions of each row of the attention weight matrix among the attention weight matrices as the target attention head. Specifically, the electronic device may calculate Kullback-Leibler divergence and select an attention head that generates an attention weight with the largest Kullback-Leibler divergence as the target attention head. A person skilled in the art can clearly understand Kullback-Leibler divergence, so a detailed description will be omitted.

[0111] FIG. 7 is a diagram illustrating a guide weight matrix configured to have different attention weights each time an output token is generated according to an embodiment.

[0112] 7, the guide weight matrix 700 may be configured to have different attention weights each time an output token is generated. In FIG. 7, the x-axis corresponds to the output sequence and the y-axis corresponds to the input sequence. When predicting the t output token, the sixth input frame may have a larger attention weight than the previous one, which had a lower weight when predicting the previous output token.

[0113] Also, referring to Fig. 5, according to the second method, the electronic device may learn an arbitrarily determined attention head to generate an attention weight matrix of a desired form and select the thus learned attention weight matrix as the target attention layer. The electronic device may select as the target attention head an attention head that produces a particular attention weight matrix that is trained to be the target attention weight matrix corresponding to the target attention head.

[0114] As an example, a specific attention weight matrix may be learned based on a guide weight matrix having a predetermined form according to a predetermined purpose. The guide weight matrix having a predetermined form according to a predetermined purpose is the guide matrix described above in the first method. For example, the electronic device may learn a specific attention weight matrix to have a monotonic property based on the guide matrix 600 shown in FIG. 6.

[0115] As another example, in the case of a task that requires diverse inputs such as translation and summarization, a specific attention weight matrix may be trained such that the distribution of attention weights differs for each step. A specific attention weight matrix may be trained such that the attention weights of a current step are determined based on the cumulative sum of attention weights of previous steps. For example, referring to FIG. 7, a cumulative sum of attention weights of previous steps may be calculated for each step of predicting an output token, and a specific attention weight matrix may be trained such that a high attention weight is assigned to a location where the cumulative sum of attention weights of the previous step is small.

[0116] In the second method, to learn a specific attention weight matrix, the guide weight matrix can be used as a bias, and the artificial neural network learning process can naturally learn to generate attention weights that are similar in form to the guide weight matrix, or a loss function such as Kullback-Leibler divergence can be explicitly added as the artificial neural network learning loss.

[0117] In step S530, the electronic device detects one or more erroneous output tokens based on a target attention head among the output tokens constituting the output sequence. The electronic device may determine the target attention head in advance before determining whether the output sequence is erroneous. The electronic device may detect one or more erroneous output tokens based on the determined target attention head.

[0118] As an example, the electronic device may obtain an attention weight matrix between an input sequence and an output sequence to be determined whether or not the target attention head has an error (hereinafter referred to as a target attention weight matrix). The electronic device may detect one or more erroneous attention weights in which the difference between the attention weights of the target attention weight matrix and the attention weights of the guide weight matrix is ​​equal to or greater than a first threshold. For example, the electronic device may detect one or more erroneous attention weights in which the difference between the Kullback-Leibler divergence of the attention weights of the target attention weight matrix and the attention weights of the guide weight matrix is ​​equal to or greater than a first threshold. The electronic device may determine an output token corresponding to the erroneous attention weight as the erroneous output token.

[0119] As another example, in the case of a task that requires various views of the input, such as translation or summarization, the electronic device may detect one or more error attention weights among the attention weights in the target attention weight matrix that have a similarity to the attention weights of the previous step that is greater than or equal to a second threshold.

[0120] In step S540, the electronic device corrects the output sequence based on the erroneous output token. In one embodiment, the electronic device may detect the erroneous output token after determining the entire output sequence, and correct the output sequence based on the erroneous output token. In this case, the electronic device can correct the error in the output sequence by removing the erroneous output token from the output sequence.

[0121] Alternatively, an electronic device according to another embodiment may detect an error output token for each step of predicting an output token in real time and correct the output sequence based on the error output token. If the error output token is not corrected, the electronic device has no choice but to determine the next output token based on the error output token. However, the electronic device may not only remove the error output token from the output sequence, but also determine the next input token from among other output token candidates after removing the error output token. According to an embodiment, the input token of the step in which the error output token is output may be determined as the next input token. When a voice corresponding to the sentence "tell me a joke" is input, if a phenomenon occurs in which "me a" is jumped after "tell" and "joke" is output, "joke" is not input as the next input token, but "tell" is input again as the input token. Alternatively, the error output token may be excluded and the output token candidate with the highest probability may be determined as the next input token.

[0122] It can be seen that the electronic device according to the embodiment corrects the phenomenon of one word repetition as shown in Table 3 below. [Table 3] The method according to the embodiment may be embodied in the form of program instructions to be executed by various computer means and recorded on a computer readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and the program instructions may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code, such as produced by a compiler, but also high level language code, which is executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0123] The software may include computer programs, codes, instructions, or a combination of one or more thereof, which may configure or independently or in combination with a processor to operate as desired. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, either permanently or temporarily, to be interpreted by the processor or to provide instructions or data to the processor. The software may be distributed and stored and executed in a distributed manner on computer systems coupled to a network. The software and data may be stored on one or more computer readable recording media.

[0124] As mentioned above, even if the embodiments are described by limited drawings, those having ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques can be performed in a different order than the described method, and / or the components of the described systems, structures, devices, circuits, etc. can be combined or combined in a different manner than the described method, or replaced or substituted by other components or equivalents to achieve appropriate results.

[0125] Therefore, the scope of the present invention should not be limited to the disclosed embodiment, but should be determined by the appended claims and their equivalents. [Explanation of symbols]

[0126] 110 User terminal 120 Electronic device 210 Input Data 215 Input feature vector 220 Encoder 225 Encoded Features 230 Decoder 240 Encoder 250 Decoder 310, 320, 330 Attention weight matrix 401, 402 Electronic equipment< / eos>

Claims

1. A method for improving performance of an attention-based sequence-to-sequence model implemented on an electronic device, comprising: determining, by a processor of the electronic device, an output sequence corresponding to an input sequence based on an attention-based sequence-to-sequence model; selecting, by the processor, one or more target attention heads from a plurality of attention heads, each of which is configured to generate a respective attention weight matrix; detecting, by the processor, one or more erroneous output tokens among output tokens constituting the output sequence based on the target attention head; correcting, by the processor, the output sequence based on the one or more erroneous output tokens; A performance improvement method comprising:

2. The selecting step includes: selecting as the target attention head an attention head that produces a particular attention weight matrix that is trained to be a target attention weight matrix corresponding to the target attention head; The performance enhancing method of claim 1 .

3. The particular attention weight matrix is ​​learned based on a guide weight matrix having a predetermined form. The performance enhancing method of claim 2.

4. the guide weight matrix is ​​determined based on at least one of an output sequence length, an input frame length, a start shift, an end shift, and a diffusion ratio; The performance enhancing method of claim 3.

5. The particular attention weight matrix is ​​trained so that the distribution of attention weights varies at each step. The performance enhancing method of claim 2.

6. The particular attention weight matrix is ​​learned such that the attention weights of a current step are determined based on a cumulative sum of attention weights of a previous step. The performance enhancing method of claim 2.

7. The selecting step includes: selecting an attention head that generates an attention weight matrix that best suits a predetermined purpose as the target attention head; The performance enhancing method of claim 1 .

8. The selecting step includes: selecting the target attention head based on a guide weight matrix having a predetermined shape according to a predetermined objective; The performance enhancing method of claim 1 .

9. The selecting step includes: If the attention-based sequence-to-sequence model has a monotonic property, performing a monotonic regression analysis on an attention weight matrix generated by the plurality of attention heads to select the target attention head. The performance enhancing method of claim 1 .

10. The selecting step includes: selecting the target attention head based on the entropy of an attention weight matrix produced by the plurality of attention heads; The performance enhancing method of claim 1 .

11. The step of selecting the target attention head based on the entropy includes: selecting an attention head that produces an attention weight matrix with the maximum entropy from among the attention weight matrices as the target attention head; The performance enhancing method of claim 10.

12. The step of selecting the target attention head based on the entropy includes: selecting the target attention head based on Kullback-Leibler divergence; The performance enhancing method of claim 10.

13. The selecting step includes: selecting an attention head that produces an attention weight matrix in which the distance between the distributions of each row of the attention weight matrix is ​​the greatest as the target attention head; The performance enhancing method of claim 1 .

14. The detecting step includes: Detecting one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, the difference between the attention weights of the target attention head and a guide weight matrix being equal to or greater than a threshold; determining output tokens corresponding to the one or more error attention weights as the one or more error output tokens; The performance enhancing method of claim 1 , comprising:

15. The detecting step includes: Detecting one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, the erroneous attention weights having a similarity to the attention weights of the previous step that is equal to or greater than a threshold value; determining output tokens corresponding to the one or more error attention weights as the one or more error output tokens; The performance enhancing method of claim 1 , comprising:

16. The correcting step includes: removing the one or more erroneous output tokens from the output sequence; The performance enhancing method of claim 1 .

17. The correcting step includes: determining a next input token from among other output token candidates after removing the one or more error output tokens; The performance enhancing method of claim 1 .

18. The performance improvement method further comprises: determining an input token of a step in which the one or more error output tokens are output as the next input token; 20. The performance enhancing method of claim 17.

19. The number of the plurality of attention heads corresponds to the product of the number of attention layers and the number of decoder layers of the attention-based sequence-to-sequence model; The performance enhancing method of claim 1 .

20. A computer program including instructions stored on a computer readable storage medium, The instructions, when executed, cause the computer to perform a method according to any one of claims 1 to 19. Computer program.

21. 1. An electronic device for an attention-based sequence-to-sequence model, comprising: a processor, the processor comprising: determining an output sequence corresponding to an input sequence based on an attention-based sequence-to-sequence model; selecting one or more target attention heads from a plurality of attention heads, each of which is configured to generate a respective attention weight matrix; Detecting one or more erroneous output tokens from among the output tokens constituting the output sequence based on the target attention head; correcting the output sequence based on the one or more erroneous output tokens. electronic equipment.

22. The processor, selecting as the target attention head an attention head that produces a particular attention weight matrix that is trained to be a target attention weight matrix corresponding to the target attention head; 22. The electronic device of claim 21.

23. The particular attention weight matrix is ​​learned based on a guide weight matrix having a predetermined form.

23. The electronic device of claim 22.

24. The particular attention weight matrix is ​​trained so that the distribution of attention weights varies at each step.

23. The electronic device of claim 22.

25. The processor, selecting an attention head that generates an attention weight matrix that best suits a predetermined objective as the target attention head; 22. The electronic device of claim 21.

26. The processor, selecting the target attention head based on a guide weight matrix having a predetermined shape according to a predetermined objective; 22. The electronic device of claim 21.

27. The processor, If the attention-based sequence-to-sequence model has a monotonic property, a monotonic regression analysis is performed on an attention weight matrix generated by the plurality of attention heads to select the target attention head.

22. The electronic device of claim 21.

28. The processor, selecting the target attention head based on the entropy of an attention weight matrix produced by the plurality of attention heads; 22. The electronic device of claim 21.

29. The processor, Selecting an attention head that produces an attention weight matrix with the maximum entropy among the attention weight matrices as the target attention head; 30. The electronic device of claim 28.

30. The processor, Select the attention head that produces the attention weight matrix with the greatest distance between the distributions of each row of the attention weight matrix as the target attention head; 22. The electronic device of claim 21.

31. The processor, Detect one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, the difference between the attention weight of the target attention head and a guide weight matrix being equal to or greater than a first threshold; determining an output token corresponding to the one or more error attention weights as the one or more error output tokens; 22. The electronic device of claim 21.

32. The processor, Detect one or more erroneous attention weights among the attention weights between the input sequence and the output sequence of the target attention head, the erroneous attention weights having a similarity to the attention weights of the previous step that is equal to or greater than a second threshold; determining an output token corresponding to the error attention weight as the one or more error output tokens; 22. The electronic device of claim 21.

33. The processor, removing the one or more erroneous output tokens from the output sequence; 22. The electronic device of claim 21.

34. The processor, determining a next input token from among other output token candidates from which the one or more error output tokens have been removed; 22. The electronic device of claim 21.

35. The processor, determining an input token of the step which has output the one or more error output tokens as the next input token; 35. The electronic device of claim 34.

36. 1. An electronic device comprising: an encoder and a decoder for receiving an input sequence and outputting an output sequence based on the input sequence; At least one processor; Including, The at least one processor selecting a target attention head from among a plurality of attention heads included in the attention-based sequence-to-sequence model of the decoder, each of the plurality of attention heads being configured to generate a respective attention weight matrix; Detecting an erroneous output token included in the output sequence based on the target attention head; correcting the output sequence based on the erroneous output token; outputting the corrected output sequence. electronic equipment.

37. the encoder and the decoder are included in an artificial neural network; 37. The electronic device of claim 36.

38. The at least one processor selecting the target attention head from among a plurality of attention weight matrices stored in the attention-based sequence-to-sequence model; 37. The electronic device of claim 36.

39. The attention-based sequence-to-sequence model is By inputting example inputs that produce correct results to the encoder and the decoder, a plurality of attention weight matrices corresponding to a plurality of attention heads are stored; The at least one processor selecting an attention head that generates an attention weight matrix that best suits a predetermined objective of the sequence-to-sequence model as the target attention head; 39. The electronic device of claim 38.

40. The at least one processor learning to generate a desired type of attention weight matrix for a specific attention head among the plurality of attention heads; determining the specific attention head as the specific attention head; 37. The electronic device of claim 36.

Images