Decoding method and speech recognition device in artificial neural network
The introduction of a special token and ensemble methods in neural networks addresses the issue of low probability token prediction, enhancing the reliability and accuracy of decoding in artificial neural networks.
Patent Information
- Application Number
- JP2019171730
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-11-14
- Filing Date
- 2019-09-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-09-20
AI Technical Summary
Existing decoding methods for artificial neural networks face challenges in predicting output tokens with low probabilities, leading to degraded performance in autoregressive decoding due to reliance on previous tokens, especially in sequence-to-sequence models.
Introduce a special token that does not affect conditional probabilities and use ensemble methods across multiple neural networks to select output tokens, masking special tokens to prevent interference and ensuring higher reliability in token prediction.
Enhances the reliability and accuracy of output token prediction by smoothing the generation process, even when tokens have low probabilities, thereby improving the overall performance of neural network decoding.
Smart Images

Figure 0007589878000005 
Figure 0007589878000006 
Figure 0007589878000007
Abstract
Description
[Technical field]
[0001] SUMMARY OF THE PRESENT EMBODIMENTS An embodiment of the present invention relates to a decoding method and apparatus in an artificial neural network. [Background technology]
[0002] To solve the problem of receiving a sequence of an undefined length as an input to an artificial neural network and generating an output of an undefined length, an encoder-decoder artificial neural network model, which is one of the sequence-to-sequence models, has been introduced. In particular, the method of inputting the previous output of an artificial neural network for each token, which is a unit that constitutes a sequence, and outputting the next output in succession to calculate an output of an undefined length in an artificial neural network is called auto-regressive decoding. Summary of the Invention [Problem to be solved by the invention]
[0003] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a decoding method and device in an artificial neural network. [Means for solving the problem]
[0004] In one embodiment, a decoding method in a first artificial neural network includes a step of predicting a probability of a candidate for an output token based on at least one previously inputted input token, a step of determining the output token from among the candidates based on the predicted probability, and a step of determining a next input token by selecting either the output token or one of a predefined special token based on the determined probability of the output token.
[0005] When the special token is determined as the next input token, the reliability of the candidate for the next output token predicted based on the special token is higher than the reliability of the candidate for the next output token predicted based on the output token.
[0006] The step of determining the output token may include selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network.
[0007] An input layer of the artificial neural network may include a number of nodes corresponding to each of the candidates and a special node corresponding to the special token.
[0008] The at least one input token may include at least one of an output token previously predicted by the artificial neural network and the special token.
[0009] Determining the output token may include selecting the most probable token from the candidates.
[0010] The step of determining the next input token may include the steps of comparing the probability of the output token with a predetermined threshold probability, selecting the special token as the next input token if the probability of the output token is less than the threshold probability, and selecting the output token as the next input token if the probability of the output token is greater than the threshold probability.
[0011] Predicting a probability of a candidate for the output token may include predicting a probability of a candidate for the output token based on a correlation between the at least one input token and the output token.
[0012] The method for decoding in an artificial neural network according to an embodiment may further include determining a next output token based on the at least one input token and the next input token.
[0013] In one embodiment, the decoding method in an artificial neural network may further include a step of masking the special token so that tokens other than the special token do not pay attention to the special token.
[0014] In one embodiment, the decoding method in an artificial neural network may further include a step of setting a correlation between a next output token and the special token to a predetermined value or less when the special token is determined as the next input token.
[0015] The step of determining the output token may include the steps of determining whether the output token is identical to the special token, and if the output token is identical to the special token, determining a second most probable token among the candidates as the output token.
[0016] In a decoding method in a first artificial neural network, the decoding method in the artificial neural network includes the steps of predicting a probability of a candidate for an output token based on at least one previously input token, selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network linked to the first artificial neural network and the probability of the candidate predicted by the first artificial neural network, comparing the probability of the output token between an artificial neural network among the at least one second artificial neural network that has determined the highest probability of the output token and the first artificial neural network, and determining a next input token of the first artificial neural network by selecting either the output token or a predefined special token based on the comparison result.
[0017] The step of determining the next input token may include a step of selecting the special token as the next input token of the first artificial neural network if a difference between the probability of the output token of the artificial neural network determining the highest probability of the output token and the probability of the output token of the first artificial neural network is greater than a predetermined threshold, and a step of selecting the output token as the next input token of the first artificial neural network if a difference between the probability of the output token of the artificial neural network determining the highest probability of the output token and the probability of the output token of the first artificial neural network is less than a predetermined threshold.
[0018] A confidence level of the candidate next output token predicted based on the special token is higher than a confidence level of the candidate next output token predicted based on the output token if the confidence level is greater than the threshold value.
[0019] In one embodiment, a speech recognition device includes a processor that predicts a probability of a candidate output token based on at least one previously input token, determines the output token from among the candidates based on the predicted probability, and determines a next input token by selecting either the output token or one of a predefined special token based on the determined probability of the output token.
[0020] A speech recognition device according to one embodiment includes a processor that determines a next input token of the first artificial neural network by predicting a probability of a candidate for an output token based on at least one input token input up to now, selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network linked to a first artificial neural network and the probability of the candidate predicted by the first artificial neural network, comparing the probability of the output token between an artificial neural network among the at least one second artificial neural network that has determined the highest probability of the output token and the first artificial neural network, and selecting one of the output token and a predefined special token based on the comparison result.
[0021] In one embodiment, a speech recognition system includes a memory storing instructions and one or more processors, which, when executed by the processor, cause the processor to extract speech features from input speech, encode the speech features to generate encoded features, determine probabilities of output token candidates based on the encoded features and previously determined recognition results, and determine a next output token based on weights of the output token candidates determined by a language model artificial neural network and weights of the output token candidates determined by a speech recognition artificial neural network. Effect of the Invention
[0022] According to the present invention, a decoding method and device for an artificial neural network can be provided. [Brief description of the drawings]
[0023] [Figure 1] FIG. 2 is a diagram for explaining an operation method of a decoder that performs autoregressive decoding using one artificial neural network according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a method for masking special tokens in an artificial neural network to which an attention mechanism according to an embodiment is applied. [Diagram 3] FIG. 1 is a block diagram illustrating an artificial neural network system with an encoder-decoder structure according to an embodiment. [Figure 4] FIG. 1 is a block diagram showing an artificial neural network system having an encoder-decoder structure using an ensemble approach according to an embodiment. [Diagram 5] FIG. 1 is a block diagram illustrating an application embodiment in a speech recognition artificial neural network with additional language models according to one embodiment. [Figure 6] FIG. 1 illustrates a speech recognition artificial neural network system having an additional language model artificial neural network according to one embodiment. [Figure 7A] FIG. 1 is a diagram showing an embodiment in which an ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder according to an embodiment. [Figure 7B] FIG. 1 is a diagram showing an embodiment in which an ensemble method is applied to a language model artificial neural network decoder using a special token according to an embodiment and a speech recognition artificial neural network decoder. [Figure 8] 1 is a flowchart illustrating a decoding method in an artificial neural network according to an embodiment. [Figure 9] 1 is a flow chart illustrating a method for handling low probability tokens in an autoregressive based sequence generating artificial neural network according to one embodiment. [Figure 10] FIG. 2 is a block diagram of a processing device for low probability tokens in an autoregressive based sequence generating artificial neural network according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] The specific structural or functional description of the embodiment is disclosed for the purpose of illustration only, and may be modified in various forms. Therefore, the embodiment is not limited to the specific disclosed form, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.
[0025] Although terms such as first or second may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be named a second component, and similarly, a second component may be named a first component.
[0026] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.
[0027] The terms used in this specification are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this specification, the terms "include" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.
[0028] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as ideal or overly formal unless expressly defined in this specification.
[0029] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the duplicated description thereof will be omitted.
[0030] FIG. 1 is a diagram for explaining an autoregressive decoding method in an artificial neural network according to an embodiment.
[0031] 1, a decoder 120 according to one embodiment receives as input the encoded features 110. For example, the decoder 120 may receive input from an encoder (not shown) connected to the front end of the decoder 120 in the artificial neural network.
[0032] The encoder and decoder 120 may be a sequence-to-sequence encoder-decoder implemented by a neural network. As an example, the neural network is a deep neural network (DNN), but is not limited thereto. The DNN may include one or more fully connected networks (FCN), deep convolutional neural networks (DCN), recurrent neural networks (RNN), recurrent deep neural networks (RDNN), and bidirectional recurrent neural networks (BDRNN), and the DNN may be configured by combining FCN, DCN, RNN, RDNN, and BDRNN. The nodes in the layers of the neural network are in a nonlinear relationship that affects each other, and the parameters of the neural network, such as the values output from each node and the relationship between the nodes, may be optimized by learning, for example, loss based back propagation.
[0033] The neural network may be a neural network model embodied in a processor, and various processes are realized as a specialized computational structure that can provide an intuitive mapping between input patterns and output patterns, or pattern recognition of input patterns after learning through the neural network model. The learning ability to generate the mapping or perform such exemplary pattern recognition is referred to as the learning ability of the neural network. The learning ability may also be such that the specialized computational structure classifies input patterns or portions of input patterns into members belonging to one or more predetermined groups. Furthermore, the neural network trained through learning has a generalization ability, for example, to generate relatively accurate or reliable outputs for input patterns for which the neural network has not been trained.
[0034] As an example, a sequence-to-sequence encoder-decoder is a network structure in which an encoder and a decoder are integrated, and can generate a sequence of recognition results from an input sequence. For example, the encoder and decoder 120 embodied in the sequence-to-sequence structure directly generates a recognition result corresponding to an input speech from the input speech. Before receiving the input speech, the encoder and decoder 120 are pre-trained to generate a sequence of recognition results from a pre-trained input sequence.
[0035] The encoder encodes an input sequence to generate encoded features 110. The encoder extracts features from the input sequence to generate encoded information. The encoded features 110 are applied to a decoder 120. The decoder 120 generates a recognition result based on the encoded features 110.
[0036] 1, upon receiving the encoded features 110, a decoder 120 determines token 1 (token1) 105 as an output token based on an input token, start token 115. Having determined token 1 (token1) 105 as an output token, the decoder 120 determines token 1 (token1) 105 as the next input token.
[0037] The decoder 120 of the artificial neural network of the encoder-decoder structure determines the output tokens based on the information calculated from the encoder at each step, which may depend on the input tokens selected up to the previous step.
[0038] For example, the decoder 120, having determined that the token 1 (token1) 105 is an input token, predicts the probability of the candidate output token based on the token 1 (token1) 105. 1 or c m is a candidate output token, and p 1 Or p mis the probability of each candidate. Based on the probability, the output token is determined from among the candidates. For example, the token with the highest probability from among the candidates is selected as the output token.
[0039] In step i, token t i The conditional probability of can be expressed as equation (1).
[0040] p(t i |t 1 ,t 2 ,...,t i-1 ,H e ) (1) H e corresponds to the feature 110 encoded in the hidden representation of the encoder. 1 Or t i-1 is the input token selected so far. For ease of explanation, Fig. 1 illustrates the encoded features 110 in the decoder 120 as being applied only to the nodes in the first step, but the encoded features 110 may be applied to other nodes in subsequent steps as well.
[0041] A token selected as an output token in a sequence-to-sequence model that performs general autoregressive decoding is determined as the next input token. Since the output token is determined depending on the input token selected up to the previous step, if the input token has a low probability after being selected as an output token in the sequence-to-sequence model, the performance of predicting the next token by the selected output token decreases.
[0042] For example, the output token candidate c 1 or c m The least likely of these is c i may be determined as the output token. If the next token must be generated based on a token with low probability in the autoregressive decoding process due to the feature of generating the next output based on the previous token, that is, if the probability of the previous token is low, it is difficult to generate subsequent tokens smoothly. For example,1 or c m Of these, c i is most likely, but c 1 or c m There may be cases where the probabilities of all of the NNs are not high. Or, in a model ensemble where one of the tokens generated by the various NN models is selected and used, the next token must be generated based on the tokens generated by the other NNs as a result of the ensemble.
[0043] The artificial neural network is i-1 If the probability of is low, p(t i |t 1 ,t 2 ,...,t i-1 ,H e ) to generate the output token t i It becomes impossible to predict.
[0044] Input token t with low probability i-1 Even if the output token t i To predict token t smoothly, we use the conditional probability formula (1) i-1 It may be possible to solve this problem by calculating the probability excluding . However, since the system is designed to learn from the training data with the above conditional probability, there is a concern that calculating the conditional probability independently of the previous token will defeat the original purpose.
[0045] According to one embodiment of the decoding method, a special token t NC Introduce input token t i-1 Even if the probability of t is low, i can be predicted satisfactorily.
[0046] Referring to FIG. 1, the output token c i p is the probability i , a predetermined threshold probability θ nc¬Compare with the threshold probability θ nc¬ is a predetermined value, which is the minimum value of the input token probability that can obtain the desired output. For example, i p is the probability i is the threshold probability θ nc¬ If it is greater than c, output token c i Since it is determined that the desired output can be obtained when determining the next input token as the output token c i can be determined as the next input token.
[0047] Output token c i p is the probability i is the threshold probability θ nc¬ If it is less than c, output token c i When determining that the next input token is c, it is determined that the desired output will not be obtained, so the output token c i It is difficult to determine the next input token as the next token. In this case, if the output token c i The probability of p i is the threshold probability θ nc¬ If it is less than c, output token c i Instead, a predefined special token NC is determined as the next input token.
[0048] The reliability of the candidate of the next output token predicted based on the special token is higher than the reliability of the candidate of the next output token predicted based on the output token whose probability is lower than the threshold probability. The artificial neural network may be trained in advance to satisfy such a characteristic. The specific contents of training the artificial neural network will be described later. For example, i p is the probability i is the threshold probability θ nc¬ If ,c,, the confidence of the candidate for the next output token predicted based on the special token NC is c i , which is higher than the confidence of the candidate next output token predicted based on
[0049] The confidence level is the confidence level for a token, and means the degree to which it is close to the correct output token of the training data. For example, c i p is the probability i is the threshold probability θ nc¬ If c is smaller than , the most probable output token among the candidates for the next output token predicted based on the special token NC is c i The next output token is closer to the correct output token than the most probable output token among the candidates for the next output token predicted based on
[0050] If the artificial neural network determines the next input token by selecting one of the output token and the special token, the artificial neural network can more appropriately determine the next output token based on at least one input token and the next input token. For example, if the artificial neural network determines the special token NC as the next input token, the artificial neural network can determine the next output token, token 3, based on the input token, token 1, and the next input token, special token NC.
[0051] FIG. 2 is a diagram illustrating a method for masking special tokens in an artificial neural network to which an attention mechanism according to an embodiment is applied.
[0052] 2, in one embodiment, the decoder 220 predicts the probability of a candidate output token based on the correlation between at least one input token and the output token, and the output token can predict the probability of a candidate output token with different weights depending on the input token.
[0053] For example, the artificial neural network may be an artificial neural network to which an attention mechanism is applied, where the artificial neural network can be trained to better represent the relationships between tokens that it has predicted so far.
[0054] To prevent the special token from affecting other tokens, the special token is masked to force other tokens not to be attended to by the special token.
[0055] For example, in the case of an output token corresponding to the input token "is", attention is paid not only to the input tokens "Hi", "this", and "is", but also to the special token NC. Since the special token must not affect other tokens, the output token corresponding to the input token "is" is forced to not pay attention to the special token NC by masking the special token.
[0056] Hereinafter, the encoder-decoder structured artificial neural network system will be described in detail with reference to FIGS.
[0057] FIG. 3 is a block diagram illustrating an artificial neural network system in an encoder-decoder configuration according to one embodiment.
[0058] 3, an embodiment of an encoder-decoder structured artificial neural network system includes an artificial neural network 310 including an encoder 320 and a decoder 330. The encoder-decoder structured artificial neural network includes a pre-processor 340, an attention masking calculation unit 350, and a token modification unit 360. The decoder 330 shown in FIG. 3 corresponds to the decoder 220 described above with reference to FIGS. 1 and 2.
[0059] In one embodiment, the encoder-decoder structure of the artificial neural network system processes tokens with low probability in one artificial neural network 310 .
[0060] The sequence-to-sequence encoder 320-decoder 330 is a network structure in which the encoder 320 and the decoder 330 are integrated, and generates a recognition result sequence from an input sequence.
[0061] The preprocessor 340 is a module that preprocesses the input of the artificial neural network, for example, by removing noise contained in the input signal and processing the input signal into a suitable form for input to the artificial neural network.
[0062] At run time, the token corrector 360 checks whether the probability of the predicted output token exceeds a threshold probability θ NC The output tokens can be modified to predict the next output token by substituting a special token for an output token lower than .
[0063] In addition, the token modifying device 360 judges whether the output token is the same as the special token. If the output token is the same as the special token, the token with the second highest probability among the candidates is determined as the output token. This prevents the special token from being output.
[0064] The attention masking calculation unit 350 masks the attention weights so that other tokens do not have a conditional probability on the special token, so that other tokens do not pay attention to the special token. The attention masking calculation unit 350 can operate in the same manner when performing learning.
[0065] Although Fig. 3 has been described based on limited drawings as an example for explaining the invention, a person having ordinary skill in the art can apply various technical modifications and variations based on Fig. 3. For example, the decoder 330 is a broad concept including an attention masking calculation device 350 and a token modification device 360.
[0066] FIG. 4 is a block diagram illustrating an artificial neural network system having an encoder-decoder structure using an ensemble method according to an embodiment.
[0067] Referring to FIG. 4, an artificial neural network system according to one embodiment includes a number of artificial neural networks 410, 420, and 430.
[0068] In an artificial neural network system with an encoder-decoder structure using an ensemble method, the number of encoders corresponding to the m decoders is not necessarily m. There may be a decoder without an encoder. For example, the artificial neural network 420, 430 may have only the decoders 422, 432, and may not have the encoders 421, 431.
[0069] According to the ensemble approach, an output token can be selected from output token candidates based on a combination of probabilities of the output token candidates predicted by multiple decoders 412, 422, 432 using multiple artificial neural networks 410, 420, 430.
[0070] When using the ensemble approach, each of the decoders 412, 422, and 432 determines a different output token as the most probable. For example, the decoder 412 of the artificial neural network 410 determines that c1 is the most probable output token, the decoder 422 of the artificial neural network 420 determines that c2 is the most probable output token, and the decoder 432 of the artificial neural network 430 determines that c3 is the most probable output token.
[0071] One output token may be determined even if each of the decoders 412, 422, and 432 determines a different output token as the most probable one. For example, an output token may be selected from among the output token candidates based on a combination of the probabilities of the output token candidates predicted by the multiple decoders 412, 422, and 432.
[0072] If the selected output token is forced to be input as the next input token to all decoders, there is a possibility that some decoders may input as their next input token an output token that is not judged to have the highest probability.
[0073] For example, when multiple decoders 412, 422, 432 select c1 as the output token based on the combination of probabilities of output token candidates c1, c2, c3 predicted by them, the decoders 422 and 432 may not be able to generate the next output token based on the most probable output token candidates c2 and c3 according to their own criteria, and may have to generate the next token based on the output token candidate c1 generated by the other decoder 412. Due to the autoregressive nature of finding the next token based on the previous token, the artificial neural network 420, 430 that inputs the next input token c1 with low probability in its own decoder 422, 432 may not be able to smoothly predict the next output token.
[0074] In one embodiment of the ensemble-based encoder-decoder artificial neural network system, a special token that cannot affect the conditional probability is introduced, and the system can correctly predict the output token by inputting a token that the system does not determine to be the most probable.
[0075] In the case of ensembling multiple artificial neural networks, the output tokens determined for each j-th artificial neural network are max Probability P j (t max ) and t max For example, among the output token candidates c1, c2, and c3, the probability p2(c1) for c1 determined as the output token in the artificial neural network 420 is compared with the probability p1(c1) for c1 in the artificial neural network 410 that determined c1 as the highest.
[0076] The difference between the output token probability of the artificial neural network 410 that determined the highest output token probability and the output token probability of the jth artificial neural network
[0077]
number
[0078] The difference between the output token probability of the artificial neural network with the highest output token probability and the output token probability of the jth artificial neural network
[0079]
number
[0080] The method of masking special tokens described with reference to FIG. 2 is similarly applicable to an artificial neural network system having an encoder-decoder structure using an ensemble method as shown in FIG.
[0081] At run time, the token corrector 460 checks whether the probability of the predicted output token exceeds a threshold probability θ NC The output tokens can be modified to predict the next output token by substituting a special token for an output token lower than .
[0082] The token corrector 460, at run time, calculates the difference between the output token probability of the jth artificial neural network and the output token probability of the artificial neural network 410 that has determined the highest output token probability.
[0083]
number
[0084] In addition, the token modifying device 460 determines whether the output token is the same as the special token. If the output token is the same as the special token, the token with the second highest probability among the candidates is determined as the output token. This prevents the special token from being output.
[0085] The attention masking calculator 450 masks the attention weights so that other tokens cannot be paid attention to the special tokens, so that other tokens do not have a conditional probability on the special tokens. The attention masking calculator 450 is operated the same not only during learning, but also during execution-inference or implementation.
[0086] 5 is a block diagram illustrating an application embodiment in a speech recognition artificial neural network with an additional language model artificial neural network according to one embodiment. The speech recognition artificial neural network is implemented as an example of a speech recognition device. The speech recognition device can collect speech or input from a user using a receiver or sensor (e.g., a microphone) that is a component of the speech recognition device, and can receive a voice signal from the receiver, which may be separate or external to the speech recognition device.
[0087] The voice recognition device may be an electronic device used by an individual user, for example a wearable device such as a smart phone, a smart pad, a smart band, a personal digital assistant (PDA), a laptop, a component therein or a standalone device related thereto. As a different example, the voice recognition device includes electronic devices shared by multiple users, for example a voice recognition speaker and a voice recognition TV.
[0088] Referring to FIG. 5, a speech recognition artificial neural network having a language model artificial neural network according to an embodiment includes a speech recognition model artificial neural network 510 and a language model artificial neural network 520.
[0089] In the case of speech recognition, the basic system is composed of an encoder 511 that receives speech and a decoder 512 that outputs a sequence of words that is the recognition result. In addition, the performance can be improved by further utilizing a language model artificial neural network 520 formed by a decoder 521 that determines how likely a sequence of words is to occur in a general sentence.
[0090] A speech recognition model artificial neural network that predicts words at each point in time based on actual speech creates new words by listening to speech and combining phonemes according to pronunciation, but in the case of a language model artificial neural network trained on general sentences, when a new word that was not learned during training (such as a new place or a neologism) appears, the output probability of that word is extremely low. When trying to predict the next word using a word with such a low probability as input, the language model artificial neural network has to output the next word from an input that it has not learned, making it difficult to make a correct prediction.
[0091] In one embodiment, the language model artificial neural network 520 also uses a token modification device 560 and an attention masking calculation device 550 to introduce a special token that cannot affect the conditional probability described above with reference to FIG. 4, and can correctly predict an output token by inputting a token that the network does not determine to have the highest probability.
[0092] Hereinafter, an embodiment of an application in a speech recognition artificial neural network having an additional language model artificial neural network will be described in detail with reference to FIGS. 6 to 7B.
[0093] FIG. 6 illustrates a speech recognition artificial neural network system having an additional language model artificial neural network according to one embodiment.
[0094] Referring to FIG. 6, a speech recognition artificial neural network system with an additional language model artificial neural network according to one embodiment includes an encoder 610, a speech recognition artificial neural network decoder 620, and a language model artificial neural network decoder 630.
[0095] The speech recognition artificial neural network system with an additive language model artificial neural network extracts speech features 640 from an input speech. The input speech is a speech signal including information for each of a plurality of frames, and the speech features 640 are a sequence of information extracted at least in units of one frame, and are expressed as a multi-dimensional vector. For convenience of explanation, the following embodiment will be described in which the input speech is "Hi MinChul this is".
[0096] A speech recognition artificial neural network system with an additional language model artificial neural network generates a sequence of recognition results from a sequence of input speech using an ensemble of the language model artificial neural network decoder 630 and the speech recognition artificial neural network decoder 620. An "ensemble" refers to individual artificial neural network modules considered together. For example, an ensemble can be a combination of the language model artificial neural network decoder 630 and the speech recognition artificial neural network decoder 620.
[0097] The language model artificial neural network decoder 630 and the speech recognition artificial neural network decoder 620 output the respective recognition results in units of tokens, and generate the final recognition result by ensembling the respective recognition results according to the ensemble weighting value. For example, the speech recognition artificial neural network decoder 620 determines output token candidates based on the input speech and the previously determined recognition results, and the language model artificial neural network decoder 630 determines output token candidates based on the previously determined recognition results, and generates the final recognition result by ensembling the respective output token candidates according to the ensemble weighting value (weighted sum).
[0098] The encoder 610 and decoder 620 are pre-trained to generate a sequence of recognition results from a sequence of ground truth text pairs corresponding to input speech, while the language model artificial neural network decoder 630 is pre-trained to generate a sequence of recognition results from any text sequence.
[0099] The encoder 610 encodes the speech features 640 to generate encoded features 650. The encoder 610 transforms the dimensions of the speech features 640 to generate encoded information. The encoded features 650 are applied to the speech recognition artificial neural network decoder 620. The speech recognition artificial neural network decoder 620 generates output token candidates based on the encoded features 650 and previously determined recognition results on a token-by-token basis, and the language model artificial neural network decoder 630 generates output token candidates based on previously determined recognition results on a token-by-token basis. The two recognition results can be ensembled according to a predetermined ensemble weight value to generate a final recognition result. For example, the ensemble weight value of the speech recognition artificial neural network decoder 620 and the language model artificial neural network decoder 630 can be 1:0.2.
[0100] Hereinafter, with reference to Figs. 7A and 7B, an embodiment in which the ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder will be described in detail.
[0101] FIG. 7A is a diagram showing an embodiment in which an ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder according to an embodiment.
[0102] 7A, a speech recognition artificial neural network decoder 710 and a language model artificial neural network decoder 720 according to an embodiment correspond to the speech recognition artificial neural network decoder 620 and the language model artificial neural network decoder 630 shown in FIG. 6, respectively.
[0103] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 predict the probability of a candidate output token based on the input token "Hi" as an autoregressive decoder. For example, the speech recognition artificial neural network decoder 710 outputs "MinChul" and "Bixby" as candidates for the output token based on the input token "Hi" and the encoded speech features, and predicts the respective probabilities to be 0.7 and 0.1. Also, the language model artificial neural network decoder 720 outputs "MinChul" and "Bixby" as candidates for the output token based on the input token "Hi", and predicts the respective probabilities to be 0.001 and 0.8.
[0104] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 use the ensemble weights to determine the output token from the candidates. For example, "MinChul" has a final weight of 0.7+0.2×0.001=0.7002, and "Bixby" has a final weight of 0.1+0.2×0.8=0.26, so "MinChul" is determined as the output token.
[0105] In an autoregressive decoding environment, the language model artificial neural network decoder 720 inputs "MinChul", which is determined as the result of the ensemble other than "Bixby" that has the highest probability as its own criterion, as the next input token. Since "MinChul" has a low probability in the language model artificial neural network decoder 720, it is difficult to smoothly predict the next output token.
[0106] For example, based on the next input token "MinChul", the language model artificial neural network decoder 720 outputs "this" and "dis" as candidates for the next output token and predicts their respective probabilities as 0.01 and 0.01. Based on the next input token "MinChul", the speech recognition artificial neural network decoder 710 outputs "this" and "dis" as candidates for the next output token and predicts their respective probabilities as 0.34 and 0.38. Because "this" and "dis" are pronounced similarly, the speech recognition artificial neural network decoder 710 may predict the probabilities of "this" and "dis" to be similar.
[0107] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 use the ensemble weights to determine the next output token from the candidates. For example, "this" has a final weight of 0.34+0.2×0.01=0.342, and "dis" has a final weight of 0.38+0.2×0.01=0.382, so "dis" is determined as the next output token. As a result, the speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 according to FIG. 7A output "dis", which is different from the input speech "this".
[0108] In the above embodiment, the probability of "MinChul" determined by the ensemble between the speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 among the candidates of the current output token predicted by the language model artificial neural network decoder 720 is lower than a predetermined threshold probability. Due to the low probability of "MinChul", the language model artificial neural network decoder 720 cannot distinguish "this" and "dis" well from the next output token. As a result, as the performance of the language model artificial neural network decoder 720, which should complement the performance of the speech model artificial neural network decoder 710, decreases, an erroneous result such as outputting "dis" different from "this" as in the above embodiment is derived.
[0109] Hereinafter, an embodiment of the language model artificial neural network decoder using a special token and an ensemble method in the speech recognition artificial neural network decoder will be described in detail with reference to FIG. 7B.
[0110] FIG. 7B is a diagram showing an embodiment in which an ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder using a special token according to an embodiment.
[0111] Referring to FIG. 7B, the speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 according to an embodiment use the ensemble weights to determine "MinChul" as an output token from among the candidates.
[0112] The language model artificial neural network decoder 720 compares the probability of "MinChul" with a predetermined threshold probability. For example, the predetermined threshold probability may be 0.01. The language model artificial neural network decoder 720 determines that the probability of "MinChul" is 0.001, which is smaller than the threshold probability 0.01, and therefore selects the special token NC as the next input token instead of "MinChul".
[0113] Based on the next input token, the special token NC, the language model artificial neural network decoder 720 outputs "this" and "dis" as candidates for the next output token, and predicts their respective probabilities as 0.4 and 0.1. Based on the next input token "MinChul", the speech recognition artificial neural network decoder 710 outputs "this" and "dis" as candidates for the next output token, and predicts their respective probabilities as 0.34 and 0.38.
[0114] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 use the ensemble weights to determine the next output token from among the candidates. For example, "this" has a final weight of 0.34+0.2×0.4=0.42, and "dis" has a final weight of 0.38+0.2×0.1=0.4, so "this" is determined as the next output token.
[0115] FIG. 8 is a flowchart showing a decoding method in an artificial neural network according to an embodiment. The operations shown in FIG. 8 are performed in the order and manner shown, but the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the embodiment shown. A plurality of operations shown in FIG. 8 may be performed in parallel or simultaneously. One or more blocks and combinations of blocks shown in FIG. 8 are realized by a computer based on special-purpose hardware performing a specific function, or a combination of special-purpose hardware and computer instructions. In addition to the description shown in FIG. 8 below, the description of FIG. 1 to FIG. 7 is applicable to FIG. 8, so the description of the overlapping contents will be omitted.
[0116] Referring to Figure 8, steps S810 to S830 according to one embodiment are performed by the decoder 330 described above with reference to Figure 3. The decoder 330 may be implemented by one or more hardware components, or one or more components including hardware and software.
[0117] In step S810, the decoder 330 predicts the probability of a candidate output token based on at least one input token input up to the present time.
[0118] In step S820, the decoder 330 determines which of the candidates to output based on probability, selects which of the candidates has the highest probability, or selects which of the candidates to output based on a combination of the probability of the candidate predicted by a second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network.
[0119] In step S830, the decoder 330 determines the next input token by selecting either the output token or one of the predefined special tokens based on the probability of the output token.
[0120] The at least one input token includes at least one of an output token and a special token previously predicted by the artificial neural network.
[0121] The input layer of the artificial neural network includes a number of nodes corresponding to candidates and special nodes corresponding to special tokens.
[0122] FIG. 9 is a flow chart for illustrating a method for processing low probability tokens in an autoregressive-based sequence generating artificial neural network according to an embodiment. The operations shown in FIG. 9 are performed in the order and manner shown, but the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the illustrated embodiment. A plurality of operations shown in FIG. 9 may be performed in parallel or simultaneously. One or more blocks and combinations of blocks shown in FIG. 9 are realized by a computer based on special purpose hardware performing a specific function, or a combination of special purpose hardware and computer instructions. In addition to the following description shown in FIG. 9, the description of FIG. 1 to FIG. 8 is applicable to FIG. 9, so the description of the overlapping contents will be omitted.
[0123] 9, in step S901, an autoregressive sequence generating artificial neural network according to an embodiment receives an input, for example, an input speech in a speech recognition artificial neural network system.
[0124] In step S902, the encoder generates coded features from the received input, for example, by transforming the dimensions of the audio features to generate coded information.
[0125] In step S903, the decoder predicts the probability of the candidate output token based on the encoded features. At each step, the decoder determines the output token based on the information calculated from the encoder, which may depend on the input token selected up to the previous step.
[0126] In step S904, the decoder determines the output token from among the candidates based on the probability. For example, the most probable t i (top likelihood token) is determined as the output token.
[0127] In step S905, the decoder determines whether the output token is identical to a special token.
[0128] In step S906, if the output token is the same as the special token, the decoder determines the second most probable token among the candidates as the output token, thereby preventing the special token from being output.
[0129] In step S907, if the output token is not identical to the special token, the decoder compares the probability of the output token with a predetermined threshold probability.
[0130] In step S908, if the probability of the output token is greater than the threshold probability, the decoder selects the output token as the next input token.
[0131] In step S909, if the probability of the output token is less than the threshold probability, the decoder selects a special token as the next input token instead of the output token.
[0132] In step S910, the decoder masks the special token to force other tokens not to pay attention to the special token, to prevent the special token from affecting different tokens.
[0133] A special token t that cannot affect the conditional probability NC Introduce input token t i-1 Even if the probability of t is low, i The special token is trained to have the characteristics shown in formula (2).
[0134]
number
[0135] In other words, when the auxiliary artificial neural network is classified, the correct token is identified with a probability θ Replace For example, in the case of a speech recognition artificial neural network system having an additional language model artificial neural network, the language model artificial neural network may be an auxiliary artificial neural network, and in this case, the correct token can be identified with a probability θ Replace can be changed to and trained.
[0136] FIG. 10 is a block diagram of a processing device for low probability tokens in an autoregressive based sequence generating artificial neural network according to one embodiment.
[0137] 10, an apparatus 1000 for processing low probability tokens in an autoregression-based sequence generating artificial neural network according to an embodiment includes a sensor 1010, a processor 1030, and a communication interface 1070. The apparatus 1000 for processing low probability tokens in an autoregression-based sequence generating artificial neural network further includes a memory 1050, and a display 1090. The sensor 1010, the processor 1030, the memory 1050, the communication interface 1070, and the display 1090 communicate via a communication bus 1005.
[0138] The sensor 1010 includes, for example, a microphone sensor and an audio sensor.
[0139] The processor 1030 may perform at least one of the methods or an algorithm corresponding to at least one of the methods described above with reference to Figures 1 to 9. The processor 1030 executes a program to control the device 1000 for processing low probability tokens in an autoregressive based sequence generating artificial neural network. The program code executed by the processor 1030 is stored in the memory 1050.
[0140] The processor 1030 is configured, for example, by a central processing unit (CPU) or a graphics processing unit (GPU).
[0141] The memory 1050 stores data processed by the processor 1030. For example, the memory 1050 stores a program. The stored program may be a set of syntax that is coded to assist the user in walking and is executable by the processor 1030. The memory 1050 may be a volatile memory or a non-volatile memory.
[0142] The communication interface 1070 is connected to the multiple sensors 1010, the processor 1030, and the memory 1050 to transmit and receive data. The communication interface 1070 is connected to other external devices to transmit and receive data. Hereinafter, the expression "transmitting and receiving "A" indicates transmitting and receiving "information or data indicating A".
[0143] The communication interface 1070 is realized by a circuit network in the processing device 1000 for low probability tokens in an autoregressive-based sequence generating artificial neural network. For example, the communication interface 1070 includes an internal bus and an external bus. In another example, the communication interface 1070 may be an element that connects the processing device 1000 for low probability tokens in an autoregressive-based sequence generating artificial neural network with an external device. The communication interface 1070 receives data from an external device and transmits data to the processor 1030 and the memory 1050.
[0144] The display 1090 displays the decoded result. For example, the translation result and the voice recognition result are displayed on the display 1090.
[0145] The pre-processing processors 340, 440, 441, 442, attention masking calculation devices 350, 450, 550, encoders 320, 411, 421, 431, 511, decoders 330, 412, 422, 432, 512, 521, token modification devices 360, 460, 560, sensor 1010, processor 1030, memory 1050, communication interface 1070, and display 1090 described above with reference to Figures 1 to 10 may be embodied as hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general purpose or special purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA), programmable logic unit (PLU), microprocessor, or other device that executes and responds to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, the processing device may be described as being one in which only one processing device is used, but one skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.
[0146] The software may include computer programs, codes, instructions, or any combination of one or more thereof, capable of configuring or instructing a processing device to operate as desired, either independently or in combination. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, either permanently or temporarily, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across computer systems coupled to a network, and may be stored and executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.
[0147] The method according to the present invention may be embodied in the form of program instructions to be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and the program instructions may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DYIJDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.
[0148] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted with other components or equivalents to achieve appropriate results.
[0149] Therefore, the scope of the present invention should not be limited to the disclosed embodiment, but should be determined by the appended claims and their equivalents.
Claims
1. A decoding method in a first artificial neural network, predicting probabilities of candidate output tokens based on at least one previously input token; determining the output token from among the candidates based on the predicted probabilities; determining a next input token by selecting one of the output token and a predefined special token based on the determined probability of the output token, where if the probability of the output token is less than a predefined threshold probability, the special token is selected as the next input token, and if the probability of the output token is greater than the threshold probability, the output token is selected as the next input token; A decoding method in an artificial neural network comprising:
2. When the special token is determined as the next input token, 2. The method of claim 1, wherein the reliability of the next output token candidate predicted based on the special token is higher than the reliability of the next output token candidate predicted based on the output token.
3. 2. The method of claim 1, wherein determining the output token comprises selecting the output token from among the candidates based on a combination of probabilities of the candidates predicted by at least one second artificial neural network in conjunction with the first artificial neural network and probabilities of the candidates predicted by the first artificial neural network.
4. A decoding method in an artificial neural network according to any one of claims 1 to 3, wherein an input layer of the artificial neural network includes a plurality of nodes corresponding to each of the candidates and a special node corresponding to the special token.
5. A method for decoding in an artificial neural network according to any one of claims 1 to 3, wherein said at least one input token comprises at least one of an output token previously predicted by said artificial neural network and said special token.
6. 2. A method for decoding in an artificial neural network as claimed in claim 1, wherein said step of determining an output token comprises the step of selecting the most probable token from among said candidates.
7. The step of determining a next input token comprises: comparing the probability of the output token with the predetermined threshold probability.
2. The method of decoding in an artificial neural network of claim 1, further comprising:
8. 2. The method of decoding in an artificial neural network of claim 1, wherein predicting the probability of a candidate for the output token comprises predicting the probability of a candidate for the output token based on a correlation between the at least one input token and the output token.
9. The method of decoding in an artificial neural network according to any one of claims 1 to 8, further comprising the step of determining a next output token based on said at least one input token and said next input token.
10. A method for decoding in an artificial neural network according to any one of claims 1 to 9, further comprising a step of masking the special token so that tokens other than the special token do not pay attention to the special token.
11. A decoding method in an artificial neural network as described in any one of claims 1 to 10, further comprising a step of setting a correlation between the next output token and the special token to a predetermined value or less when the special token is determined as the next input token.
12. The step of determining an output token comprises: determining whether the output token is identical to the special token; if the output token is identical to the special token, determining the second most probable token among the candidates as the output token; 2. A method for decoding in an artificial neural network according to claim 1, comprising:
13. A computer program product that causes a computer to execute the decoding method according to any one of claims 1 to 12.
14. A speech recognition device in an artificial neural network, comprising: predicting probabilities of candidate output tokens based on at least one previously input token; determining the output token from among the candidates based on the predicted probabilities; a processor for determining a next input token by selecting one of the output token and a predefined special token based on the determined probability of the output token, wherein if the probability of the output token is less than a predefined threshold probability, the processor selects the special token as the next input token, and if the probability of the output token is greater than the threshold probability, the processor selects the output token as the next input token; A voice recognition device comprising:
15. When the special token is determined as the next input token, The speech recognition apparatus according to claim 14 , wherein a reliability of a candidate for a next output token predicted based on the special token is higher than a reliability of a candidate for a next output token predicted based on the output token.
16. 16. The speech recognition device according to claim 14, wherein an input layer of the artificial neural network includes a plurality of nodes corresponding to each of the candidates and a special node corresponding to the special token.
17. 17. A speech recognition apparatus according to any one of claims 14 to 16, wherein the at least one input token comprises at least one of an output token previously predicted by the artificial neural network and the special token.
18. 15. The speech recognition apparatus of claim 14, wherein the processor selects the token with the highest probability from among the candidates.
19. 15. The speech recognition apparatus of claim 14, wherein the processor selects the output token from among the candidates based on a combination of probabilities of the candidates predicted by the artificial neural network and probabilities of the candidates predicted by other artificial neural networks in conjunction with the artificial neural network.
20. The processor, 15. The speech recognition apparatus of claim 14, further comprising: comparing the probability of the output token to the predetermined threshold probability before determining the next input token.
21. A speech recognition apparatus according to any one of claims 14 to 20, wherein the processor is adapted to predict a candidate probability of the output token based on a correlation between the at least one input token and the output token.
22. A speech recognition apparatus according to any one of claims 14 to 21, wherein the processor is adapted to determine a next output token based on the at least one input token and the next input token.
23. A speech recognition apparatus according to any one of claims 14 to 22, wherein the processor is operable to mask the special token so that tokens other than the special token do not draw attention to the special token.
24. 15. The speech recognition apparatus of claim 14, wherein the processor, when the special token is determined as the next input token, sets a correlation between a next output token and the special token to a predetermined value or less.
25. The processor, determining whether the output token is identical to the special token; 15. The speech recognition apparatus of claim 14, further comprising: determining a second most probable token among said candidates as said output token if said output token is identical to said special token.
Citation Information
Patent Citations
Pointer sentinel mixture architecture
WO2018057809A1
Speech recognition system
WO2018168509A1
