Decoding method in artificial neural network, speech recognition device and speech recognition system

Abstract

To provide a method and device for processing low probability tokens in a sequence generation artificial neural network based on self-regression.SOLUTION: A decoder in an artificial neural network predicts probability of a candidate output tokens based on at least one previously inputted input token, and then determines the output token from among the candidates based on the predicted probability, and determines the next input token by selecting either the output token or a predefined special token based on the determined probability of the output token.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION An embodiment of the present invention relates to a decoding method and apparatus in an artificial neural network. [Background technology]

[0002] To solve the problem of receiving a sequence of indefinite length as input to an artificial neural network and generating an output of indefinite length, an encoder-decoder artificial neural network model, which is one of the sequence-to-sequence models, has been introduced. In particular, the method of inputting the previous output of the artificial neural network for each token, which is the unit that makes up a sequence, and outputting the next output in succession to calculate an output of indefinite length in an artificial neural network is called auto-regressive decoding. Summary of the Invention [Problem to be solved by the invention]

[0003] SUMMARY OF THE INVENTION An object of the present invention is to provide a decoding method and apparatus in an artificial neural network. [Means for solving the problem]

[0004] In one embodiment, a decoding method in a first artificial neural network includes the steps of predicting the probability of a candidate output token based on at least one previously input token, determining the output token from the candidates based on the predicted probability, and determining a next input token by selecting either the output token or a predefined special token based on the determined probability of the output token.

[0005] When the special token is determined as the next input token, the reliability of the candidate for the next output token predicted based on the special token is higher than the reliability of the candidate for the next output token predicted based on the output token.

[0006] The step of determining the output token may select the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network.

[0007] The input layer of the artificial neural network may include a plurality of nodes corresponding to each of the candidates and a special node corresponding to the special token.

[0008] The at least one input token may include at least one of an output token previously predicted by the artificial neural network and the special token.

[0009] Determining the output token may include selecting the most probable token from the candidates.

[0010] The step of determining the next input token may include the steps of comparing the probability of the output token with a predetermined threshold probability, selecting the special token as the next input token if the probability of the output token is less than the threshold probability, and selecting the output token as the next input token if the probability of the output token is greater than the threshold probability.

[0011] Predicting the candidate probabilities of the output tokens may include predicting the candidate probabilities of the output tokens based on a correlation between the at least one input token and the output tokens.

[0012] According to an embodiment, the method for decoding in an artificial neural network may further include determining a next output token based on the at least one input token and the next input token.

[0013] The decoding method in an artificial neural network according to one embodiment may further include a step of masking the special token so that tokens other than the special token do not pay attention to the special token.

[0014] In one embodiment, the decoding method in an artificial neural network may further include a step of setting a correlation between the next output token and the special token to a predetermined value or less when the special token is determined as the next input token.

[0015] The step of determining the output token may include the steps of determining whether the output token is identical to the special token, and if the output token is identical to the special token, determining the second most probable token from the candidates as the output token.

[0016] A decoding method in a first artificial neural network includes the steps of predicting the probability of a candidate output token based on at least one previously input token; selecting the output token from the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network linked with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network; comparing the probability of the output token between the first artificial neural network and an artificial neural network among the at least one second artificial neural network that has determined the highest probability of the output token; and determining the next input token of the first artificial neural network by selecting either the output token or a predefined special token based on the comparison result.

[0017] The step of determining the next input token may include a step of selecting the special token as the next input token of the first artificial neural network if the difference between the probability of the output token of the artificial neural network that determined the highest probability of the output token and the probability of the output token of the first artificial neural network is greater than a predetermined threshold, and a step of selecting the output token as the next input token of the first artificial neural network if the difference between the probability of the output token of the artificial neural network that determined the highest probability of the output token and the probability of the output token of the first artificial neural network is less than a predetermined threshold.

[0018] If the confidence of the candidate next output token predicted based on the special token is greater than the threshold, the confidence is higher than the confidence of the candidate next output token predicted based on the output token.

[0019] In one embodiment, a speech recognition device includes a processor that predicts the probability of a candidate output token based on at least one previously input token, determines the output token from among the candidates based on the predicted probability, and determines a next input token by selecting either the output token or one of a predefined special token based on the determined probability of the output token.

[0020] According to one embodiment, a speech recognition device includes a processor that predicts the probability of a candidate output token based on at least one input token input up to now, selects the output token from the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network linked to a first artificial neural network and the probability of the candidate predicted by the first artificial neural network, compares the probability of the output token between the first artificial neural network and an artificial neural network that determines the highest probability of the output token among the at least one second artificial neural network, and determines the next input token of the first artificial neural network by selecting either the output token or a predefined special token based on the comparison result.

[0021] According to one embodiment, a speech recognition system includes a memory for storing instructions and one or more processors, which, when executed by the processor, cause the processor to extract speech features from input speech, encode the speech features to generate encoded features, determine probabilities of candidate output tokens based on the encoded features and previously determined recognition results, and determine a next output token based on weights of the candidate output tokens determined by a language model artificial neural network and weights of the candidate output tokens determined by a speech recognition artificial neural network. [Effects of the Invention]

[0022] According to the present invention, a decoding method and device for an artificial neural network can be provided. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 2 is a diagram illustrating an operation method of a decoder that performs autoregressive decoding using one artificial neural network according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a method for masking special tokens in an artificial neural network to which an attention mechanism is applied according to an embodiment. [Figure 3] FIG. 1 is a block diagram illustrating an artificial neural network system with an encoder-decoder structure according to one embodiment. [Figure 4] FIG. 1 is a block diagram illustrating an artificial neural network system having an encoder-decoder structure using an ensemble method according to one embodiment. [Figure 5] FIG. 1 is a block diagram illustrating an application embodiment in a speech recognition artificial neural network with additional language models according to one embodiment. [Figure 6] FIG. 1 illustrates a speech recognition artificial neural network system with an additional language model artificial neural network according to one embodiment. [Figure 7A] FIG. 1 is a diagram illustrating an embodiment in which an ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder according to an embodiment. [Figure 7B] FIG. 1 is a diagram illustrating an embodiment in which an ensemble method is applied to a language model artificial neural network decoder using special tokens and a speech recognition artificial neural network decoder according to an embodiment. [Figure 8] 1 is a flowchart illustrating a decoding method in an artificial neural network according to an embodiment. [Figure 9] 1 is a flow chart illustrating a method for handling low probability tokens in an autoregressive-based sequence-generating artificial neural network according to one embodiment. [Figure 10] FIG. 1 is a block diagram of an apparatus for processing low probability tokens in an autoregression-based sequence-generating artificial neural network according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0025] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed only to distinguish one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component.

[0026] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0027] The terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0028] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0029] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components will be given the same reference numerals regardless of the drawing number, and redundant description thereof will be omitted.

[0030] FIG. 1 is a diagram illustrating an autoregressive decoding method using an artificial neural network according to an embodiment.

[0031] 1, one embodiment of a decoder 120 receives as input the encoded features 110. For example, the decoder 120 may receive input from an encoder (not shown) connected to the front end of the decoder 120 in an artificial neural network.

[0032] The encoder and decoder 120 may be a sequence-to-sequence encoder-decoder implemented as a neural network. An example of the neural network is a deep neural network (DNN), but is not limited to this. The DNN may include one or more of a fully connected network (FCN), a deep convolutional neural network (DCN), a recurrent neural network (RNN), a recurrent deep neural network (RDNN), and a bidirectional recurrent neural network (BDRNN). The DNN may be configured by combining an FCN, a DCN, an RNN, an RDNN, and a BDRNN. Nodes in a layer of a neural network are in a nonlinear and mutually influential relationship. Neural network parameters, such as the values ​​output from each node and the relationships between nodes, can be optimized by learning, for example, loss-based back propagation.

[0033] A neural network may be a neural network model implemented in a processor, and various processes are implemented as a specialized computational structure that can provide an intuitive mapping between input patterns and output patterns, or pattern recognition of input patterns, after training through the neural network model. The learning ability to generate the mapping or perform such exemplary pattern recognition is referred to as the learning ability of the neural network. The learning ability also allows the specialized computational structure to classify input patterns or portions of input patterns into members belonging to one or more predetermined groups. Furthermore, a neural network trained through training has a generalization ability, for example, to generate relatively accurate or reliable outputs for input patterns for which the neural network has not been trained.

[0034] For example, a sequence-to-sequence encoder-decoder is a network structure in which an encoder and a decoder are integrated, and can generate a sequence of recognition results from an input sequence. For example, the encoder and decoder 120 embodied in the sequence-to-sequence structure directly generates a recognition result corresponding to an input speech from the input speech. Before receiving the input speech, the encoder and decoder 120 are pre-trained to generate a sequence of recognition results from a pre-trained input sequence.

[0035] The encoder encodes the input sequence to generate coded features 110. The encoder extracts features from the input sequence to generate coded information. The coded features 110 are applied to a decoder 120. The decoder 120 generates a recognition result based on the coded features 110.

[0036] 1, upon receiving the encoded features 110, the decoder 120 determines token 1 (token1) 105 as the output token based on the input token, start token 115. Having determined token 1 (token1) 105 as the output token, the decoder 120 determines token 1 (token1) 105 as the next input token.

[0037] The decoder 120 of the encoder-decoder structured artificial neural network determines the output token at each step based on the information calculated from the encoder, which may depend on the input token selected up to the previous step.

[0038] For example, the decoder 120, which has determined token 1 (token1) 105 as the input token, predicts the probability of the candidate output token based on token 1 (token1) 105. m are candidates for output tokens, and p1 to p m is the probability of each candidate. Based on the probability, an output token is determined from among the candidates. For example, the token with the highest probability among the candidates is selected as the output token.

[0039] At step i, token t i The conditional probability of can be expressed as equation (1).

[0040] p(t i |t1,t2,...,t i-1 ,H e ) (1) H e corresponds to the features 110 encoded in the hidden representation of the encoder. i-1 is the input token selected so far. For convenience of explanation, Fig. 1 shows the encoded features 110 applied only to the nodes of the first step in the decoder 120, but the encoded features 110 may be applied to other nodes of subsequent steps as well.

[0041] A token selected as an output token in a sequence-to-sequence model that performs general autoregressive decoding is determined as the next input token. Since the output token is determined depending on the input token selected up to the previous step, if the input token has a low probability after being selected as an output token in the sequence-to-sequence model, the performance of predicting the next token by the selected output token will decrease.

[0042] For example, output token candidates c1 to c m The least probable c i may be determined as the output token. Because of the feature of generating the next output based on the previous token, if the next token must be generated based on a token with low probability in the autoregressive decoding process, that is, if the probability of the previous token is low, it is difficult to generate subsequent tokens smoothly. For example, c1 to c m Of these, relatively speaking, i is most likely, but c1 or c mThere may be cases where the probabilities of all of the NNs are not high. Or, in a model ensemble autoregressive decoding environment, where one of the tokens generated by various NN models is selected and used, the next token must be generated based on the tokens generated by other NNs as a result of the ensemble.

[0043] The artificial neural network receives an input token t i-1 If the probability of is low, p(t i |t1,t2,...,t i-1 ,H e ) to generate the output token t i It becomes impossible to predict.

[0044] Input token t with low probability i-1 Even if the output token t i To predict fluently, we use the conditional probability formula (1) for token t i-1 This problem may be solved by calculating the probability excluding the previous token. However, since the system is designed to learn from the training data using the conditional probability described above, there is a concern that calculating the conditional probability independently of the previous token will defeat the original purpose.

[0045] According to one embodiment of the decoding method, a special token t that does not affect the conditional probability is NC Introduce input token t i-1 Even if the probability of t is low, i can be predicted satisfactorily.

[0046] Referring to FIG. 1, the output token c determined for token 1 is i The probability of p i , a predetermined threshold probability θ nc￢ Compare with the threshold probability θ nc￢ is a predetermined value, which is the minimum value of the input token probability that can obtain the desired output. For example, i The probability of p iis the threshold probability θ nc￢ If it is greater than, output token c i As it is determined that the desired output can be obtained when determining the next input token as the output token c i can be determined as the next input token.

[0047] Output token c i The probability of p i is the threshold probability θ nc￢ If it is less than, output token c i When determining the next input token, it is determined that the desired output will not be obtained, so the output token c i It is difficult to determine the next input token. In this case, if the output token c i The probability p i is the threshold probability θ nc￢ If it is less than, output token c i Instead, a predefined special token NC is determined as the next input token.

[0048] The reliability of the next output token candidate predicted based on the special token is higher than the reliability of the next output token candidate predicted based on the output token with a probability lower than the threshold probability. The artificial neural network may be trained in advance to satisfy such a characteristic. The specific details of training the artificial neural network will be described later. For example, i The probability of p i is the threshold probability θ nc￢ If the confidence of the candidate next output token predicted based on the special token NC is smaller than c i is higher than the confidence of the candidate next output token predicted based on

[0049] The confidence level is the confidence level for a token, and means the degree to which it is close to the correct output token of the training data. For example, c i The probability of p i is the threshold probability θ nc￢ If the probability is smaller than , the most probable output token among the candidates for the next output token predicted based on the special token NC is ci The output token is closer to the correct output token than the most probable output token among the candidates for the next output token predicted based on

[0050] If the artificial neural network determines the next input token by selecting either the output token or the special token, it can more appropriately determine the next output token based on at least one input token and the next input token. For example, if the artificial neural network determines the special token NC as the next input token, it can determine the next output token, token 3, based on the input token, token 1, and the next input token, special token NC.

[0051] FIG. 2 is a diagram illustrating a method for masking special tokens in an artificial neural network to which an attention mechanism according to an embodiment is applied.

[0052] 2, in one embodiment, the decoder 220 predicts the probability of a candidate output token based on the correlation between at least one input token and an output token, and the output token can predict the probability of a candidate output token with different weights depending on the input token.

[0053] For example, the artificial neural network may be an artificial neural network to which an attention mechanism is applied, and the artificial neural network can be trained to better represent the relationships between tokens that it has predicted so far.

[0054] To prevent the special token from affecting other tokens, the special token is masked to force other tokens not to be attended to by the special token.

[0055] For example, in the case of an output token corresponding to the input token "is", attention is paid not only to the input tokens "Hi", "this", and "is", but also to the special token NC. Since special tokens should not affect other tokens, the output token corresponding to the input token "is" masks the special token to force it not to pay attention to the special token NC.

[0056] Hereinafter, the encoder-decoder structured artificial neural network system will be described in detail with reference to FIGS.

[0057] FIG. 3 is a block diagram illustrating an artificial neural network system with an encoder-decoder structure according to one embodiment.

[0058] Referring to Figure 3, an embodiment of an encoder-decoder structured artificial neural network system includes an artificial neural network 310 including an encoder 320 and a decoder 330. The encoder-decoder structured artificial neural network includes a pre-processor 340, an attention masking calculation unit 350, and a token modification unit 360. The decoder 330 shown in Figure 3 corresponds to the decoder 220 described above with reference to Figures 1 and 2.

[0059] In one embodiment, an artificial neural network system with an encoder-decoder architecture processes tokens with low probabilities in one artificial neural network 310 .

[0060] The sequence-to-sequence encoder 320-decoder 330 is a network structure in which the encoder 320 and the decoder 330 are integrated, and generates a sequence of recognition results from an input sequence.

[0061] The preprocessor 340 is a module that preprocesses inputs to the artificial neural network, for example, by removing noise contained in the input signals and processing the input signals into a suitable form for inputting them to the artificial neural network.

[0062] At runtime, the token corrector 360 determines whether the probability of the predicted output token exceeds a threshold probability θNC The output tokens lower than 1 can be replaced with special tokens to predict the next output token.

[0063] The token modifying device 360 ​​also determines whether the output token is identical to a special token. If the output token is identical to a special token, the token with the second highest probability among the candidates is determined as the output token. This prevents the special token from being output.

[0064] The attention masking calculation unit 350 masks the attention weights so that other tokens do not have a conditional probability of being associated with the special token, preventing other tokens from being attentive to the special token. The attention masking calculation unit 350 can operate in the same manner when performing learning.

[0065] 3 has been described based on limited drawings as an example for explaining the invention, but a person skilled in the art can apply various technical modifications and variations based on Fig. 3. For example, the decoder 330 is a broad concept including an attention masking calculation device 350 and a token modification device 360.

[0066] FIG. 4 is a block diagram illustrating an artificial neural network system with an encoder-decoder structure using an ensemble method according to one embodiment.

[0067] Referring to FIG. 4, an artificial neural network system according to one embodiment includes multiple artificial neural networks 410, 420, and 430.

[0068] In an ensemble-based encoder-decoder artificial neural network system, the number of encoders corresponding to m decoders does not necessarily have to be m. A decoder without an encoder may exist. For example, the artificial neural networks 420 and 430 may have only decoders 422 and 432, but no encoders 421 and 431.

[0069] According to the ensemble method, an output token can be selected from among output token candidates based on a combination of the probabilities of the output token candidates predicted by a plurality of decoders 412, 422, 432 using a plurality of artificial neural networks 410, 420, 430.

[0070] When using the ensemble approach, each decoder 412, 422, and 432 determines a different output token as the most probable. For example, decoder 412 of artificial neural network 410 determines that c1 is the most probable output token, decoder 422 of artificial neural network 420 determines that c2 is the most probable output token, and decoder 432 of artificial neural network 430 determines that c3 is the most probable output token.

[0071] A single output token can be determined even if each decoder 412, 422, 432 determines a different output token as the most probable one. For example, an output token can be selected from the output token candidates based on a combination of the probabilities of the output token candidates predicted by the multiple decoders 412, 422, 432.

[0072] If the selected output token is forced to be input as the next input token to all decoders, there is a possibility that a decoder will input an output token that is not judged to have the highest probability as the next input token.

[0073] For example, if multiple decoders 412, 422, and 432 ultimately select c1 as the output token based on the combination of probabilities of candidate output tokens c1, c2, and c3 predicted by the decoders 412, 422, and 432, the decoders 422 and 432 may not be able to generate the next output token based on the candidate output tokens c2 and c3 that are most probable based on their own criteria, and may have to generate the next token based on c1, which is the candidate output token generated by another decoder 412. Due to the autoregressive nature of determining the next token based on the previous token, the artificial neural networks 420 and 430 that input the next input token c1 with a low probability in their own decoders 422 and 432 may not be able to smoothly predict the next output token.

[0074] In one embodiment, an artificial neural network system with an encoder-decoder structure using an ensemble method also introduces a special token that cannot affect the conditional probability, and can correctly predict the output token by inputting a token that it does not determine to be the most probable.

[0075] When multiple artificial neural networks are ensembled, the output tokens determined for each j-th artificial neural network are max Probability P for j (t max ) and t max For example, among the output token candidates c1, c2, and c3, the probability p2(c1) for c1 determined as the output token in the artificial neural network 420 is compared with the probability p1(c1) for c1 in the artificial neural network 410 that determined c1 as the highest.

[0076] The difference between the output token probability of the artificial neural network 410 that determined the highest output token probability and the output token probability of the jth artificial neural network

[0077]

number

[0078] The difference between the output token probability of the artificial neural network with the highest output token probability and the output token probability of the jth artificial neural network

[0079]

number

[0080] The method for masking special tokens described with reference to FIG. 2 is similarly applicable to an artificial neural network system with an encoder-decoder structure using an ensemble approach as shown in FIG.

[0081] At runtime, the token corrector 460 determines whether the probability of the predicted output token exceeds a threshold probability θ NC The output tokens lower than 1 can be modified to predict the next output token by substituting a special token.

[0082] The token corrector 460, at run time, calculates the difference between the output token probability of the artificial neural network 410 that has determined the highest output token probability and the output token probability of the jth artificial neural network.

[0083]

number

[0084] The token modifying device 460 also determines whether the output token is identical to a special token. If the output token is identical to a special token, the token with the second highest probability among the candidates is determined as the output token. This prevents the special token from being output.

[0085] The attention masking calculation unit 450 masks the attention weights so that other tokens cannot be attracted to the special token, so that other tokens do not have a conditional probability of being attracted to the special token. The attention masking calculation unit 450 operates in the same manner not only during learning but also during execution (inference or realization).

[0086] 5 is a block diagram illustrating an application embodiment of a speech recognition artificial neural network with an additional language model artificial neural network according to one embodiment. The speech recognition artificial neural network is implemented as an example of a speech recognizer. The speech recognizer can collect speech or input from a user using a receiver or sensor (e.g., a microphone) that is a component of the speech recognizer, and can receive audio signals from the receiver, which may be separate or external to the speech recognizer.

[0087] The voice recognition device may be an electronic device used by an individual user, such as a wearable device such as a smartphone, a smart pad, a smart band, a personal digital assistant (PDA), a laptop, a component therein, or a standalone device related thereto. In different examples, the voice recognition device may also include electronic devices shared by multiple users, such as a voice recognition speaker and a voice recognition TV.

[0088] Referring to FIG. 5, a speech recognition artificial neural network with a language model artificial neural network according to an embodiment includes a speech recognition model artificial neural network 510 and a language model artificial neural network 520.

[0089] In the case of speech recognition, the basic system is composed of an encoder 511 that receives speech as input and a decoder 512 that outputs a sequence of words as the recognition result. In addition, the performance can be further improved by utilizing a language model artificial neural network 520 formed by a decoder 521 that determines how likely a sequence of words is to occur in a general sentence.

[0090] A speech recognition model artificial neural network, which predicts words at each point in time based on actual speech, creates new words by listening to speech and combining phonemes according to pronunciation. However, in the case of a language model artificial neural network trained on ordinary sentences, when a new word that was not learned during training (such as a new place or a neologism) appears, the output probability of that word becomes extremely low. When trying to predict the next word using a word with such a low probability as input, the language model artificial neural network has to output the next word from an input that it has not learned, making it difficult to make an accurate prediction.

[0091] In one embodiment, the language model artificial neural network 520 also uses a token modification device 560 and an attention masking calculation device 550 to introduce special tokens that cannot affect the conditional probability described above with reference to FIG. 4, and can correctly predict output tokens using tokens that it does not determine as having the highest probability as input.

[0092] Hereinafter, an embodiment of an application in a speech recognition artificial neural network having an additional language model artificial neural network will be described in detail with reference to FIGS. 6 to 7B.

[0093] FIG. 6 illustrates a speech recognition artificial neural network system with an additional language model artificial neural network according to one embodiment.

[0094] Referring to FIG. 6, a speech recognition artificial neural network system with an additional language model artificial neural network according to one embodiment includes an encoder 610, a speech recognition artificial neural network decoder 620, and a language model artificial neural network decoder 630.

[0095] A speech recognition artificial neural network system with an additive language model artificial neural network extracts speech features 640 from input speech. The input speech is a speech signal containing information for each of multiple frames, and the speech features 640 are a sequence of information extracted for at least one frame, and are represented as a multidimensional vector. For convenience of explanation, the following embodiment will be described assuming that the input speech is "Hi MinChul, this is."

[0096] A speech recognition artificial neural network system with an additional language model artificial neural network generates a sequence of recognition results from a sequence of input speech using an ensemble of a language model artificial neural network decoder 630 and a speech recognition artificial neural network decoder 620. An "ensemble" refers to individual artificial neural network modules considered together. For example, an ensemble can be a combination of a language model artificial neural network decoder 630 and a speech recognition artificial neural network decoder 620.

[0097] The language model artificial neural network decoder 630 and the speech recognition artificial neural network decoder 620 output their respective recognition results on a token-by-token basis and ensemble the respective recognition results according to ensemble weights to generate a final recognition result. For example, the speech recognition artificial neural network decoder 620 determines output token candidates based on the input speech and previously determined recognition results, and the language model artificial neural network decoder 630 determines output token candidates based on the previously determined recognition results and ensemble the output token candidates according to ensemble weights (weighted sums) to generate a final recognition result.

[0098] The encoder 610 and decoder 620 are pre-trained to generate a sequence of recognition results from a sequence of ground truth text pairs corresponding to input speech, while the language model artificial neural network decoder 630 is pre-trained to generate a sequence of recognition results from any text sequence.

[0099] The encoder 610 encodes the speech features 640 to generate encoded features 650. The encoder 610 transforms the dimensions of the speech features 640 to generate encoded information. The encoded features 650 are applied to the speech recognition artificial neural network decoder 620. The speech recognition artificial neural network decoder 620 generates output token candidates on a token-by-token basis based on the encoded features 650 and previously determined recognition results, and the language model artificial neural network decoder 630 generates output token candidates on a token-by-token basis based on previously determined recognition results. The two recognition results can be ensembled according to a predetermined ensemble weight to generate a final recognition result. For example, the ensemble weight ratio of the speech recognition artificial neural network decoder 620 and the language model artificial neural network decoder 630 can be 1:0.2.

[0100] Hereinafter, with reference to FIGS. 7A and 7B, an embodiment in which the ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder will be described in detail.

[0101] FIG. 7A is a diagram illustrating an embodiment in which an ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder according to an embodiment.

[0102] Referring to FIG. 7A, a speech recognition artificial neural network decoder 710 and a language model artificial neural network decoder 720 according to one embodiment correspond to the speech recognition artificial neural network decoder 620 and the language model artificial neural network decoder 630 shown in FIG. 6, respectively.

[0103] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 function as autoregressive decoders and predict the probability of a candidate output token based on the input token "Hi." For example, the speech recognition artificial neural network decoder 710 outputs "MinChul" and "Bixby" as candidate output tokens based on the input token "Hi" and the encoded speech features, predicting the respective probabilities to be 0.7 and 0.1. Similarly, the language model artificial neural network decoder 720 outputs "MinChul" and "Bixby" as candidate output tokens based on the input token "Hi," predicting the respective probabilities to be 0.001 and 0.8.

[0104] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 use the ensemble weights to determine an output token from the candidates. For example, “MinChul” has a final weight of 0.7+0.2×0.001=0.7002, and “Bixby” has a final weight of 0.1+0.2×0.8=0.26, so “MinChul” is determined as the output token.

[0105] In an autoregressive decoding environment, the language model artificial neural network decoder 720 inputs "MinChul," determined as the result of the ensemble other than "Bixby," which has the highest probability as its own reference, as the next input token. Because "MinChul" has a low probability in the language model artificial neural network decoder 720, it is difficult to smoothly predict the next output token.

[0106] For example, based on the next input token "MinChul," the language model artificial neural network decoder 720 outputs "this" and "dis" as candidates for the next output token, with predicted probabilities of 0.01 and 0.01, respectively. Based on the next input token "MinChul," the speech recognition artificial neural network decoder 710 outputs "this" and "dis" as candidates for the next output token, with predicted probabilities of 0.34 and 0.38, respectively. Because "this" and "dis" are pronounced similarly, the speech recognition artificial neural network decoder 710 may predict the probabilities of "this" and "dis" to be similar.

[0107] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 use the ensemble weights to determine the next output token from the candidates. For example, "this" has a final weight of 0.34+0.2×0.01=0.342, and "dis" has a final weight of 0.38+0.2×0.01=0.382, so "dis" is determined as the next output token. As a result, the speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 shown in FIG. 7A output "dis", which is different from the input speech "this".

[0108] In the above-described embodiment, among the candidates for the current output token predicted by the language model artificial neural network decoder 720, the probability of "MinChul" determined by the ensemble between the speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 is lower than a predetermined threshold probability. Due to the low probability of "MinChul," the language model artificial neural network decoder 720 is unable to properly distinguish between "this" and "dis" from the next output token. As a result, as the performance of the language model artificial neural network decoder 720, which should complement the performance of the speech model artificial neural network decoder 710, deteriorates, an erroneous result may be obtained, such as outputting "dis" instead of "this" as in the above-described embodiment.

[0109] Hereinafter, an embodiment of the ensemble method in a language model artificial neural network decoder using special tokens and a speech recognition artificial neural network decoder will be described in detail with reference to FIG. 7B.

[0110] FIG. 7B is a diagram illustrating an embodiment in which an ensemble method is applied to a language model artificial neural network decoder and a speech recognition artificial neural network decoder using special tokens according to an embodiment.

[0111] Referring to FIG. 7B, the speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 according to an embodiment use the ensemble weights to determine "MinChul" as the output token from among the candidates.

[0112] The language model artificial neural network decoder 720 compares the probability of "MinChul" with a predetermined threshold probability. For example, the predetermined threshold probability may be 0.01. The language model artificial neural network decoder 720 determines that the probability of "MinChul" is 0.001, which is less than the threshold probability of 0.01, and therefore selects the special token NC as the next input token instead of "MinChul".

[0113] Based on the next input token, the special token NC, the language model artificial neural network decoder 720 outputs "this" and "dis" as candidates for the next output token, predicting their respective probabilities as 0.4 and 0.1. Based on the next input token, "MinChul," the speech recognition artificial neural network decoder 710 outputs "this" and "dis" as candidates for the next output token, predicting their respective probabilities as 0.34 and 0.38.

[0114] The speech recognition artificial neural network decoder 710 and the language model artificial neural network decoder 720 use the ensemble weights to determine the next output token from the candidates. For example, "this" has a final weight of 0.34+0.2×0.4=0.42, and "dis" has a final weight of 0.38+0.2×0.1=0.4, so "this" is determined as the next output token.

[0115] FIG. 8 is a flowchart illustrating a decoding method in an artificial neural network according to one embodiment. The operations shown in FIG. 8 are performed in the order and manner shown, but the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the illustrated embodiment. Multiple operations shown in FIG. 8 may be performed in parallel or simultaneously. One or more blocks and combinations of blocks shown in FIG. 8 may be implemented by a computer based on special-purpose hardware performing a specific function, or by a combination of special-purpose hardware and computer instructions. In addition to the following description of FIG. 8, the descriptions of FIGS. 1 to 7 are applicable to FIG. 8, and therefore, overlapping descriptions will be omitted.

[0116] Referring to Figure 8, steps S810 to S830 according to one embodiment are performed by the decoder 330 described above with reference to Figure 3. The decoder 330 may be implemented by one or more hardware components, or one or more components including hardware and software.

[0117] In step S810, the decoder 330 predicts the probability of a candidate output token based on at least one input token that has been input up to now.

[0118] In step S820, decoder 330 determines the candidate output token based on probability, selects the candidate token with the highest probability, or selects the candidate output token based on a combination of the candidate probability predicted by a second artificial neural network in conjunction with the first artificial neural network and the candidate probability predicted by the first artificial neural network.

[0119] In step S830, the decoder 330 determines the next input token by selecting either the output token or one of the predefined special tokens based on the probability of the output token.

[0120] The at least one input token includes at least one of an output token and a special token previously predicted by the artificial neural network.

[0121] The input layer of the artificial neural network includes a plurality of nodes corresponding to candidates and special nodes corresponding to special tokens.

[0122] FIG. 9 is a flowchart illustrating a method for processing low-probability tokens in an autoregressive-based sequence-generating artificial neural network according to one embodiment. The operations illustrated in FIG. 9 are performed in the order and manner shown, although the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the illustrated embodiment. Multiple operations illustrated in FIG. 9 may be performed in parallel or simultaneously. One or more blocks and combinations of blocks illustrated in FIG. 9 may be implemented by a computer based on special-purpose hardware performing a specific function, or by a combination of special-purpose hardware and computer instructions. In addition to the following description of FIG. 9, the descriptions of FIGS. 1 to 8 are applicable to FIG. 9, and therefore, overlapping descriptions will be omitted.

[0123] 9, in step S901, an autoregressive sequence generation artificial neural network according to one embodiment receives an input, for example, an input speech in a speech recognition artificial neural network system.

[0124] In step S902, the encoder generates coded features from the received input, for example, by transforming the dimensions of the audio features to generate coded information.

[0125] In step S903, the decoder predicts the probability of candidate output tokens based on the encoded features. At each step, the decoder determines the output token based on information calculated from the encoder, which may depend on the input token selected in the previous step.

[0126] In step S904, the decoder determines the output token from among the candidates based on probability. For example, the most probable t i (top likelihood token) is determined as the output token.

[0127] In step S905, the decoder determines whether the output token is identical to the special token.

[0128] In step S906, if the output token is the same as the special token, the decoder determines the token with the second highest probability among the candidates as the output token, thereby preventing the special token from being output.

[0129] In step S907, if the output token is not identical to the special token, the decoder compares the probability of the output token with a predetermined threshold probability.

[0130] In step S908, if the probability of the output token is greater than the threshold probability, the decoder selects the output token as the next input token.

[0131] In step S909, if the probability of the output token is less than the threshold probability, the decoder selects a special token as the next input token instead of the output token.

[0132] In step S910, the decoder masks the special token to force other tokens not to pay attention to the special token, to prevent the special token from affecting different tokens.

[0133] A special token t that cannot affect the conditional probability NC Introduce an input token t i-1 Even if the probability of t is low, i The special token is trained to have the characteristics shown in Equation (2).

[0134]

number

[0135] In other words, when the auxiliary artificial neural network is classified, the correct token is identified with a probability θ Replace For example, in the case of a speech recognition artificial neural network system having an additional language model artificial neural network, the language model artificial neural network may be an auxiliary artificial neural network, and in this case, the correct token can be identified with a probability θ Replace can be changed to and trained.

[0136] FIG. 10 is a block diagram of an apparatus for processing low probability tokens in an autoregressive-based sequence-generating artificial neural network according to one embodiment.

[0137] 10 , an apparatus 1000 for processing low probability tokens in an autoregression-based sequence generating artificial neural network according to one embodiment includes a sensor 1010, a processor 1030, and a communication interface 1070. The apparatus 1000 for processing low probability tokens in an autoregression-based sequence generating artificial neural network further includes a memory 1050 and a display 1090. The sensor 1010, the processor 1030, the memory 1050, the communication interface 1070, and the display 1090 communicate via a communication bus 1005.

[0138] The sensor 1010 includes, for example, a microphone sensor and an audio sensor.

[0139] The processor 1030 may perform at least one of the methods or an algorithm corresponding to at least one of the methods described above with reference to Figures 1 to 9. The processor 1030 executes a program to control the processing unit 1000 for low probability tokens in an autoregressive-based sequence generating artificial neural network. The program code executed by the processor 1030 is stored in the memory 1050.

[0140] The processor 1030 is configured by, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0141] The memory 1050 stores data processed by the processor 1030. For example, the memory 1050 stores a program. The stored program may be a set of syntax that is coded to assist the user in walking and is executable by the processor 1030. The memory 1050 may be a volatile memory or a non-volatile memory.

[0142] The communication interface 1070 is connected to the multiple sensors 1010, the processor 1030, and the memory 1050 to transmit and receive data. The communication interface 1070 is connected to other external devices to transmit and receive data. Hereinafter, the expression "transmitting and receiving "A" means transmitting and receiving "information or data indicating A."

[0143] The communication interface 1070 is implemented as a circuit network within the device 1000 for processing low-probability tokens in an autoregression-based sequence generating artificial neural network. For example, the communication interface 1070 includes an internal bus and an external bus. As another example, the communication interface 1070 may be an element connecting the device 1000 for processing low-probability tokens in an autoregression-based sequence generating artificial neural network with an external device. The communication interface 1070 receives data from an external device and transmits the data to the processor 1030 and the memory 1050.

[0144] The display 1090 displays the decoded results, such as translation results and speech recognition results.

[0145] The pre-processors 340, 440, 441, 442, attention masking calculation devices 350, 450, 550, encoders 320, 411, 421, 431, 511, decoders 330, 412, 422, 432, 512, 521, token modification devices 360, 460, 560, sensor 1010, processor 1030, memory 1050, communication interface 1070, and display 1090 described above with reference to Figures 1 to 10 may be embodied as hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA), programmable logic unit (PLU), microprocessor, or other device that executes and responds to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, a single processing device may be described; however, those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0146] Software includes computer programs, codes, instructions, or a combination of one or more thereof, which can configure a processing device to operate as desired or can independently or in combination instruct the processing device. The software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software can be distributed across computer systems coupled to a network and stored and executed in a distributed manner. The software and data can be stored on one or more computer-readable recording media.

[0147] The methods according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and stored on a computer-readable storage medium. The storage medium may include program instructions, data files, data structures, and the like, alone or in combination. The storage medium and program instructions may be specially designed and constructed for the purposes of the present invention, or they may be well-known and available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that is executed by a computer using an interpreter, for example. A hardware device may be configured to operate as one or more software modules to perform the operations described in the present invention, or vice versa.

[0148] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted with other components or equivalents, and still achieve suitable results.

[0149] Therefore, the scope of the present invention should not be limited to the disclosed embodiments, but should be determined by the appended claims and their equivalents.

[0150] <<<Additional Notes>>> (Appendix 1) In the first artificial neural network decoding method, predicting probabilities of candidate output tokens based on at least one previously input token; determining the output token from among the candidates based on the predicted probability; determining a next input token by selecting either the output token or a predefined special token based on the determined probability of the output token; A decoding method in an artificial neural network, comprising: (Appendix 2) When the special token is determined as the next input token, A decoding method in an artificial neural network as described in Appendix 1, wherein the reliability of the candidate next output token predicted based on the special token is higher than the reliability of the candidate next output token predicted based on the output token. (Appendix 3) 2. The decoding method in an artificial neural network of claim 1, wherein determining the output token comprises selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network. (Appendix 4) A decoding method in an artificial neural network described in any one of Appendices 1-3, wherein the input layer of the artificial neural network includes a plurality of nodes corresponding to each of the candidates and a special node corresponding to the special token. (Appendix 5) 4. A decoding method in an artificial neural network according to any one of claims 1 to 3, wherein the at least one input token includes at least one of an output token predicted in advance by the artificial neural network and the special token. (Appendix 6) 2. The method of decoding in an artificial neural network according to claim 1, wherein the step of determining the output token comprises the step of selecting the token with the highest probability from the candidates. (Appendix 7) determining the next input token by comparing the probability of the output token with a predetermined threshold probability; selecting the special token as the next input token if the probability of the output token is less than the threshold probability; selecting the output token as the next input token if the probability of the output token is greater than the threshold probability; 2. A decoding method in an artificial neural network according to claim 1, comprising: (Appendix 8) 2. The decoding method in an artificial neural network of claim 1, wherein predicting the probability of a candidate output token comprises predicting the probability of a candidate output token based on a correlation between the at least one input token and the output token. (Appendix 9) 9. A decoding method in an artificial neural network according to any one of claims 1-8, further comprising the step of determining a next output token based on the at least one input token and the next input token. (Appendix 10) A decoding method in an artificial neural network as described in any one of appendices 1-9, further comprising a step of masking the special token so that tokens other than the special token do not attract attention to the special token. (Appendix 11) A decoding method in an artificial neural network described in any one of Appendices 1-10, further comprising a step of setting the correlation between the next output token and the special token to a predetermined value or less when the special token is determined as the next input token. (Appendix 12) The step of determining an output token comprises: determining whether the output token is identical to the special token; If the output token is identical to the special token, determining the second most probable token among the candidates as the output token; 2. A decoding method in an artificial neural network according to claim 1, comprising: (Appendix 13) In the first artificial neural network decoding method, predicting probabilities of candidate output tokens based on at least one previously input token; selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network; comparing the probability of the output token between the artificial neural network that determines the highest probability of the output token among the at least one second artificial neural network and the first artificial neural network; determining a next input token of the first artificial neural network by selecting one of the output token and a predefined special token based on the comparison result; A decoding method in an artificial neural network, comprising: (Appendix 14) determining the next input token by selecting the special token as the next input token of the first artificial neural network if the difference between the probability of the output token of the artificial neural network that determined the highest probability of the output token and the probability of the output token of the first artificial neural network is greater than a predetermined threshold; selecting the output token as the next input token of the first artificial neural network if the difference between the probability of the output token of the first artificial neural network and the artificial neural network that determined the highest probability of the output token is less than a predetermined threshold; 14. A decoding method in an artificial neural network according to claim 13, comprising: (Appendix 15) 15. A decoding method in an artificial neural network as described in Appendix 14, wherein the reliability of the candidate next output token predicted based on the special token is higher than the reliability of the candidate next output token predicted based on the output token if the reliability is greater than the threshold. (Appendix 16) A decoding method in an artificial neural network described in any one of Appendices 13-15, wherein the input layer of the artificial neural network includes a plurality of nodes corresponding to each of the candidates and a special node corresponding to the special token. (Appendix 17) 16. A decoding method in an artificial neural network according to any one of appendices 13-15, wherein the at least one input token comprises at least one of an output token previously predicted by the artificial neural network and the special token. (Appendix 18) 14. The method of decoding in an artificial neural network of claim 13, wherein predicting the probability of a candidate output token comprises predicting the probability of a candidate output token based on a correlation between the at least one input token and the output token. (Appendix 19) 19. A method for decoding in an artificial neural network according to any one of claims 13-18, further comprising the step of determining a next output token based on the at least one input token and the next input token. (Appendix 20) 20. A decoding method in an artificial neural network according to any one of appendices 13-19, further comprising a step of masking the special token so that tokens other than the special token do not draw attention to the special token. (Appendix 21) A decoding method in an artificial neural network described in any one of Appendixes 13-20, further comprising a step of setting a correlation between the next output token and the special token to a predetermined value or less when the special token is determined as the next input token. (Appendix 22) The step of determining an output token comprises: determining whether the output token is identical to the special token; If the output token is identical to the special token, determining the second most probable token among the candidates as the output token; 14. A decoding method in an artificial neural network according to claim 13, comprising: (Appendix 23) A computer program that causes a computer to execute the decoding method according to any one of Supplementary Note 1 to Supplementary Note 22. (Appendix 24) A speech recognition device in an artificial neural network, comprising: predicting probabilities of candidate output tokens based on at least one previously input token; determining the output token from among the candidates based on the predicted probability; a processor that determines a next input token by selecting either the output token or a predefined special token based on the determined probability of the output token; A speech recognition device comprising: (Appendix 25) When the special token is determined as the next input token, 25. The speech recognition device of claim 24, wherein the reliability of the next output token candidate predicted based on the special token is higher than the reliability of the next output token candidate predicted based on the output token. (Appendix 26) 26. The speech recognition device according to claim 24, wherein an input layer of the artificial neural network includes a plurality of nodes corresponding to each of the candidates and a special node corresponding to the special token. (Appendix 27) 27. The speech recognition device of claim 24, wherein the at least one input token comprises at least one of an output token previously predicted by the artificial neural network and the special token. (Appendix 28) 25. The speech recognition apparatus of claim 24, wherein the processor selects the token with the highest probability from the candidates. (Appendix 29) 25. The speech recognition apparatus of claim 24, wherein the processor selects the output token from among the candidates based on a combination of probabilities of the candidates predicted by the artificial neural network and probabilities of the candidates predicted by another artificial neural network in conjunction with the artificial neural network. (Appendix 30) The processor: comparing the probability of the output token with a predetermined threshold probability; If the probability of the output token is less than the threshold probability, selecting the special token as the next input token; 25. The speech recognition apparatus of claim 24, wherein if the probability of the output token is greater than the threshold probability, the output token is selected as the next input token. (Appendix 31) 31. The speech recognition device of claim 24, wherein the processor predicts a probability of a candidate output token based on a correlation between the at least one input token and the output token. (Appendix 32) 32. The speech recognition device of claim 24, wherein the processor determines a next output token based on the at least one input token and the next input token. (Appendix 33) 33. The speech recognition device according to any one of appendices 24-32, wherein the processor masks the special token so that tokens other than the special token do not draw attention to the special token. (Appendix 34) 25. The speech recognition device of claim 24, wherein, when the special token is determined as the next input token, the processor sets a correlation between the next output token and the special token to a predetermined value or less. (Appendix 35) The processor: determining whether the output token is identical to the special token; 25. The speech recognition device of claim 24, wherein if the output token is identical to the special token, the token with the second highest probability among the candidates is determined to be the output token. (Appendix 36) predicting probabilities of candidate output tokens based on at least one input token input up to the present; selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network; comparing the probability of the output token between the artificial neural network that determines the highest probability of the output token among the at least one second artificial neural network and the first artificial neural network; a processor that determines a next input token for the first artificial neural network by selecting one of the output token and a predefined special token based on the comparison result. (Appendix 37) The processor: If the difference between the probability of the output token of the artificial neural network having the highest probability of the output token and the probability of the output token of the first artificial neural network is greater than a predetermined threshold, select the special token as the next input token of the first artificial neural network; 37. The speech recognition device of claim 36, wherein if a difference between the probability of the output token of the artificial neural network having the highest probability and the probability of the output token of the first artificial neural network is smaller than a predetermined threshold, the output token is selected as the next input token of the first artificial neural network. (Appendix 38) A speech recognition device as described in Appendix 37, wherein the reliability of a candidate for the next output token predicted based on the special token is higher than the reliability of a candidate for the next output token predicted based on the output token when the reliability is greater than the threshold. (Appendix 39) a memory for storing instructions; one or more processors; Including, When the instructions are executed by the processor, the processor: Extracting speech features from the input speech and encoding the speech features to generate encoded features; determining a probability of a candidate output token based on the encoded features and a previously determined recognition result; A speech recognition system that determines a next output token based on weights of the candidate output tokens determined by a language model artificial neural network and weights of the candidate output tokens determined by a speech recognition artificial neural network. < / t1>

Claims

1. In a decoding method in a first artificial neural network, predicting probabilities of candidate output tokens based on at least one previously input token; selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network; comparing the probability of the output token between the first artificial neural network and the at least one second artificial neural network that determines the highest probability of the output token; determining a next input token for the first artificial neural network by selecting one of the output token and a predefined special token based on the result of the comparison; and determining the next input token comprises: selecting the special token as the next input token of the first artificial neural network if the difference between the probability of the output token of the first artificial neural network and the probability of the output token of the artificial neural network that has determined the highest probability of the output token is greater than a predetermined threshold; selecting the output token as the next input token of the first artificial neural network if the difference between the probability of the output token of the first artificial neural network and the artificial neural network that determined the highest probability of the output token is less than a predetermined threshold; A decoding method in an artificial neural network, comprising:

2. 2. The decoding method in an artificial neural network according to claim 1, wherein the reliability of the next output token candidate predicted based on the special token is higher than the reliability of the next output token candidate predicted based on the output token when the reliability is greater than the threshold.

3. 3. The decoding method in an artificial neural network according to claim 1, wherein the input layer of the artificial neural network includes a plurality of nodes corresponding to each of the candidates and a special node corresponding to the special token.

4. 4. A method for decoding in an artificial neural network according to claim 1, wherein said at least one input token comprises at least one of an output token previously predicted by said artificial neural network and said special token.

5. 2. The method of decoding in an artificial neural network of claim 1, wherein predicting the probability of a candidate output token comprises predicting the probability of a candidate output token based on a correlation between the at least one input token and the output token.

6. The method of decoding in an artificial neural network according to any one of claims 1 to 5, further comprising the step of determining a next output token based on said at least one input token and said next input token.

7. A decoding method in an artificial neural network according to any one of claims 1 to 6, further comprising the step of masking the special token so that tokens other than the special token do not pay attention to the special token.

8. A decoding method in an artificial neural network as described in any one of claims 1 to 7, further comprising a step of setting the correlation between the next output token and the special token to a predetermined value or less when the special token is determined as the next input token.

9. The step of selecting an output token comprises: determining whether the output token is identical to the special token; If the output token is the same as the special token, determining the second most probable token among the candidates as the output token; 2. The method of decoding in an artificial neural network of claim 1, comprising:

10. predicting probabilities of candidate output tokens based on at least one input token input up to the present; selecting the output token from among the candidates based on a combination of the probability of the candidate predicted by at least one second artificial neural network in conjunction with the first artificial neural network and the probability of the candidate predicted by the first artificial neural network; comparing the probability of the output token between the first artificial neural network and the at least one second artificial neural network that determines the highest probability of the output token; a processor for determining a next input token of the first artificial neural network by selecting one of the output token and a predefined special token based on a result of the comparison, wherein the processor, when determining the next input token, If the difference between the probability of the output token of the first artificial neural network and the probability of the output token of the artificial neural network having the highest probability is greater than a predetermined threshold, select the special token as the next input token of the first artificial neural network; A speech recognition device that selects the output token as the next input token of the first artificial neural network if the difference between the probability of the output token of the artificial neural network that determined the output token to be the highest and the probability of the output token of the first artificial neural network is smaller than a predetermined threshold.

11. The speech recognition device of claim 10, wherein the reliability of the candidate next output token predicted based on the special token is higher than the reliability of the candidate next output token predicted based on the output token when the reliability is greater than the threshold.

12. a memory for storing instructions; one or more processors; A speech recognition system comprising: A speech recognition system, wherein the instructions, when executed by the processor, cause the processor to perform the decoding method of any one of claims 1 to 9.

Images