Method and apparatus for determining output lexical units

Abstract

A method and device for determining an output lexical unit are provided. The method for determining an output lexical unit includes: predicting a first probability of each candidate output lexical unit of a first model; predicting a second probability of each candidate output lexical unit of a second model interacting with the first model; adjusting the second probability of each candidate output lexical unit based on the first probability; and determining the output lexical unit among the candidate output lexical units based on the first probability and the adjusted second probability.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0127876, filed on October 15, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to a method and apparatus for determining an output lexical unit, and more particularly, to a method of forming an ensemble of two or more artificial neural networks. Background Art

[0003] An encoder-decoder artificial neural network model has been introduced for generating an output having an uncertain length after receiving a sequence having an uncertain length as an input. The encoder-decoder artificial neural network model is a sequence-to-sequence model.

[0004] Sequence-to-sequence models can be used in various application areas to output a sequence in a domain different from the input sequence. For example, sequence-to-sequence models can be mainly used in chatbots and machine translation. A chatbot can be constructed by configuring the input sequence and the output sequence as questions and answers, respectively. In machine translation, a translator can be constructed by configuring the input sequence and the output sequence as input sentences and translated sentences, respectively. Sequence-to-sequence models can also be used for text summarization and speech-to-text (STT) recognition. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a method for determining an output lexical unit includes: predicting a first probability for each candidate output lexical unit of a first model; predicting a second probability for each candidate output lexical unit of a second model that interacts with the first model; adjusting the second probability of each candidate output lexical unit based on the first probability; and determining the output lexical unit among the candidate output lexical units based on the first probability and the adjusted second probability.

[0007] The step of adjusting may include: determining a ranking of candidate output tokens of the first model based on the first probability; and adjusting the second probability based on the ranking.

[0008] Adjusting the second probability based on the ranking may include adjusting the second probability using a function determined based on the ranking.

[0009] The step of adjusting the second probability based on the ranking may include adjusting the second probability based on one or both of a proportionality factor and a threshold factor determined according to the ranking.

[0010] The adjustment step may include: determining a ranking of candidate output lexical units of the first model based on a first probability; extracting a target candidate output lexical unit on which adjustment will be performed from the candidate output lexical units based on the ranking; and adjusting a second probability corresponding to the extracted target candidate output lexical unit.

[0011] The step of adjusting the second probability corresponding to the extracted target candidate output lexical unit may include adjusting the second probability corresponding to the extracted target candidate output lexical unit using a function determined based on the ranking.

[0012] The method may further include normalizing the adjusted second probability.

[0013] The normalizing step may include normalizing the adjusted second probabilities so that a sum of the adjusted second probabilities is 1.

[0014] The method may further include: performing a beam search to search for a preset number of candidate output lexical units among the candidate output lexical units; and normalizing the adjusted second probability when performing the beam search.

[0015] The step of determining the output lexical unit may include: calculating a weighted sum of the first probability and the adjusted second probability; and determining a candidate output lexical unit having a maximum weighted sum among the candidate output lexical units as the output lexical unit.

[0016] The weight applied to the first probability may be greater than the weight applied to the adjusted second probability.

[0017] The step of predicting the first probability may include predicting the first probability based on output lexical units previously determined by the first model and the second model. The step of predicting the second probability may include predicting the second probability based on output lexical units previously determined by the first model.

[0018] In another general aspect, an apparatus for determining an output lexical unit includes one or more processors. The one or more processors can be configured to: predict a first probability for each candidate output lexical unit of a first model; predict a second probability for each candidate output lexical unit of a second model that interacts with the first model; adjust the second probability based on the first probability; and determine the output lexical unit among the candidate output lexical units based on the first probability and the adjusted second probability.

[0019] The one or more processors may be configured to: determine a ranking of candidate output tokens of the first model based on the first probability; and adjust the second probability based on the ranking.

[0020] The one or more processors may be configured to adjust the second probability using a function determined based on the ranking.

[0021] The one or more processors may be configured to adjust the second probability based on one or both of a scaling factor and a threshold factor determined according to the ranking.

[0022] The one or more processors may be configured to: determine a ranking of candidate output lexical units of the first model based on a first probability; extract a target candidate output lexical unit on which adjustment will be performed from the candidate output lexical units based on the ranking; and adjust a second probability corresponding to the extracted target candidate output lexical unit.

[0023] The one or more processors may be configured to adjust a second probability corresponding to the extracted target candidate output token using a function determined based on the ranking.

[0024] The one or more processors may be configured to normalize the adjusted second probability.

[0025] The one or more processors may be configured to normalize the adjusted second probabilities such that a sum of the adjusted second probabilities is one.

[0026] The one or more processors may be configured to: perform a beam search to search for a preset number of candidate output lexical units among the candidate output lexical units; and normalize the adjusted second probability when performing the beam search.

[0027] The one or more processors may be configured to: calculate a weighted sum of the first probability and the adjusted second probability; and determine a candidate output lexical unit having a maximum weighted sum among the candidate output lexical units as the output lexical unit.

[0028] The weight applied to the first probability may be greater than the weight applied to the adjusted second probability.

[0029] The one or more processors may be configured to: predict a first probability based on output lexical units previously determined according to the first model and the second model; and predict a second probability based on output lexical units previously determined according to the first model.

[0030] In another general aspect, a device for determining a lexical unit includes one or more processors. The one or more processors can be configured to: rank candidate lexical units based on a first probability of the candidate lexical units; determine a weight of each candidate lexical unit based on the ranking of each candidate lexical unit; adjust the second probability of the candidate lexical unit by weighting each second probability with a corresponding weight; calculate a weighted sum of each candidate lexical unit based on the first probability and the adjusted second probability; and output the candidate lexical unit with the maximum weighted sum.

[0031] The one or more processors may be configured to: rank the candidate tokens by grouping the candidate tokens into two or more ranked groups based on the first probabilities of the candidate tokens; and determine a different weight for each ranked group.

[0032] The one or more processors may be configured to determine a weight for each candidate token such that a first candidate token having a first probability higher than a first probability of a second candidate token has a greater weight than a weight of the second candidate token.

[0033] The one or more processors may be configured to determine a weight of each candidate token to be greater than 1, 1, or between 0 and 1 based on the ranking of each candidate token.

[0034] The one or more processors may be configured to rank the candidate tokens in descending order from a candidate token having a highest first probability to a candidate token having a lowest first probability.

[0035] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a diagram illustrating an example of an integration method of combining a plurality of artificial neural network models.

[0037] Figure 2 is a diagram illustrating an example of an integrated model.

[0038] Figure 3 is a flow chart illustrating an example of the operation of an integrated model.

[0039] Figure 4 is a flow chart illustrating an example of a method of adjusting a second probability based on a first probability.

[0040] Figure 5 is a graph showing an example of a function for adjusting the second probability.

[0041] Figure 6is a diagram showing an example of normalizing the adjusted second probability.

[0042] Figure 7 is a diagram illustrating an example of an apparatus for determining an output lexical unit.

[0043] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0044] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is only an example and is not limited to the order set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order.

[0045] The features described herein can be implemented in different forms and should not be interpreted as being limited to the examples described herein. Instead, the examples described herein have only been provided to illustrate some of the many possible ways of implementing the methods, devices and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0046] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion mentioned in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0047] Throughout the specification, when a component is described as being "connected to" or "coupled to" another component, the component may be directly "connected to" or "coupled to" the other component, or one or more other components may be present in between. Conversely, when an element is described as being "directly connected to" or "directly coupled to" another element, there may be no other elements present in between. Similarly, similar expressions such as "between" and "immediately between" and "adjacent to" and "directly adjacent to" should also be interpreted in the same manner. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0048] The terms used herein are only used to describe various examples and will not be used to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "include", "comprising", and "having" specify the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by ordinary technicians in the field to which the present disclosure belongs and based on the understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant field and the disclosure of the present application, and will not be interpreted in an idealized or overly formal sense.

[0050] Furthermore, in the description of the exemplary embodiments, when it is considered that a detailed description of thus-known structures or functions will lead to obscure interpretation of the exemplary embodiments after understanding the disclosure of the present application, such description will be omitted.

[0051] The following example embodiments can be implemented in various forms of products (e.g., personal computers (PCs), laptop computers, tablet PCs, smart phones, televisions (TVs), smart home appliances, smart vehicles, self-service terminals, and wearable devices). Hereinafter, examples will be described in detail with reference to the accompanying drawings, and the same reference numerals in the drawings always represent the same elements.

[0052] Figure 1 is a diagram illustrating an example of an integration method of combining a plurality of artificial neural network models.

[0053] Ensemble methods can be used to combine the individual outputs of multiple models and obtain a single final output. The models described herein may include models implemented by artificial neural networks. For example, the model may be a sequence-to-sequence model implemented by an artificial neural network.

[0054] The sequence-to-sequence model can be used in the field of various applications that require technology for outputting a sequence of a different domain from the input sequence. The sequence-to-sequence model may include an encoder and a decoder. The encoder may receive the input sequence in sequence order and may form a single vector by compressing the entire input sequence. The single vector may be referred to as a context vector. When the input sequence is compressed into a single context vector, the encoder may send the context vector to the decoder. The decoder may then receive the context vector and generate an output sequence.

[0055] The sequence-to-sequence model can be mainly used for speech recognition. For example, a speech recognition model implemented with a sequence-to-sequence architecture can generate a recognition result corresponding to the input speech from the input speech in response to the input speech. For example, the recognition result can be a text corresponding to the input speech. In the following, for ease of description, the integration method used in such a speech recognition model implemented with a sequence-to-sequence architecture is described as an example of the integration method. However, the example of the model is not limited to the speech recognition model implemented with a sequence-to-sequence architecture, and various types of models or devices may be applicable and adopted.

[0056] Reference Figure 1 The speech recognition model may include an acoustic model and a language model combined with each other in the speech recognition model. The speech recognition model may combine an acoustic model and a language model, wherein the acoustic model is configured to receive speech information (e.g., a speech sequence) as input and output text (e.g., a word sequence) as a recognition result, and the language model is configured to predict the probability of a word sequence. The speech recognition model may thereby improve the performance of speech recognition. For example, Figure 1 As shown in , the acoustic model includes an encoder 110 and an acoustic model decoder 120, and the language model includes a language model decoder 130. According to an example, the acoustic model may indicate only the acoustic model decoder 120.

[0057] Hereinafter, the operation of the acoustic model will be described. The encoder 110 receives a speech feature vector 112 obtained by converting the speech information 111 into the form of a vector corresponding to the speech information 111 without directly receiving and processing the speech information 111. The encoder 110 receives the speech feature vector 112 in a sequential order, and then finally compresses all sets of the speech information 111 to generate a context vector 113. The context vector 113 may also be referred to as a feature or encoded feature including the speech information 111. When the speech information 111, which is time-based information, is all compressed into a single context vector 113, the encoder 110 sends the context vector 113 to the acoustic model decoder 120. The acoustic model decoder 120 receives the context vector 113 and outputs the recognized words one by one in a sequential order.

[0058] The acoustic model decoder 120 may perform autoregressive decoding, which refers to a decoding method that determines an output token based on tokens that have been recognized up to the previous step of each token, and may be used to calculate an output having an indefinite length through an artificial neural network.

[0059] The acoustic model decoder 120 receiving the context vector 113 receives the start lexical unit SOS indicating the beginning of the sentence as an initial input. The acoustic model decoder 120 receiving the start lexical unit SOS predicts the lexical unit that is most likely to appear next. The term "lexical unit" used herein indicates a unit that forms a single sequence, and the unit can be a word unit, a subword unit, and a character unit. In the following, for ease of description, the lexical unit will be described as an indicator word unit. In the first time step, the acoustic model decoder 120 predicts "Hi" as the next word to appear. That is, the acoustic model decoder 120 determines "Hi" as an output lexical unit and outputs the output lexical unit as a speech recognition result. The acoustic model decoder 120 inputs "Hi" determined as an output lexical unit as the input of the next time step. In the second time step, the acoustic model decoder 120 predicts "Bixby" as the next word from the input word "Hi". The acoustic model decoder 120 inputs "Bixby" as the input of the next time step. The acoustic model decoder 120 may predict the next word occurrence as described above, and input the predicted word as input for the next time step, and repeat such prediction and input until the end token EOS indicating the end of the sentence is predicted as the next word.

[0060] The acoustic model decoder 120 may obtain output lexical units based on information calculated from the encoder 110 in each step. When obtaining output lexical units, the acoustic model decoder 120 may obtain the output lexical units based on the input lexical units that have been selected until the previous step. For example, when the acoustic model decoder 120 receives lexical units, the acoustic model decoder 120 may predict the probability of each candidate output lexical unit as a candidate for the lexical unit based on the received lexical units, and determine the final output lexical unit based on the predicted probability. In this example, the acoustic model decoder 120 may select a candidate output lexical unit with the maximum probability among the candidate output lexical units as the final output lexical unit.

[0061] The candidate output token t in step i i The conditional probability of can be expressed by Equation 1.

[0062] [Equation 1]

[0063] p(t i |t l , t 2 , ..., t i-1 , H e )

[0064] In Equation 1, H e represents the hidden representation of encoder 110 and may correspond to context vector 113. In addition, t 1 to i-1 Denotes the tokens that have been recognized up to the current time step. The acoustic model decoder 120 may determine the next output token based on the context vector 113 and the recognized token.

[0065] The speech recognition model can improve the performance of speech recognition by additionally using a language model that predicts the probability of a word sequence together with the sound model. The language model can predict the probability of an entire sentence. Previously, when the previous word was given, the language model could predict the probability of the next word appearing. The language model can be configured using only the language model decoder 130 without an additional encoder and can be trained using only text.

[0066] When there are similar-sounding words, the language model can achieve more accurate speech recognition. For example, when the word "Minchul" and the word "feels" are given, the language model can determine that the word "happy" is more likely to follow the given word than the word "habit". That is, the language model can determine that the probability of the word "happy" following the given word is greater than the probability of the word "habit" following the given word.

[0067] Acoustic models can process the relationship between signal characteristics and language elements and, for example, estimate the pronunciation of a speech signal. Language models can estimate the grammatical or semantic correctness of a word or sentence.

[0068] The encoder 110, the acoustic model decoder 120, and the language model decoder 130 may be implemented by an artificial neural network (e.g., a recurrent neural network (RNN)). The encoder 110 may be an RNN configured to receive a sequence (e.g., speech) as an input. The acoustic model decoder 120 and the language model decoder 130 may be RNNs configured to output a sequence (e.g., text as a result of speech recognition). In addition to RNNs, the encoder 110, the acoustic model decoder 120, and the language model decoder 130 may also be implemented by, for example, a deep neural network (DNN) and a recurrent deep neural network (RDNN).

[0069] The artificial neural network may include multiple layers, each layer including multiple neurons. Neurons in adjacent layers may be connected via synapses. Through training, weights may be assigned to the synapses, and the parameters may include such weights.

[0070] The layers of the artificial neural network may include an input layer, a hidden layer, and an output layer. For example, the input layer may receive input for training or recognition and send the received input to the hidden layer, and the output layer may generate an output of the artificial neural network based on the signal received from the neurons of the hidden layer. The hidden layer may change the training data or target recognition data (as the target to be recognized) sent through the input layer into a predictable value, wherein the hidden layer may be an intermediate layer between the input layer and the output layer.

[0071] The speech recognition model may generate a sequence of final recognition results from a sequence of input speech using an integration of the acoustic model decoder 120 and the language model decoder 130 .

[0072] The acoustic model decoder 120 and the language model decoder 130 may output their respective recognition results in units of lexical units, and generate a final recognition result by combining the recognition results based on an integrated weight. For example, the acoustic model decoder 120 may determine a candidate output lexical unit based on the input speech and a previously determined recognition result, and the language model decoder 130 may determine a candidate output lexical unit based on a previously determined recognition result. The speech recognition model may generate a final recognition result by calculating the weighted sum of the probabilities of each candidate output lexical unit based on a preset integrated weight.

[0073] For example, the acoustic model decoder 120 and the language model decoder 130 may output candidate output lexical units "eight" and "ate" based on the previously determined output lexical units "my", "number", and "is". The acoustic model decoder 120 may predict the probability of the candidate output lexical unit "eight" and the probability of the candidate output lexical unit "ate" as 0.35 and 0.4, respectively. The language model decoder 130 may predict the probability of the candidate output lexical unit "eight" and the probability of the candidate output lexical unit "ate" as 0.8 and 0.001, respectively. In this example, when the integrated weight ratio of the acoustic model decoder 120 and the language model decoder 130 is 0.8:0.2, the candidate output lexical unit "eight" may have a final weight of 0.44 (0.8×0.35+0.2×0.8=0.44), and the candidate output lexical unit "ate" may have a final weight of 0.3202 (0.8×0.4+0.2×0.001=0.3202). Therefore, the speech recognition model may determine the candidate output lexical unit "eight" with a larger weight as the final output lexical unit.

[0074] In an integration method for combining the outputs of multiple models and obtaining a single final output, the model assigned with the largest weight when calculating the weighted sum may be referred to as a primary model, and models other than the primary model may be referred to as auxiliary models. For example, in a speech recognition model, the acoustic model may be the primary model, and the language model may be the auxiliary model.

[0075] In the case where the final recognition result is generated based only on the weighted sum of the model, when the prediction result of the auxiliary model is not absolutely large enough to overturn the result of the weighted sum, the expected result may not be obtained even if the prediction result is correct. For example, in the case where the acoustic model and the language model are trained or modeled by an artificial neural network configured to train or learn training data, when a rare lexical unit in the context that does not appear frequently in the training data is the correct answer, the problem described above is more likely to be experienced.

[0076] For example, when the input speech is "My favorite animal is sloth", because the word "slots" and the word "sloth" in the input speech have similar pronunciations, the acoustic model can predict the probability of the word "slots" which is more commonly used and often learned as the correct answer as 0.4. In this example, the acoustic model can predict the probability of the word "sloth" as the correct answer as 0.306 (see Table 1 below). In Table 1, y * represents the candidate output lexical unit, P AsR (y *|…) represents the candidate output lexical unit y predicted by the acoustic model * The probability, P LM (y * |…) represents the candidate output lexical unit y based on the language model prediction * probability.

[0077] [Table 1]

[0078] <![CDATA[y * ]]> <![CDATA[P ASR (and * |…)]]> <![CDATA[P LM (and * |…)]]> sloth 0.306 0.05 slots 0.4 0.003 … … … dog 0.05 0.3 cat 0.03 0.24 … … …

[0079] In contrast, by considering the portion of the input speech "my favorite animal is", the language model may predict the probability of the word "sloth", which indicates the kind of animal and is a grammatically and semantically natural word or sentence, as 0.05. In addition, the language model may predict the probability of the word "slots" as 0.003. However, referring to Table 1, when only given previous word training or learning is used, the language model may predict the probability of the word "dog", which indicates the kind of animal and is often seen in the training data, as 0.3. Compared with the respective probabilities of the word "sloth" and the word "slots", the predicted probability of the word "dog" of 0.3 is an absolutely larger value.

[0080] In this case, the acoustic model as the main model may have a larger weight than the language model as the auxiliary model, so even if the language model predicts the word "sloth" as having a greater probability than the word "slots" selected by the main model, the word "slots" may be determined as the integration result.

[0081] Therefore, in order to solve the problem described in the above content, the integration method can adjust the probability of the candidate output lexical units of the auxiliary model based on the probability of the candidate output lexical units of the main model, and determine the final output lexical unit by calculating the weighted sum using the probability adjusted based on the main model.

[0082] Figure 2 is a diagram illustrating an example of an integrated model.

[0083] Reference Figure 2 , the integrated model includes a first model 210, a second model 220, a regulator 230, and an integrator 250. According to an example, the integrated model may further include a normalizer 240.

[0084] The integrated model can be configured to perform the above reference Figure 1 The integrated model can obtain a single output by combining the output of the first model 210 and the output of the second model 220. The first model 210 can be the same as the one described above. Figure 1The main model described above (eg, acoustic model), the second model 220 may be Figure 1 Auxiliary model described.

[0085] The adjuster 230 may use the ranking of the candidate output tokens in the first model 210 to re-adjust the predicted probability values ​​of the second model 220. Figures 3 to 5 The detailed operation of the regulator 230 will be described.

[0086] The normalizer 240 may perform normalization so that the sum of the adjusted probabilities of the candidate output lexical units of the second model 220 is 1. The probability value obtained by the weighted sum may be used for another purpose in addition to determining the final output lexical unit. For another purpose, the normalizer 240 may normalize the sum of the probabilities of the candidate output lexical units to 1 and adjust the probabilities to have meaningful values. Figure 6 The detailed operation of the normalizer 240 is described.

[0087] Figure 3 is a flow chart illustrating an example of the operation of an integrated model.

[0088] In the following we will refer to Figure 3 Operations 310 to 340 described above can be referred to in detail. Figure 2 The integrated model described herein can be implemented by one or more hardware modules, one or more software modules, or various combinations thereof.

[0089] Reference Figure 3 In operation 310, the first model predicts a first probability of a candidate output lexical unit of the first model. As described above, the first model may be a main model of the integrated model, and may be an acoustic model of the speech recognition model. The acoustic model may predict candidate output lexical units that are candidates for output lexical units, and predict the probability of each candidate.

[0090] In operation 320, a second model interacting with the first model predicts a second probability of a candidate output lexical unit of the second model. The second model may be an auxiliary model of the integrated model, and may be a language model of the speech recognition model. The language model may predict candidate output lexical units that are candidates for output lexical units, and predict the probability of each candidate. The acoustic model and the language model may predict candidate output lexical units based on the same set of candidate output lexical units.

[0091] In operation 330, the regulator adjusts the second probability based on the first probability. As described above, in the case where the final recognition result is generated based only on the weighted sum of each model, when the prediction result of the auxiliary model is not absolutely large enough to overturn the result of the weighted sum, even if the prediction result is correct, the expected result may not be obtained. Therefore, the regulator can adjust the second probability of the candidate output lexical unit of the second model based on the first probability of the candidate output lexical unit of the first model.

[0092] For example, when the input speech is "my favorite animal is a sloth", the language model or the second model may predict the probability of the candidate "sloth" to be greater than the probability of the candidate "slots". However, such a predicted probability may not be absolutely large enough to overturn the result of the weighted sum calculation, and therefore, the candidate "slots" selected by the acoustic model or the first model may be determined as the integrated result (see Table 2 below).

[0093] In one example, the weighted sum calculation may include a method of calculating the weighted sum of each probability value to be subjected to the weighted sum calculation, and a method of calculating the weighted sum of the values ​​obtained by applying the logarithm to each probability. Hereinafter, for ease of description, the method of calculating the weighted sum of each probability value to be subjected to the weighted sum calculation will be mainly described.

[0094] [Table 2]

[0095]

[0096] In order to solve the problem described in the above content, the regulator can adjust the prediction value of the language model or the second model to generate a final recognition result from the candidates (e.g., the first n candidates, n is a positive integer greater than or equal to 1 and less than the number of candidates), the probability of which is highly predicted by the acoustic model or the first model. The regulator can use a predetermined function to re-adjust the second probability of the second model based on the ranking of the candidate output lexical units of the first model. In the following, reference will be made to Figure 4 and Figure 5 A method for adjusting a second probability based on a first probability is described below.

[0097] Figure 4 is a flow chart showing an example of a method for adjusting a second probability based on a first probability. Figure 4 Operations 410 to 430 described above can be referred to in detail. Figure 2 and Figure 3 The described regulator is implemented.

[0098] Reference Figure 4In operation 410, the regulator determines the ranking of the candidate output lexical units based on the first probabilities of the candidate output lexical units of the first model. The regulator may determine the ranking of the candidate output lexical units in a sequential order starting from the candidate output lexical unit with the largest first probability. In one example, the regulator may rank the candidate output lexical units in descending order from the candidate output lexical unit with the highest first probability to the candidate output lexical unit with the lowest first probability.

[0099] In operation 420, the regulator extracts target candidate output lexical units on which regulation will be performed from the candidate output lexical units based on the determined ranking. The regulator may extract the target candidate output lexical units based on a threshold coefficient determined according to the ranking. The regulator may determine all candidate output lexical units as target candidate output lexical units, or determine a part of the candidate output lexical units as target candidate output lexical units. For example, the regulator may determine the first n candidate output lexical units among all candidate output lexical units. Alternatively, the regulator may determine the first n candidate output lexical units and the last n candidate output lexical units among all candidate output lexical units. Alternatively, the regulator may determine all candidate output lexical units as target candidate output lexical units.

[0100] In operation 430, the regulator uses a function determined based on the ranking to adjust the second probability corresponding to the extracted target candidate output lexical unit. The regulator can adjust the second probability of the target candidate output lexical unit based on a proportional coefficient determined according to the ranking. The function used in operation 430 can be implemented in various ways (e.g., a linear function or a nonlinear function). For example, with reference to Table 2 above, a function is used that increases the respective second probabilities of the first n candidate output lexical units by 10 times and increases the respective second probabilities of the remaining candidate output lexical units by 0.1 times. As a result of the increase, the second probability of the candidate "sloth" as one of the first n candidate output lexical units increases from 0.05 to 0.5, and the second probability of the candidate "slots" as another of the first n candidate output lexical units increases from 0.003 to 0.03. In addition, the second probability of the candidate "dog" not included in the first n candidate output tokens decreases from 0.3 to 0.03, and the second probability of the candidate "cat" not included in the first n candidate output tokens decreases from 0.24 to 0.024.

[0101] Because the correct answer is likely to exist among the first n candidate output lexical units predicted by the first model (or main model), although they may not be completely correct, the regulator can adjust the probability of the candidate output lexical units of the second model (or auxiliary model) so that the first n candidate output lexical units predicted by the first model have a sufficiently large probability. Figure 5 Describe the function that makes the top n candidate output tokens predicted by the first model have sufficiently large probabilities.

[0102] Figure 5 is a graph showing an example of a function for adjusting the second probability.

[0103] When a lexical unit that does not appear frequently in the training data is the correct answer or truth, the predictive performance of the language model that predicts the output lexical unit based on the context and the previous output lexical unit can be significantly reduced. At the same time, the acoustic model can predict the output lexical unit based on the input speech information, so even in such a case, it can have a consistent level of predictive performance. In one example, the function used here can be designed to reflect such a situation.

[0104] Reference Figure 5 , the function used here can be multiplying the second probability of each candidate in the top 20% of the candidates in the ranking determined based on the first probability by a factor of ten (e.g., a factor / weight of 1), maintaining the second probability of each candidate in the top 20% to 60% of the candidates in the ranking, and multiplying the second probability of each of the top 60% to 100% of the candidates in the ranking by a factor of 0.1.

[0105] By this function, the second probability of each candidate in the top 20% of the candidates in the ranking determined based on the first probability can be increased tenfold, so it is more likely that the final output lexical unit is determined among the top 20% of the candidates in the ranking. In addition, the candidate with the largest second probability among the top 20% of the candidates (for example, the candidate that best fits the context among the candidates determined based on the input speech) can be determined as the final output lexical unit.

[0106] Return to reference Figure 3 , in operation 340, the integrator determines the final output lexical unit among the candidate output lexical units based on the first probability and the adjusted second probability. For example, the integrator may calculate the weighted sum of the first probability and the adjusted second probability based on a weight ratio (e.g., 1:0.3). In this example, referring to Table 2 above, the integrator may output the final weight of 0.456 for the candidate "sloth" by obtaining the weighted sum of the first probability 0.306 and the adjusted second probability 0.5, and output the final weight of 0.409 for the candidate "slots" by obtaining the weighted sum of the first probability 0.4 and the adjusted second probability 0.03. In this example, the integrator may determine the candidate "sloth" with the maximum weight as the final output lexical unit.

[0107] In the case where the final recognition result is generated based only on the weighted sum of each model, the candidate "slots" may be determined as the final output lexical unit. However, the integrated model may determine the candidate "sloth" as the final output lexical unit by adjusting the probability of the candidate output lexical unit of the auxiliary model based on the probability of the candidate output lexical unit of the main model.

[0108] Figure 6 is a diagram showing an example of normalizing the adjusted second probability.

[0109] The normalizer may be configured to normalize the adjusted second probabilities so that the sum of the adjusted second probabilities is 1. In addition to the purpose of determining the final output token, the probability value obtained by the weighted sum may be used for another purpose. The normalizer may perform normalization so that the sum of the probabilities of the candidate output tokens is 1, and the normalizer may adjust the probabilities to have meaningful values, rather than simply using the probabilities to determine the final output token.

[0110] For example, in the case of a beam search, the probabilities of the candidate output tokens may be used not only to determine the final output token, but also to be used later. The beam search may be used to compare the scores of the candidate sentences, and the score of the entire sentence may be modeled or trained using the product of the probabilities of each token. When modeling the beam search, the sum of the probabilities of the candidate output tokens may need to be 1. Therefore, in order for the integrator to perform the beam search, the adjusted second probability may need to be normalized.

[0111] Figure 6 An example of beam searching is shown.

[0112] For example, as shown in the figure, the words "red", "dog", "blue", and "runs" may be generated as candidate words that may follow the words "a", "the", and "red" in the first time step. A preset number of candidate words that are most likely to follow the words in the first time step may be selected in sequence order from among the generated words. For example, when the preset number is 3, the words "red", "dog", and "blue" may be selected, and the word "runs" may be excluded.

[0113] In addition, the words "dog", "dog" and "cat" may be generated as candidate words that may follow the words "red", "dog" and "blue" in the second time step. A preset number of candidate words may also be selected in sequence order from among the generated candidate words.

[0114] Through the beam search as described above, the recognition result “a red dog runs quickly today 610” can be effectively obtained without searching all candidate paths.

[0115] Figure 7 is a diagram illustrating an example of an apparatus for determining an output lexical unit.

[0116] Reference Figure 7 The device 700 for determining an output lexical unit includes one or more sensors 710, a processor 730, and a communication interface 770. The device 700 also includes a memory 750 and a display 790. The sensor 710, the processor 730, the memory 750, the communication interface 770, and the display 790 can communicate with each other via a communication bus 705. The device 700 may include the above reference Figures 1 to 6 Describe the integrated model.

[0117] For example, the sensor 710 may include a microphone sensor and a voice or sound sensor.

[0118] The processor 730 may execute the above reference Figures 1 to 6 The processor 730 may execute a program and control the device 700. The code of the program to be executed by the processor 730 may be stored in the memory 750.

[0119] For example, the processor 730 may be a central processing unit (CPU) or a graphics processing unit (GPU).

[0120] Processor 730 can predict a first probability for each candidate output lexical unit of the first model, predict a second probability for each candidate output lexical unit of the second model, adjust the second probability based on the first probability, and determine a final output lexical unit among the candidate output lexical units based on the first probability and the adjusted second probability.

[0121] The processor 730 may determine the ranking of the candidate output lexical units of the first model based on the first probability, and may adjust the second probability based on the determined ranking. In one example, the processor 730 may rank the candidate output lexical units by grouping the candidate output lexical units into two or more ranking groups based on the first probability of the candidate output lexical units, and may determine different weights for each ranking group. Based on the ranking of each candidate output lexical unit, the weight of each candidate output lexical unit may be determined to be greater than 1, 1, or between 0 and 1. The processor 730 may adjust the second probability using a function determined based on the ranking. The processor 730 may adjust the second probability based on a proportional coefficient determined according to the ranking. The processor 730 may determine the ranking of the candidate output lexical units of the first model based on the first probability, extract the target candidate output lexical units to be adjusted from the candidate output lexical units based on the ranking, and adjust the second probability corresponding to the extracted target candidate output lexical units based on the ranking. The processor 730 may normalize the adjusted second probability. The processor 730 may normalize the second probability so that the sum of the adjusted second probabilities is 1. The processor 730 may calculate a weighted sum of the first probability and the adjusted second probability, and determine the candidate output lexical unit with the largest weighted sum among the candidate output lexical units as the final output lexical unit. The processor 730 may predict the first probability based on the output lexical unit previously determined according to the first model and the second model, and may predict the second probability based on the output lexical unit previously determined.

[0122] The memory 750 may store data processed by the processor 730. For example, the memory 750 may store a program. The program to be stored may be a set of grammars that can be executed by the processor 730 by being encoded to perform speech recognition. The memory 750 may be a volatile memory or a nonvolatile memory.

[0123] The communication interface 770 may transmit and receive data by connecting to the sensor 710, the processor 730, and the memory 750. The communication interface 770 may transmit and receive data by connecting to an external device. The term "transmitting and receiving A" used herein may indicate "transmitting and receiving information or data indicating A".

[0124] The communication interface 770 may be implemented by a circuit in the device 700. For example, the communication interface 770 may include an internal bus and an external bus. For another example, the communication interface 770 may be an element that connects the device 700 and another external device. The communication interface 770 may receive data from the external device and send the received data to the processor 730 and the memory 750.

[0125] The display 790 may display the decoding result. For example, the display 790 may display the result of outputting the lexical unit.

[0126] In this regard Figure 2 and Figure 7 The described equipment and other equipment, units, modules, devices and other components for determining output lexical units are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components can also access, operate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor or a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. The hardware components may have any one or more of different processing configurations, where examples of different processing configurations include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0127] Figure 1 and Figures 3 to 6The method for performing the operations described in the present application shown in the is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented to execute instructions or software as described above to perform the operations described in the present application performed by the method. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0128] The instructions or software for controlling a processor or computer to implement a hardware component and perform the method as described above are written as a computer program, a code segment, an instruction, or any combination thereof, to individually or collectively instruct or configure a processor or computer to operate as a machine or a special-purpose computer to perform the operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by a processor or computer. In another example, the instructions or software include high-level code executed by a processor or computer using an interpreter. A programmer of ordinary skill in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods as described above.

[0129] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R, B ... D-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk and any other device, wherein any other device is configured to store instructions or software and any associated data, data files and data structures in a non-temporary manner, and provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0130] Although the present disclosure includes specific examples, it will be apparent that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered to be illustrative only and not for purposes of limitation. The description of features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents.

[0131] Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as being included in the disclosure.

Claims

1. A method for determining an output lexical unit, the method comprising: Predicting a first probability of each candidate output lexical unit of a first model, wherein the first model is an acoustic model of a speech recognition model for recognizing input speech; Predicting a second probability for each candidate output lexical unit of a second model interacting with the first model, wherein the second model is a language model of the speech recognition model; adjusting a second probability of each candidate output token based on the first probability; and determining an output lexical unit among the candidate output lexical units based on the first probability and the adjusted second probability, The adjustment steps include: Determining a ranking of candidate output tokens of the first model based on the first probability; and The second probability is adjusted based on the ranking.

2. The method according to claim 1, wherein: The step of adjusting the second probability based on the ranking comprises: The second probability is adjusted using a function determined based on the ranking.

3. The method according to claim 1, wherein: The step of adjusting the second probability based on the ranking comprises: The second probability is adjusted based on one or both of a scaling factor and a threshold factor determined according to the ranking.

4. The method according to claim 1, wherein: The step of adjusting the second probability based on the ranking comprises: extracting, from among the candidate output tokens, a target candidate output token on which conditioning is to be performed based on the ranking; and A second probability corresponding to the extracted target candidate output lexical unit is adjusted.

5. The method according to claim 4, wherein: The step of adjusting the second probability corresponding to the extracted target candidate output lexical unit comprises: A second probability corresponding to the extracted target candidate output lexical unit is adjusted using a function determined based on the ranking.

6. The method according to claim 1, further comprising: The adjusted second probability is normalized.

7. The method according to claim 6, wherein: The normalization steps include: The adjusted second probabilities are normalized so that the sum of the adjusted second probabilities is 1.

8. The method according to claim 1, further comprising: performing a beam search to search for a preset number of candidate output lexical units among the candidate output lexical units; as well as The adjusted second probability is normalized when performing the beam search.

9. The method according to claim 1, wherein: The steps of determining the output token include: calculating a weighted sum of the first probability and the adjusted second probability; and A candidate output lexical unit having a maximum weighted sum among the candidate output lexical units is determined as the output lexical unit.

10. The method according to claim 6, wherein: The weight applied to the first probability is greater than the weight applied to the adjusted second probability.

11. The method according to claim 1, wherein: The steps for predicting the first probability include: predicting a first probability based on output tokens previously determined by the first model and the second model, The step of predicting the second probability includes: A second probability is predicted based on the output token previously determined by the first model.

12. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 11.

13. A device for determining an output lexical unit, the device comprising: One or more processors configured to: Predicting a first probability of each candidate output lexical unit of a first model, wherein the first model is an acoustic model of a speech recognition model for recognizing input speech; Predicting a second probability for each candidate output lexical unit of a second model interacting with the first model, wherein the second model is a language model of the speech recognition model; adjusting a second probability of each candidate output token based on the first probability; and determining an output lexical unit among the candidate output lexical units based on the first probability and the adjusted second probability, Wherein, the one or more processors are configured to: Determining a ranking of candidate output tokens of the first model based on the first probability; and The second probability is adjusted based on the ranking.

14. The device according to claim 13, wherein: The one or more processors are configured to: The second probability is adjusted using a function determined based on the ranking.

15. The apparatus according to claim 13, wherein: The one or more processors are configured to: The second probability is adjusted based on one or both of a scaling factor and a threshold factor determined according to the ranking.

16. The apparatus according to claim 13, wherein: The one or more processors are configured to: extracting, from among the candidate output tokens, a target candidate output token on which conditioning is to be performed based on the ranking; and A second probability corresponding to the extracted target candidate output lexical unit is adjusted.

17. The device according to claim 16, wherein: The one or more processors are configured to: A second probability corresponding to the extracted target candidate output lexical unit is adjusted using a function determined based on the ranking.

18. The apparatus of claim 13, wherein: The one or more processors are configured to: The adjusted second probability is normalized.

19. The apparatus according to claim 18, wherein: The one or more processors are configured to: The adjusted second probabilities are normalized so that the sum of the adjusted second probabilities is 1.

20. The apparatus of claim 13, wherein: The one or more processors are configured to: performing a beam search to search for a preset number of candidate output tokens among the candidate output tokens; and The adjusted second probability is normalized when performing the beam search.

21. The apparatus of claim 13, wherein: The one or more processors are configured to: calculating a weighted sum of the first probability and the adjusted second probability; and A candidate output lexical unit having a maximum weighted sum among the candidate output lexical units is determined as the output lexical unit.

22. The apparatus according to claim 21, wherein The weight applied to the first probability is greater than the weight applied to the adjusted second probability.

23. The apparatus of claim 13, wherein: The one or more processors are configured to: predicting a first probability based on output tokens previously determined according to the first model and the second model; as well as A second probability is predicted based on the output token previously determined according to the first model.

24. A device for determining a lexical unit, comprising: One or more processors configured to: Ranking the candidate tokens based on the first probabilities of the candidate tokens; Determining a weight of each candidate token based on the ranking of each candidate token; adjusting the second probability of the candidate token by weighting each second probability with a corresponding weight; Calculate a weighted sum of each candidate lexical unit based on the first probability and the adjusted second probability; as well as Output the candidate token with the maximum weighted sum.

25. The apparatus of claim 24, wherein: The one or more processors are configured to: ranking the candidate tokens by grouping the candidate tokens into two or more ranked groups based on the first probabilities of the candidate tokens; as well as Determine the different weights for each ranking group.

26. The apparatus of claim 24, wherein: The one or more processors are configured to: The weight of each candidate token is determined such that a first candidate token having a first probability higher than a first probability of a second candidate token has a greater weight than a weight of the second candidate token.

27. The apparatus of claim 24, wherein: The one or more processors are configured to: Based on the ranking of each candidate token, a weight of each candidate token is determined to be greater than 1, 1, or between 0 and 1.

28. The apparatus of claim 24, wherein: The one or more processors are configured to: The candidate tokens are ranked in descending order from the candidate token with the highest first probability to the candidate token with the lowest first probability.

Images