Multilingual speech recognition method and device

Abstract

A multilingual speech recognition method and device are disclosed. A multilingual speech recognition method according to one embodiment may comprise the steps of: determining, on the basis of a language identification result of a target token obtained as a speech recognition result of inputted speech data, a language identity score of the target token; and identifying text data corresponding to the speech data on the basis of the language identity score of the target token and a probability that the target token corresponds to the speech data.

Description

Multilingual speech recognition method and device

[0001] The following embodiments relate to a multilingual speech recognition method and device.

[0002] Speech recognition technology converts speech signals generated by speech into text data and processes them. It is also known as STT (Speech-to-Text). As voice recognition technology has enabled voice input into devices, it is being applied in diverse fields, including voice-based device control and information retrieval. Research is also actively underway to complement the application of speech recognition technology, including research on speech recognition algorithms utilizing machine learning to enhance speech recognition performance, technologies for separating individual voices from speech signals containing multiple speakers, and technologies for identifying speakers from speech signals.

[0003] The following examples provide techniques for improving the accuracy of speech recognition in a multilingual speech recognition model by introducing conditions for maintaining linguistic consistency among tokens. However, the following examples are not limited to the technical aspects described above, and other technical aspects may exist.

[0004] A multilingual speech recognition method according to one embodiment includes the steps of: determining a language homogeneity score of a target token based on a language identification result of a target token obtained as a result of speech recognition of input speech data; and identifying text data corresponding to the speech data based on the language homogeneity score of the target token and a probability that the target token corresponds to the speech data.

[0005] The step of identifying text data corresponding to the speech data may include the step of determining an automatic speech recognition (ASR) score of the target token based on a probability that the target token corresponds to the speech data; the step of correcting the ASR score of the target token based on a language identity score of the target token; and the step of identifying text data corresponding to the speech data based on the ASR score of the target token.

[0006] The step of determining the language identity score of the target token may include the step of determining the language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of the previous token sequence of the target token.

[0007] The step of determining the language identity score of the target token may include the step of determining the language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of the speech data.

[0008] The step of determining the language identity score of the target token may include the step of determining a parameter regarding the proportion of the language identity score of the target token based on a language change probability corresponding to the target token; and the step of correcting the language identity score of the target token based on the parameter.

[0009] The above parameter may be determined to a smaller value as the probability of language change of the target token increases.

[0010] A multilingual speech recognition method according to one embodiment includes the steps of: obtaining candidate text data corresponding to the speech data based on a speech recognition result of input speech data; determining a language identity score for each of the candidate text data based on a similarity of language identification results of a plurality of tokens included in the candidate text data; and identifying text data corresponding to the speech data among the candidate text data based on the language identity score of the candidate text data.

[0011] The step of determining a language identity score for each of the candidate text data may include the step of determining a language identity score for each of a plurality of tokens included in the candidate text data; and the step of determining a language identity score for the candidate text data based on a sum of the language identity scores of the plurality of tokens.

[0012] The step of determining the language identity score of each of the plurality of tokens may include the step of determining the language identity score of the token based on the similarity between the language identification result of the token included in the plurality of tokens and the language identification result of the previous token sequence of the token.

[0013] The step of determining the language identity score of each of the plurality of tokens may include the step of determining the language identity score of the token based on the similarity between the language identification result of the token included in the plurality of tokens and the language identification result of the speech data corresponding to the plurality of tokens.

[0014] The step of identifying text data corresponding to the speech data may include a step of identifying text data corresponding to the speech data among the candidate text data based on a probability that each of the candidate text data corresponds to the speech data and a language identity score of each of the candidate text data.

[0015] The step of determining the language identity score of each of the candidate text data may include the step of determining a parameter regarding the proportion of the language identity score of each of the candidate text data based on a language change probability determined corresponding to each of the tokens included in the candidate text data; and the step of correcting the language identity score of each of the candidate text data based on the parameter.

[0016] Among the above candidate text data, the parameter corresponding to the given candidate text data may be determined to be a smaller value as the probability of language change of tokens included in the given candidate text data increases.

[0017] According to one embodiment, a multilingual speech recognition device includes at least one memory storing one or more commands; and at least one processor executing the commands, wherein the commands, when executed by the at least one processor, cause the multilingual speech recognition device to determine a language homogeneity score of a target token based on a language identification result of the target token obtained as a result of speech recognition of input speech data, and to identify text data corresponding to the speech data based on the language homogeneity score of the target token and a probability that the target token corresponds to the speech data.

[0018] The above command, when executed by the at least one processor, may cause the multilingual speech recognition device to determine an automatic speech recognition (ASR) score of the target token based on a probability that the target token corresponds to the speech data, correct the ASR score of the target token based on a language identity score of the target token, and identify text data corresponding to the speech data based on the ASR score of the target token.

[0019] The above command, when executed by the at least one processor, may cause the multilingual speech recognition device to determine the language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of a previous token sequence of the target token.

[0020] The above command, when executed by the at least one processor, may cause the multilingual speech recognition device to determine the language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of the speech data when determining the language identity score of the target token.

[0021] The above command, when executed by the at least one processor, may cause the multilingual speech recognition device to determine a parameter regarding a proportion of the language identity score of the target token based on a language change probability corresponding to the target token when determining the language identity score of the target token, and to correct the language identity score of the target token based on the parameter.

[0022] According to one embodiment, a multilingual speech recognition device includes at least one memory storing one or more commands; and at least one processor executing the commands, wherein the commands, when executed by the at least one processor, cause the multilingual speech recognition device to obtain candidate text data corresponding to the voice data based on a voice recognition result of input voice data, determine a language identity score for each of the candidate text data based on a similarity of language identification results of a plurality of tokens included in the candidate text data, and identify text data corresponding to the voice data among the candidate text data based on the language identity score of the candidate text data.

[0023] The above and other aspects and features of the embodiments will become more apparent through the following description taken in conjunction with the accompanying drawings.

[0024] Figure 1 is a flowchart of the operation of a multi-lingual speech recognition method according to one embodiment.

[0025] FIG. 2A and FIG. 2B are drawings for explaining the language identification results of a token according to one embodiment.

[0026] Figures 3a and 3b illustrate a language identity score L determined based on the language identification results of a token according to one embodiment. T These are drawings illustrating graphs.

[0027] Figures 4a and 4b illustrate a language identity score L determined based on the language identification result of a token and the language identification result of speech data according to one embodiment. T These are drawings illustrating graphs.

[0028] Figure 5 is a flowchart of the operation of a multilingual speech recognition method according to one embodiment.

[0029] FIG. 6 is a diagram for explaining an operation of determining a language recognition result based on an ASR score and a language identity score according to one embodiment.

[0030] FIG. 7 is a block diagram illustrating the operation of a multilingual speech recognition device according to one embodiment.

[0031] Figure 8 is a block diagram illustrating the operation of an LHS module according to one embodiment.

[0032] Fig. 9 is an exemplary diagram of the configuration of a multilingual speech recognition device according to one embodiment.

[0033] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives included in the technical concepts described in the embodiments.

[0034] In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of the noun corresponding to an item may include one or more of said items, unless the context clearly indicates otherwise.

[0035] In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof.

[0036] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another, and do not qualify the components in any other respect (e.g., importance or order).

[0037] When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0038] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0039] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0040] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0041] Figure 1 is a flowchart of the operation of a multi-lingual speech recognition method according to one embodiment.

[0042] A multilingual speech recognition method according to one embodiment may include a method of converting speech data into text data. The multilingual speech recognition method may include a method of recognizing speech data spoken in multiple languages ​​and converting it into text data.

[0043] A multilingual speech recognition method according to one embodiment may be implemented as a multilingual speech recognition model capable of recognizing multiple languages. For example, a multilingual speech recognition model that recognizes English and Korean may output data converted from Korean text when the input speech data is speech data spoken in Korean, and may output data converted from English text when the input speech data is speech data spoken in English.

[0044] The multilingual speech recognition method can be performed by a device executing a multilingual speech recognition model. A multilingual speech recognition device executing the multilingual speech recognition method is described in detail below.

[0045] A multilingual speech recognition method according to one embodiment may include a step of determining a language homogeneity score of a target token based on a language identification result of the target token obtained as a result of speech recognition of input speech data.

[0046] A token is a unit of data output as a speech recognition result. Multilingual speech recognition models can output speech recognition results in token units. A token may correspond to text data corresponding to a portion of speech data. The tokens output in response to input speech data may correspond to time-series data. The target token may correspond to any one of multiple tokens output as a speech recognition result.

[0047] The language identity score of a token is an indicator of whether the language of the token is similar to the language of other tokens, and can be determined as a higher value as the language of the token is more similar to the language of other tokens.

[0048] The step (110) of determining a language identity score of a target token according to one embodiment may include a step of determining a language identity score of the target token based on a similarity between a language identification result of the target token and a language identification result of a previous token sequence of the target token.

[0049] The preceding token sequence of a target token may correspond to a sequence containing one or more consecutive tokens preceding the target token when the tokens are arranged in chronological order. For example, if the target token is the ith token, the preceding token sequence of the target token may correspond to a sequence containing the jth token (where j is any natural integer less than i) to the (i-1)th token.

[0050] The language identification result for a target token may include the probability that the target token corresponds to each of multiple languages. The multiple languages ​​may include languages ​​recognized by the speech recognition model. For example, the language identification result for a target token may correspond to a vector containing the probability that the target token corresponds to English and the probability that the target token corresponds to Korean.

[0051] The language identity score of the target token can be determined as a higher value as the similarity between the language identification result of the target token and the language identification result of the previous token sequence of the target token is higher.

[0052] For example, the linguistic identity score L of the i-th token T [i] can be determined as in mathematical expression 1.

[0053] [Mathematical Formula 1]

[0054]

[0055] In mathematical expression 1, M[x] is the language identification result of the xth token, which can correspond to a vector containing the probability that the xth token corresponds to each of multiple languages. sim(x, y) is the similarity between x and y, which can correspond to, for example, the calculated value of cosine similarity. merge(M[j], …, M[i-1]) is an operation to merge M[j] to M[i-1], which can correspond to, for example, the weighted average of M[j] to M[i-1]. is the result of the operation to sum the language identification results of the jth token (where j is any natural number less than i) to the (i-1)th token, which can be represented as M[j:i]. In other words, the language identity score L of the ith token T [i] can be determined based on the similarity between the weighted average of the language identification results of the previous tokens of the i-th token and the language identification result of the i-th token.

[0056] For example, referring to FIG. 2A, the text data of each token (row 2) and the language identification results of each token (rows 3 to 7) are illustrated. The language identification results of each token may include the probability that each token corresponds to German (de), English (en), French (fr), and Italian (it). If the 8th token is 'azine', the language identification result M[8] of the 8th token may include the values ​​of probability 0 for German, probability 0.9 for English, probability 0.1 for French, and probability 0 for Italian. The language identification result M[8] of the 8th token may indicate that the 8th token has the highest probability of corresponding to English.

[0057] The language identity score of the ith token can be determined based on the similarity between the language identification result M[i] of the ith token and the language identification result M[j:i-1] of the token sequence preceding the ith token (where j is any natural number less than i). For example, if the language identification results of the first to seventh tokens preceding the 8th token have many tokens that are most likely to correspond to English, the similarity between the language identification result of the 8th token and the language identification result of the previous token sequence can be determined to be high. In this case, the language identity score of the 8th token can be determined to be a high value.

[0058] For example, referring to Fig. 2b, if the 8th token is 'sin', the language identification result M[8] of the 8th token may include a probability of 0 corresponding to German, a probability of 0.1 corresponding to English, a probability of 0.9 corresponding to French, and a probability of 0 corresponding to Italian. Unlike the example illustrated in Fig. 2a, the language identification result M[8] of the 8th token may indicate that the 8th token has the highest probability of corresponding to French.

[0059] If the language identification results for the first through seventh tokens preceding the eighth token are most likely to be English, the similarity between the language identification results for the eighth token and the language identification results for the preceding token sequence may be judged to be low. In this case, the language identity score for the eighth token may be determined to be low.

[0060] Figure 3a shows the language identity score L of each token determined based on the language identification result M of each token illustrated in Figure 2a. T The graph illustrates the language identity score L of each token determined based on the language identification result M of each token illustrated in FIG. 2b. T The graph is shown. Referring to Figure 3a, if the 8th token is 'azine', the linguistic identity score L of the 8th token T [8] can be determined as 0.74. Meanwhile, referring to Fig. 3b, if the 8th token is 'sin', the linguistic identity score L of the 8th token T [8] can be determined as 0.26.

[0061] The step (110) of determining the language identity score of the target token according to one embodiment may include the step of determining the language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of the speech data corresponding to the target token. The language identification result of the speech data may include the result of determining which language the input speech data corresponds to. For example, the language identification result of the speech data may be obtained by a separate language identification model. For example, the language identification result of the speech data may correspond to data (e.g., a one-hot vector) indicating the language corresponding to the speech data. For example, the language identification result of the speech data may correspond to a vector including the probability that the speech data corresponds to each of a plurality of languages.

[0062] For example, if the language identification result of the speech data includes the values ​​of probability 0 for German, probability 1 for English, probability 0 for French, and probability 0 for Italian, then the language identity score L of each token is calculated based on the similarity between the language identification result M of each token as illustrated in Fig. 2a and the language identification result of the speech data. T The language identification result M of each token as illustrated in Fig. 2b can be determined as shown in the graph in Fig. 4a. In addition, the language identity score L of each token is determined based on the similarity between the language identification result M of each token as illustrated in Fig. 2b and the language identification result of the speech data. T It can be determined as shown in the graph in Fig. 4b. Referring to Fig. 4a, if the 8th token is 'azine', the linguistic identity score L of the 8th token T [8] can be determined as 0.9. Meanwhile, referring to Fig. 4b, if the 8th token is 'sin', the linguistic identity score L of the 8th token T [8] can be determined as 0.1.

[0063] The step (110) of determining a language identity score of a target token according to one embodiment may include a step of determining a parameter regarding a proportion of the language identity score of the target token based on a language change probability determined corresponding to the target token, and a step of correcting the language identity score of the target token based on the parameter.

[0064] The probability of a language change corresponding to a target token may correspond to the probability that the language corresponding to the target token is different from the language corresponding to other tokens. For example, if the speech data is "Play the popular Bruno Mars song these days," the probability of a language change for the token corresponding to "Bruno Mars" in English may be determined to be higher than the token corresponding to other Korean speech data.

[0065] The parameter regarding the weight of the target token's language identity score can be determined as a smaller value as the probability of the target token's language change increases. For example, the parameter regarding the weight of the target token's language identity score can be determined as the inverse of the target token's language change probability.

[0066] In one embodiment, the language change probability of a target token can be obtained by a model that estimates the language change probability of the token. For example, the model that estimates the language change probability of the token may be an n-gram model. For example, the model that estimates the language change probability of the token may be a neural network trained to estimate the language change probability.

[0067] For example, the probability of a target token changing language can be obtained using a language model used in speech recognition (e.g., a count-based language model such as an n-gram-based language model, a learning-based language model, etc.). For example, the probability of a target token changing language can be obtained using a prediction network of a speech recognition model with a transducer structure or a decoder of a speech recognition model with a transformer structure.

[0068] For example, the probability that the i-th token is in a specific language is can be obtained as follows. can correspond to the probability of a language model. The probability of a language model is given by a token sequence w0: w in the training corpus. i-1 w after l is a value indicating the probability of w appearing. l can mean a token corresponding to a specific language l. For example, the specific language may be English, and the probability that the ith token is English Each token w corresponding to English l Probability of language model for can be determined by the sum of

[0069] For example, the probability of a language model can be determined as an appropriate probability value even for unseen contexts.

[0070] If the probability of a language model corresponding to a token decreases sharply—in other words, if entropy increases—then the token can be judged to have a high probability of language change. Increasing entropy can mean that the probability of a language model corresponding to a specific token is lower than the probability of a language model corresponding to a preceding token.

[0071] For example, the linguistic identity score L of the i-th token T [i] can be determined as in mathematical expression 2.

[0072] [Equation 2]

[0073]

[0074] In mathematical expression 2, α[i] may correspond to a parameter regarding the proportion of the language identity score of the i-th token.

[0075] For example, if the speech data is data that includes an English word in the middle of a Korean sentence, such as “Play the Bruno Mars song that’s popular these days,” the probability of a language change for a token corresponding to the English word “Bruno Mars” may be determined to be a higher value by an n-gram model, etc., than for tokens corresponding to other Korean speech data. The parameter α[i] regarding the proportion of the identity score of token i corresponding to “Bruno Mars” with a high probability of language change may be determined to a smaller value than that for other tokens. The proportion of the language identity score reflected in the final score for determining whether token i is determined as a speech recognition result due to the parameter α[i] regarding the language proportion may be smaller than the proportion of the ASR score. By lowering the proportion of the language identity score of a token with a high probability of language change through the parameter regarding the language proportion, a token corresponding to a language different from that of other tokens may be determined as a speech recognition result.

[0076] A multilingual speech recognition method according to one embodiment may include a step (120) of identifying text data corresponding to speech data based on a language identity score of a target token and a probability that the target token corresponds to speech data. The probability that the target token corresponds to speech data may be a probability that the target token, determined based on pronunciation in an automatic speech recognition (ASR) module of a multilingual speech recognition model, corresponds to a speech recognition result of the speech data.

[0077] A step (120) of identifying text data corresponding to speech data according to an embodiment may include a step of determining an ASR score of a target token based on a probability that the target token corresponds to speech data, a step of correcting the ASR score of the target token based on a language identity score of the target token, and a step of identifying text data corresponding to the speech data based on the ASR score of the target token.

[0078] The ASR score of a target token can be determined based on the probability of which text the target token corresponds to, based on the pronunciation of the speech data. The ASR score can be determined based on the speech recognition result output from the ASR module of a multilingual speech recognition model.

[0079] For example, the ASR score of a target token can be adjusted by the sum, weighted sum, average, or weighted average of the ASR score and the language identity score of the target token. For example, the adjusted ASR score of the i-th token is ASR T '[i] can be determined as in mathematical expression 3.

[0080] [Equation 3]

[0081]

[0082] In Equation 3, ASR T [i] is the ASR score of the i-th token, L T [i] is the language identity score of the ith token, and λ is ASR. T [i] can be the weight of ASR T '[i] may be the adjusted ASR score of the i-th token.

[0083] Based on the corrected ASR score, text data corresponding to speech data can be determined. Among the candidates for the i-th token, the candidate for the i-th token with the highest corrected ASR score can be determined as the text data corresponding to the speech data. For example, referring to FIGS. 3A and 3B, when the ASR score for the case where the 8th token is 'azine' is similar to the ASR score for the case where the 8th token is 'sin', the language identity score L for the case where the 8th token is 'azine' T [8] is 0.74, which is the language identity score L when the 8th talk is 'sin'. T [8] Since it is higher than 0.26, the adjusted ASR score can be determined to be a higher value when the 8th token is 'azine' than when it is 'sin'. For example, when the 8th token is 'sin', the adjusted ASR score can be determined to be 0.7, and when the 8th token is 'azine', the adjusted ASR score can be determined to be 0.9. Since the adjusted ASR score is higher when the 8th token is 'azine' than when the 8th token is 'sin', the text data that includes 'azine' as the 8th token can be determined as the text data corresponding to the speech data. In other words, the text data that includes 'azine' as the 8th token can be output as the speech recognition result of the speech data.

[0084] Figure 5 is a flowchart of the operation of a multilingual speech recognition method according to one embodiment.

[0085] The multilingual speech recognition method described above with reference to FIG. 1 corresponds to a method for determining a speech recognition result based on a language identification result at the token level, and the multilingual speech recognition method described with reference to FIG. 5 may correspond to a method for determining a speech recognition result based on a language identification result at the token sequence level. As described above, a token sequence may correspond to a sequence including one or more consecutive tokens when tokens output as speech recognition results of speech data are arranged in chronological order.

[0086] Referring to FIG. 5, a multilingual speech recognition method according to one embodiment may include a step (510) of obtaining candidate text data corresponding to the speech data based on a speech recognition result of the input speech data. The candidate text data may correspond to a token sequence including one or more consecutive tokens.

[0087] A plurality of candidate text data may be obtained as a result of speech recognition of input speech data. At least some tokens of the plurality of candidate text data may differ from each other. For example, a first candidate text data of "Please recommend me a magazine for men" and a second candidate text data of "Please recommend me a magazine for men" may be obtained as a result of speech recognition of speech data uttered "Please recommend me a magazine for men." As an example, the candidate text data may correspond to a speech recognition result of speech data determined based on pronunciation in an ASR module of a multilingual speech recognition model.

[0088] A multilingual speech recognition method according to one embodiment may include a step (520) of determining a language identity score of candidate text data based on the similarity of language identification results of a plurality of tokens included in the candidate text data.

[0089] The step (520) of determining a language identity score of candidate text data according to one embodiment may include the step of determining a language identity score of each of a plurality of tokens included in the candidate text data, and the step of determining the language identity score of the candidate text data based on the sum of the language identity scores of the plurality of tokens. For example, the language identity score of the candidate text data may be determined as a sum, a weighted sum, an average, or a weighted average of the language identity scores of the plurality of tokens included in the candidate text data. For example, the language identity score of the candidate text data may be determined as a value obtained by converting the sum or the weighted sum of the language identity scores of the plurality of tokens included in the candidate text data.

[0090] According to one embodiment, the step of determining the language identity score of each of the plurality of tokens may include the step of determining the language identity score of the token based on the similarity between the language identification result of the token included in the plurality of tokens and the language identification result of the token's previous token sequence. As an example, the language identity score of the token may be determined by mathematical expression 1.

[0091] According to one embodiment, the step of determining a language identity score of each of the plurality of tokens may include the step of determining the language identity score of the token based on a similarity between a language identification result of a token included in the plurality of tokens and a language identification result of speech data corresponding to the plurality of tokens.

[0092] For example, the linguistic identity score L of candidate text data S can be determined as in mathematical formula 4.

[0093] [Equation 4]

[0094]

[0095] In mathematical expression 4, I may correspond to the length of the tokens of the candidate text data or the number of tokens included in the candidate text data. For example, L T [i] may correspond to the language identity score of the ith token included in the candidate text data determined by mathematical expression 1. For example, L T [i] can be determined based on the similarity between the language identification result of the speech data corresponding to the data indicating the language corresponding to the speech data (e.g., one-hot vector) and the language identification result of the i-th token. In other words, the language identity score L of the candidate text data. S may correspond to the average of the linguistic identity scores of the tokens contained in the candidate text data.

[0096] The step (520) of determining a language identity score of candidate text data according to one embodiment may include a step of determining a parameter regarding a proportion of the language identity score of the candidate text data based on a language change probability determined corresponding to each token included in the candidate text data, and a step of correcting the language identity score of the candidate text data based on the parameter.

[0097] The parameter regarding the weight of the language identity score of the candidate text data can be determined as a smaller value as the language change probability of the tokens included in the candidate text data increases. For example, the parameter regarding the weight of the language identity score of the candidate text data can be determined as the sum, weighted sum, product, average, or inverse of the weighted average of the language change probabilities of each token included in the candidate text data. For example, the parameter regarding the weight of the language identity score of the candidate text data can be determined as a value converted from the product, sum, or inverse of the weighted sum of the language change probabilities of each token included in the candidate text data.

[0098] For example, the linguistic identity score L of candidate text dataS can be determined as in mathematical formula 5.

[0099] [Equation 5]

[0100]

[0101] In Equation 5, I is the length of the token of the candidate text data or the number of tokens included in the candidate text data, L T [i] may correspond to the language identity score of the ith token included in the candidate text data, and β may correspond to a parameter regarding the proportion of the language identity score of the candidate text data. For example, β may be determined as the product of parameters regarding the proportion of the language identity scores of tokens included in the candidate text data.

[0102] A multilingual speech recognition method according to one embodiment may include a step (530) of identifying text data corresponding to speech data among candidate text data based on a language identity score of the candidate text data.

[0103] In one embodiment, the step (530) of identifying text data corresponding to other speech data may include a step of identifying text data corresponding to speech data among the candidate text data based on a probability that the candidate text data corresponds to speech data and a language identity score of the candidate text data. The probability that the candidate text data corresponds to speech data may be a probability that the candidate text data, determined based on pronunciation in an automatic speech recognition (ASR) module of a multilingual speech recognition model, corresponds to a speech recognition result of the speech data. As an example, the probability that the candidate text data corresponds to speech data may be determined based on the probability that each token included in the candidate text data corresponds to the speech data.

[0104] A step (530) of identifying text data corresponding to speech data according to an embodiment may include a step of determining an ASR score of the candidate text data based on a probability that the candidate text data corresponds to speech data, a step of correcting the ASR score of the candidate text data based on a linguistic identity score of the candidate text data, and a step of identifying text data corresponding to the speech data based on the ASR score of the candidate text data.

[0105] The ASR score of the candidate text data can be determined based on the probability of which text the candidate text data corresponds to based on the pronunciation of the speech data. The ASR score can be determined based on the speech recognition result output from the ASR module of the multilingual speech recognition model. For example, the ASR score of the candidate text data can be determined based on the probability of each token included in the candidate text data corresponding to the speech data or the ASR score of each token included in the candidate text data. For example, the ASR score of the candidate text data can be corrected by the sum, weighted sum, average, or weighted average of the ASR score of the candidate text data and the linguistic identity score of the candidate text data.

[0106] Text data corresponding to speech data can be determined based on the corrected ASR score of the candidate text data.

[0107] For example, referring to FIG. 6, candidate text data corresponding to voice data may include candidate text data corresponding to a first node (610), candidate text data corresponding to a second node (620), and candidate text data corresponding to a third node (630). Candidate text data corresponding to a specific node may include tokens corresponding to nodes included in a path from the root to the specific node. As an example, candidate text data corresponding to the first node (610) may correspond to "... megazine for men."

[0108] According to one embodiment, candidate text data corresponding to speech data includes an ASR score (ASR S ) can be determined based on the ASR score (ASR) among the possible combinations of tokens. For example, candidate text data corresponding to the first node (610), candidate text data corresponding to the second node (620), and candidate text data corresponding to the third node (630) are determined based on the ASR score (ASR) among the possible combinations of tokens. S ) may be data belonging to the top n (n is any natural number) or m% (m is any positive real number) with a high ASR score. For example, candidate text data corresponding to the first node (610), candidate text data corresponding to the second node (620), and candidate text data corresponding to the third node (630) may be data belonging to the top n (n is any natural number) or m% (m is any positive real number). S ) may be data that exceeds a predetermined threshold value.

[0109] ASR score (ASR) of candidate text data corresponding to the first node (610) S ) is 0.4, and the language identity score (L S ) can be determined as 1. The candidate text data corresponding to the first node (610) is data indicating that the language identification results of the tokens included in the path from the root to the last token, the first node (610), are all likely to correspond to English, and thus may be similar to each other, and thus the language identity score (LS ) can be determined as 1.

[0110] ASR score (ASR) of candidate text data corresponding to the third node (630) S ) is 0.6, and the language identity score (L S ) may be determined as 0.75. The candidate text data corresponding to the third node (630) may be data indicating that the language identification results of most of the tokens included in the path from the root to the last token, the first node (610), are most likely to correspond to English, while referring to the fourth node (640), the language identification results of some tokens may be data indicating that the language identity score (L) is most likely to correspond to French. Accordingly, the language identity score (L) may be data indicating that the language identification results of some tokens are most likely to correspond to French. S ) is the language identity score (L) of candidate text data corresponding to the first node (610), which is data indicating that the language identification result of all tokens is most likely to correspond to English. S ) can be determined to be a value smaller than .

[0111] For example, the ASR score of candidate text data (ASR S ) is the ASR score (ASR S ) and language identity score (L S ) can be corrected to the sum of the corrected ASR score (ASR S ') can be determined as text data corresponding to the first node (610) with the highest number of candidate text data corresponding to voice data. Candidate text data corresponding to the first node (610) can be output as a voice recognition result of voice data.

[0112] FIG. 7 is a block diagram illustrating the operation of a multilingual speech recognition device according to one embodiment.

[0113] A multilingual speech recognition device (700) according to one embodiment may be a device that performs the multilingual speech recognition method described above with reference to FIGS. 1 to 6. The multilingual speech recognition device (700) may be a device that executes a multilingual speech recognition model.

[0114] Referring to FIG. 7, a multilingual speech recognition device (700) can input speech data (701) and output text data (702). The text data (702) may be data obtained by converting the speech data (701) into a text format as a result of speech recognition of the speech data (701). The multilingual speech recognition device (700) may include an ASR module (710), an LHS module (720), and a score integration module (730). FIG. 7 illustrates a logical structure corresponding to the operation of a multilingual speech recognition method executed in the multilingual speech recognition device (700), and does not limit the hardware structure of the multilingual speech recognition device (700).

[0115] The ASR module (710) may be a module that determines an ASR score of a token or token sequence corresponding to speech data (701). As described above, the ASR score may be determined based on the probability of which text a target token corresponds to based on the pronunciation of the speech data (701).

[0116] The LHS (language homogeneity score) module (720) may be a module that determines the language homogeneity score of a token or token sequence corresponding to speech data (701). As described above, the determination may be based on the similarity of the language identification results of the tokens.

[0117] The score integration module (730) may be a module that determines text data (702) corresponding to speech data (701) based on the ASR score determined by the ASR module (710) and the language identity score determined by the LHS module (720). As described above, the ASR score may be corrected by the sum, weighted sum, average, or weighted average of the ASR score and the language identity score. The score integration module (730) may obtain a corrected ASR score by calculating the sum, weighted sum, average, or weighted average of the ASR score and the language identity score. The text data (702) with the maximum corrected ASR score may be output as the speech recognition result of the speech data (701).

[0118] Figure 8 is a block diagram illustrating the operation of an LHS module according to one embodiment.

[0119] The LHS module (800) illustrated in FIG. 8 may correspond to the LHS module (720) of FIG. 7. In other words, the LHS module (800) may be a module that determines a language identity score of a token or token sequence corresponding to input speech data.

[0120] Referring to Fig. 8, the LHS module (800) may include a code switch module (810), a language identification module (820), and an LHS calculation module (830). Fig. 8 illustrates a logical structure corresponding to the operation of a multilingual speech recognition method executed in the multilingual LHS module (800).

[0121] The language identification module (820) may be a module that determines a language identification result (M) of a token or token sequence corresponding to input speech data. The language identification result may include probability data regarding which token or token sequence corresponds to each recognizable language.

[0122] The code switch module (810) may be a module that determines the language change probability of a token or a token sequence corresponding to input speech data. For example, the code switch module (810) may include a model (e.g., an n-gram model, a learning model) that estimates the language change probability of a token. The code switch module (810) may output the language change probability of a token or a token sequence using the model that estimates the language change probability of a token. The LHS module (800) may or may not include the code switch module (810).

[0123] The LHS calculation module (830) may be a module that determines a language identity score of a token or token sequence based on the language identification result of the token or token sequence obtained by the language identification module (820). According to one embodiment, the LHS calculation module (830) may also determine a language identity score of a token or token sequence based on the language identification result of the token or token sequence obtained by the language identification module (820) and the language change probability of the token or token sequence obtained by the code switch module (810).

[0124] Fig. 9 is an exemplary diagram of the configuration of a multilingual speech recognition device according to one embodiment.

[0125] Referring to FIG. 9, a multilingual speech recognition device (900) according to one embodiment may include a processor (901), a memory (903), and an input / output device (I / O) (905). The multilingual speech recognition device (900) may include a device that performs the multilingual speech recognition method described above with reference to FIGS. 1 to 6. As an example, the multilingual speech recognition device (900) may correspond to the speech recognition device (700) described above with reference to FIG. 7.

[0126] A processor (901) according to one embodiment can perform at least one operation of the multilingual speech recognition method described above through FIGS. 1 to 6.

[0127] For example, the processor (901) may perform at least one of an operation of determining a language identity score of a target token based on a language identification result of a target token obtained as a result of speech recognition of input speech data, and an operation of identifying text data corresponding to speech data based on the language identity score of the target token and a probability that the target token corresponds to speech data.

[0128] For example, the processor (901) may perform at least one of an operation of obtaining candidate text data corresponding to the voice data based on a voice recognition result of the input voice data, an operation of determining a language identity score of the candidate text data based on a similarity of language identification results of a plurality of tokens included in the candidate text data, and an operation of identifying text data corresponding to the voice data among the candidate text data based on the language identity score of the candidate text data.

[0129] The memory (903) according to one embodiment may be a volatile memory or a non-volatile memory, and may store data related to the multilingual speech recognition method described above with reference to FIGS. 1 to 6. For example, the memory (903) may store data generated during the execution of the multilingual speech recognition method or data required to execute the multilingual speech recognition method. For example, the memory (903) may store at least one of a language identification result of a token or a token sequence, a language identity score of the token or token sequence, and an ASR score of the token or token sequence.

[0130] According to one embodiment, the memory (903) may store a program implementing the multilingual speech recognition method described above through FIGS. 1 to 6. The processor (901) may execute the program stored in the memory (903) and control the multilingual speech recognition device (900). The code of the program executed by the processor (901) may be stored in the memory (903).

[0131] A multilingual speech recognition device (900) according to one embodiment can be connected to an external device (e.g., a personal computer or a network) via an input / output device (905) and exchange data. For example, the multilingual speech recognition device (900) can receive voice data via the input / output device (905) and output text data as a result of voice recognition of the voice data.

[0132] A multilingual speech recognition device (900) according to one embodiment may further include other components not shown.

[0133] For example, the multilingual speech recognition device (900) may include a communication module. The communication module may provide a function for the multilingual speech recognition device (900) to communicate with other electronic devices or other servers via a network. According to one embodiment, the memory (903) may not be a component of the multilingual speech recognition device (900), but may be included in an external device accessible from the multilingual speech recognition device (900). In this case, the multilingual speech recognition device (900) may receive data stored in the memory (903) included in the external device via the communication module, and may transmit data to be stored in the memory (903).

[0134] For example, the multilingual speech recognition device (900) may further include other components such as a transceiver, various sensors, a database, etc.

[0135] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0136] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0137] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0138] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0139] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0140] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A step of determining a language homogeneity score of a target token based on the language identification result of the target token obtained as a result of speech recognition of the input speech data; and A step of identifying text data corresponding to the speech data based on the language identity score of the target token and the probability that the target token corresponds to the speech data. including, Multilingual speech recognition method.

2. In paragraph 1, The step of identifying text data corresponding to the above voice data is A step of determining an automatic speech recognition (ASR) score of the target token based on a probability that the target token corresponds to the speech data; A step of correcting the ASR score of the target token based on the language identity score of the target token; and A step of identifying text data corresponding to the speech data based on the ASR score of the target token. including, Multilingual speech recognition method.

3. In any one of paragraphs 1 and 2, The step of determining the language identity score of the above target token is A step of determining a language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of the previous token sequence of the target token. including, Multilingual speech recognition method.

4. In any one of the clauses 1 to 3, The step of determining the language identity score of the above target token is A step of determining a language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of the speech data. including, Multilingual speech recognition method.

5. In any one of paragraphs 1 to 4, The step of determining the language identity score of the above target token is A step of determining a parameter regarding the proportion of the language identity score of the target token based on the language change probability corresponding to the target token; and A step of correcting the language identity score of the target token based on the above parameters. including, Multilingual speech recognition method.

6. A step of obtaining candidate text data corresponding to the voice data based on the voice recognition result of the input voice data; A step of determining a language identity score for each of the candidate text data based on the similarity of the language identification results of a plurality of tokens included in the candidate text data; and A step of identifying text data corresponding to the speech data among the candidate text data based on the language identity score of the candidate text data. including, Multilingual speech recognition method.

7. In paragraph 6, The step of determining the language identity score of each of the above candidate text data is A step of determining a language identity score for each of a plurality of tokens included in the candidate text data; and A step of determining a language identity score of the candidate text data based on the sum of the language identity scores of the plurality of tokens. including, Multilingual speech recognition method.

8. In paragraph 7, The step of determining the language identity score of each of the above multiple tokens is A step of determining a language identity score of the token based on the similarity between the language identification result of the token included in the plurality of tokens and the language identification result of the previous token sequence of the token. including, Multilingual speech recognition method.

9. In paragraph 7, The step of determining the language identity score of each of the above multiple tokens is A step of determining a language identity score of the token based on the similarity between the language identification result of the token included in the plurality of tokens and the language identification result of the speech data corresponding to the plurality of tokens. including, Multilingual speech recognition method.

10. In any one of paragraphs 6 to 9, The step of identifying text data corresponding to the above voice data is A step of identifying text data corresponding to the speech data among the candidate text data based on the probability that each of the candidate text data corresponds to the speech data and the language identity score of each of the candidate text data. including, Multilingual speech recognition method.

11. In any one of the clauses 6 to 10, The step of determining the language identity score of each of the above candidate text data is A step of determining a parameter regarding the weight of the language identity score of each of the candidate text data based on the language change probability determined corresponding to each of the tokens included in the candidate text data; and A step of correcting the language identity score of each of the candidate text data based on the above parameters. including, Multilingual speech recognition method.

12. A computer program stored on a medium for executing the method of any one of claims 1 to 11 in combination with hardware.

13. In a multilingual speech recognition device (700, 900), At least one memory (903) storing one or more commands; and At least one processor (901) executing the above command Including, The above command, when executed by the at least one processor, causes the multilingual speech recognition device to: Based on the language identification result of the target token obtained as a result of speech recognition of the input speech data, a language homogeneity score of the target token is determined, Identifying text data corresponding to the speech data based on the language identity score of the target token and the probability that the target token corresponds to the speech data. Multilingual speech recognition device.

14. In paragraph 13, The above command, when executed by the at least one processor, causes the multilingual speech recognition device to: In identifying text data corresponding to the above voice data, Determine the ASR (automatic speech recognition) score of the target token based on the probability that the target token corresponds to the speech data, Based on the language identity score of the target token, the ASR score of the target token is corrected, Based on the ASR score of the target token, identifying text data corresponding to the speech data, Multilingual speech recognition device.

15. In paragraph 13, The above command, when executed by the at least one processor, causes the multilingual speech recognition device to: In determining the language identity score of the above target token, Determine the language identity score of the target token based on the similarity between the language identification result of the target token and the language identification result of the previous token sequence of the target token. Multilingual speech recognition device.

Images