Automatic detection of language from non-character submarking signals

Abstract

In non-limiting examples of the present disclosure, systems, methods, and devices for determining a language of a text string are presented. A language detection model can be maintained. The language detection model can include identities and weights for initial and final consonants, identities and weights for prefixes and suffixes, and identities and weights for vowel sequences, where each identity is derived from a training corpus. The weights can correspond to frequencies of the text units in the corpus. A text string can be received, and a match score between the text string and a language of the language detection model can be determined. The match score can be based on initial and final consonant scores, prefix and suffix scores, and / or vowel sequence scores for each word in the text string. If the match score satisfies a threshold, a subsequent action associated with the language can be performed.

Description

Background Technology

[0001] Text input on computing devices increasingly relies on language-specific processing to refine and respond to user intent. This processing depends on a core assumption: the language of the input text is known. These systems perform poorly when the assumed language does not match the input text. To address this issue, systems can use preprocessing steps to identify the language of the incoming text string.

[0002] The aspects of the present technology disclosed herein have been considered in relation to this general technical environment. Furthermore, although a general environment has been discussed, it should be understood that the examples described herein should not be limited to the general environment identified in the background. Summary of the Invention

[0003] This disclosure is provided to introduce some concepts in a simplified form, which will be further described in the detailed description section below. This disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practice of this disclosure.

[0004] This disclosure provides non-limiting examples of systems, methods, and devices for determining the language of text strings input to a computing device. Language detection applications or services may maintain language detection models for one or more languages. A language detection model may include: identities of multiple prefixes and weights associated with these prefixes, as identified in a corpus of the language; identities of multiple suffixes and weights associated with these suffixes, as identified in a corpus of the language; identities of multiple initial consonants and weights associated with these initial consonants, as identified in a corpus of the language; identities of multiple final consonants and weights associated with these final consonants, as identified in a corpus of the language; and identities of vowel sequences and weights associated with these vowel sequences, as identified in a corpus of the language. The weights associated with each word in those partial words (e.g., "token") may correspond to the frequency of the partial word in the corpus (e.g., higher frequency, higher weight).

[0005] When a text string input is received, one or more language detection models can be applied to the text string. A match score between the text string and each language corresponding to the applied language detection model can be determined based on prefixes and suffixes included in the words of the text string, and syllables included in the words of the text string. A syllable is defined as an optional legal initial consonant sequence as defined in the model, followed by a mandatory legal vowel sequence as defined in the model, followed by an optional legal final consonant sequence as defined in the model. A legal word or stem consists only of consecutive legal syllable sequences. In some examples, the text string can be determined to be in the corresponding language if the match score of the language detection model meets a threshold. In other examples, the text string can be determined to be in the language associated with the highest-scoring language detection model. Attached Figure Description

[0006] The following figures illustrate examples of non-restrictive and non-exhaustive approaches:

[0007] Figure 1 This is a schematic diagram illustrating an example distributed computing environment used to determine the language of a text string and to perform one or more subsequent actions based on that determination.

[0008] Figure 2 This is a schematic diagram illustrating the computational environment for training a language detection model.

[0009] Figure 3 A computational environment is shown for processing exemplary words from a corpus by multiple processing engines included in a language detection training engine.

[0010] Figure 4 The various components of the language detection model are shown.

[0011] Figure 5A It is a method for determining whether a text string matches a language using a syllable-based language detection model and for performing subsequent actions based on that determination.

[0012] Figure 5B It is a method for determining whether a text string matches a language using an affix-based model and for performing subsequent actions based on that determination.

[0013] Figure 5C It is a method for determining whether a text string matches a language using both affix-based and syllable-based language detection models, and for performing subsequent actions based on that determination.

[0014] Figure 5D It is a method for selecting the best language from multiple languages ​​using a language detection model and performing subsequent actions based on that determination.

[0015] Figure 6 and Figure 7 This is a simplified diagram of a mobile computing device that can implement various aspects of this disclosure.

[0016] Figure 8 This is a block diagram of an example physical component of a computing device that can implement various aspects of this disclosure.

[0017] Figure 9 It is a simplified block diagram of a distributed computing system that can implement various aspects of this disclosure. Detailed Implementation

[0018] Various embodiments will be described in detail with reference to the accompanying drawings, wherein the same reference numerals throughout the various views denote the same parts and components. References to various embodiments herein do not limit the scope of the appended claims. Furthermore, any examples set forth in this specification are not intended to be limiting, but merely illustrate some of the many possible embodiments with respect to the appended claims.

[0019] This disclosure provides non-limiting examples of systems, methods, and apparatuses for determining the language of text strings input into a computing device construct. The construct into which the text string is input may include an operating system shell construct, an application construct, or an application service (e.g., a cloud-based service) construct. A language detection service may receive instructions to analyze the text using the language detection model described herein. The language detection service may be incorporated into and executed on a local computing device and / or in the cloud. The instructions may include determining that the text input has not previously been received by a user account in the construct currently receiving the text string input, and / or that the user account has not set a preferred language for the computing device, application, or service. In some examples, the language detection service described herein may simply apply the language detection model periodically or whenever text input is received in one or more computing constructs.

[0020] Language detection services can include separate language detection models for each of multiple languages ​​(e.g., one model for English, one for German, and one for Spanish). Each model can include the same or similar processing engine, but a weighted list of different linguistic components (e.g., prefixes, suffixes, vowel sequences, initial consonant sequences, final consonant sequences, common words), sometimes referred to in this paper as “tags”. These separate language models can be trained such that the weight of each tag corresponds to its relative frequency in the language, or more specifically, to its relative frequency in one or more corpora on which the model was trained.

[0021] At runtime, upon receiving a text string input, the language detection service can apply one or more language detection models to the text string and calculate a matching score for each word in the string against the language corresponding to the applied model. The scores for each word can be summed or otherwise functionalized to generate a total matching score for the text string and the language. In some examples, if the matching score between the text string and the language meets or exceeds a threshold, it can be determined that the text string is definitely that language, possibly / probably that language, or definitely not that language. In other examples, the language detection service can determine that the language associated with the highest-ranking matching score among multiple scores from multiple language detection models is the language of the text string.

[0022] Based on the determination that the text string is likely in a specific language, various subsequent actions can be performed. For example, a language processing model for that language (e.g., an intent determination processing model, a spell check processing model, a grammar check processing model, etc.) can be applied to the text string. The type of language processing model applied to the language can be determined based on the application or computational shell construction that receives the text string. In some examples, subsequent actions may include downloading one or more linguistic libraries or models for that language from a cloud database to the local computing device receiving the text string.

[0023] The systems, methods, and apparatus described herein offer technological advantages for identifying the language of input text. The memory costs associated with traditional language detection models (such as n-gram models) that rely on the frequency of strings of length N characters are significantly reduced by applying the mechanisms described herein, as a relatively small model can be maintained to process text input, consisting of initial and final consonants, vowel sequences, common words, and prefixes and suffixes. The language detection models described herein and their corresponding libraries can be maintained entirely on local computing devices, and larger processing models and associated language libraries (e.g., for digital assistants, for productivity applications) can be downloaded to those local computing devices only when the user's primary input language is determined. Furthermore, aspects of this disclosure combine linguistic awareness of tokenized structures with frequency analysis. This approach avoids the noise naturally present in character n-gram models, which cannot adjust for morpheme and syllable boundaries.

[0024] Figure 1This is a schematic diagram illustrating an example distributed computing environment 100 used to determine the language of a text string and to perform one or more subsequent actions based on that determination. The computing environment 100 includes a computing device 102, a network and processing subenvironment 108, a language detection subenvironment 116, which includes a language detection module 124, multiple language detection models that can be included in a language model data repository 120, and a subsequent action module 126. A common subsequent action might be downloading or otherwise deploying the complete set of language assets 114.

[0025] Network and processing subenvironment 108 includes network 110 and server computing device 112. Any computing devices described herein can communicate with each other via a network (such as network 110). Server computing device 112 illustrates one or more servers that can host one or more application services. One or more application services can be accessed via the Internet from one or more local computing devices such as computing device 102, and those application services can be executed in the cloud.

[0026] The language detection service typically shown in language detection model sub-environment 116 can be hosted and executed in the cloud (e.g., by server computing device 112), and / or the language detection service can be hosted and executed by a local computing device (e.g., computing device 102). In fact, a technical advantage of the language detection models described herein is that they are small enough (e.g., in terms of memory requirements and storage footprint) that they can easily reside on the limited storage provided by the local computing device, and therefore they do not necessarily need to be executed in the cloud.

[0027] A language detection service can perform operations associated with: receiving text strings in a computing shell construct (e.g., a desktop search bar, digital assistant input) or application construct (e.g., a web search in a web browser application, a document canvas in a word processing application) on a local computing device; processing those text strings using different language models; and determining the language of the input text. In some examples, the language detection service may enable one or more subsequent actions to be taken in connection with the input text or with the determination of the language of the input text.

[0028] The language detection model sub-environment 116 includes a language detection module 124 and a subsequent action module 126. The elements described regarding the language detection model sub-environment 116 can be included in the language detection service.

[0029] like Figure 2As shown, the language detection training engine can generate weighted lists of prefixes, suffixes, initial consonants, vowel sequences, and / or final consonants for one or more languages. These lists can be generated based on one or more corpora for each language for which the language detection service maintains its model. These corpora can be included in a training data repository.

[0030] Training on a language dataset can produce weighted lists of tags (e.g., weighted vowel sequences, weighted prefixes, weighted suffixes, weighted initial consonants, weighted final consonants). The weights assigned to the tags can correspond to the frequency of the tags in one or more corpora used to train the language-specific model. Multiple final weighted lists (e.g., ...) can be generated for each language. Figure 2 The final weighted individual language model 250 can be included in a data repository associated with the language detection service, such as a language model data repository 120. The following is combined with... Figure 2 and Figure 3 Describe additional details regarding the training of the language detection model.

[0031] In some examples, language detection models can be optimized to distinguish closely related languages ​​(e.g., Danish and Norwegian). For instance, the affix list can be modified to ensure that each first-language (e.g., Norwegian) suffix has a corresponding counterpart in its second language (e.g., Danish), and each second-language suffix has a corresponding counterpart in its first language. The training data can then be modified to make the first-language and second-language training data highly similar, thereby aligning the weights subsequently assigned to the affixes.

[0032] The language detection module 124 may include one or more processing engines applied to an input text string associated with language model data (e.g., from the language model data repository 120) to determine the language of the input text string. Further details regarding the application of the language model to the text string will be provided below. Figure 4 To describe.

[0033] Once it is determined that the input text corresponds to a specific language, the subsequent action module 126 can take one or more actions on the input text and / or perform one or more actions associated with the determined language. In some examples, the actions may include applying a language processing model for the determined language to the input text string. For example, if the input text string is determined to be in English and is received by a digital assistant service, the text string can be processed using an English language intent identification processing model. In another specific example, if the input text string is determined to be in German and is received in a word processing document, a German language spelling or grammar checking processing model can be applied to the text string.

[0034] Therefore, the subsequent action module 126 can identify the appropriate language processing model and / or action based on the construction of the received input text string and the specific language of the determined text string. In an additional example, when it is determined that the text string corresponds to a specific language, the subsequent action module 126 can identify one or more language libraries and / or language models to be downloaded to the device receiving the text. For example, if it is determined that the input text string received by computing device 102 is language X, then the subsequent action module 126 can cause one or more language libraries or language models (of language X) from the full set of language assets 114 to be downloaded from the cloud to computing device 102. In this way, the local computing device only needs to download and store the language libraries and language models that may be used by users of the local computing device.

[0035] In this example, computing device 102 displays application 103, which shows any application that can be accessed and / or executed by the local computing device. For example, application 103 may include a word processing application, a spreadsheet application, a presentation application, a task completion application, a messaging application, a web browser application, or a game application. Application 103 can execute on computing device 102 or in the cloud. The user has already entered a text string 106 on canvas 104 of application 103.

[0036] The language detection service can receive and process text from an application or shell construct executed from a local computing device, based on the text initially input into a particular construct and / or on indications that previously input text may have been processed by a language model that does not match the language of the input text. When the language detection service receives a text string 106, it can apply one or more language detection models to the text string 106.

[0037] When applying the language detection model to text string 106, tokens from each word in the string can be extracted and analyzed. For example, it can be determined whether each word in text string 106 has a prefix or suffix included in the language model (e.g., a list of weighted prefixes or weighted suffixes used for the language model). Each identified prefix or suffix can be scored based on its corresponding weight in the language model. Complete words or stems (the parts remaining after removing identified affixes) can be iteratively analyzed into syllables using a specific order of initial consonant clusters, vowel lists, and final consonant clusters. Based on some examples, inter-token scores can also be determined for words in text string 106. The final language matching score for the language detection model can be calculated based on one or more of the aforementioned token scores. The highest-scoring language detection model can be determined to correspond to the language in which the text string is located.

[0038] For clarity, an example of applying a language detection model to words in a text string is provided below.

[0039] For the word "ANTIDISESTABLISHMENTARY", the application of the language detection model described in this article may include the following operation / processing steps.

[0040] Find the longest suffix sequence that matches the word ending of ANTIESTABLISH-MENTARY.

[0041] Check if the remaining part is at least N characters (N = 2, 3, depending on the language). If not, retry with a shorter suffix.

[0042] Check if the remaining part contains a vowel; if not, try again with a shorter suffix.

[0043] Check the compatibility of the preceding consonant sequence with the list of word-final consonant sequences (the algorithm varies depending on whether the suffix begins with a vowel). If not, retry with a shorter suffix.

[0044] Once the suffix has been selected, remove the suffix and check the beginning of the remaining stems to obtain the longest matching prefix sequence ANTI-DIS-ESTABLISH.

[0045] Check if the remaining part (stem) is at least N characters (N = 1, 2, 3, depending on the language). If not, retry with a shorter prefix.

[0046] Check if the remaining part contains a vowel. If not, try again with a shorter prefix.

[0047] Check the compatibility of the subsequent consonant sequence with the list of initial consonant sequences (the algorithm varies depending on whether the prefix ends in a vowel). If not, retry with a shorter prefix.

[0048] The stem is analyzed into syllables (iteration) (#-ES) (TAB) (LI-SH).

[0049] Remove the initial consonant cluster #-ESTABLISH.

[0050] Identify the optimal sequence of vowel clusters following the consonant cluster E-STABLISH.

[0051] ST-ABLISH identifies the best word-final / word-initial consonant clusters.

[0052] Identify the optimal sequence of vowel clusters following the consonant cluster A-BLISH.

[0053] The optimal combination of consonant clusters at the end and beginning of a word is identified by BL-ISH.

[0054] Identify the optimal sequence of vowel clusters following the consonant clusters I-SH.

[0055] Identify the best word-final / word-initial consonant clusters (SH).

[0056] The log probabilities of these components are summed to determine the syllable score.

[0057] Log-probability of interpolation for syllables / prefixes / suffixes.

[0058] return Figure 1 For example, the language detection service determines that text string 106 corresponds to language A, as shown in language detection element 128. Thus, the language detection service can cause one or more actions to be performed, as shown in subsequent action element 130.

[0059] Figure 2 This is a schematic diagram illustrating the computational environment 200 for training the language detection model. The computational environment 200 includes a corpus 202, an affix detection training engine 204, a syllable delimiter marker detection training engine 212, a final weighting engine 234, and a final weighted individual language model 250.

[0060] Training can be conducted in three phases: First, a preliminary list of tags can be identified (e.g., suffix candidates 218, prefix candidates 220, word-initial consonants 222, vowels and vowel sequences 224, word-final consonants 226). Then, the preliminary list can be manually reviewed and finalized through multiple processes (e.g., suffix review 228, prefix review 230, consonant and vowel sequence review 232). Finally, once the tag list is established, the reviewed list can be weighted by applying weights from a corpus (e.g., corpus 202) to the finalized list (e.g., suffix candidates 218, prefix candidates 220, word-initial consonants 222, vowels and vowel sequences 224, word-final consonants 226). In a model that includes both 1) a weighted list of suffixes and prefixes (e.g., suffix candidates 218, prefix candidates 220) and 2) a weighted list required for syllable analysis of the remaining stem after removing prefixes and suffixes (e.g., word-initial consonant 222, vowel and vowel sequence 224, word-final consonant 226), the labeling and weighting of suffix candidates 218 and prefix candidates 220 can precede the labeling and weighting of the markers required for syllable analysis (e.g., word-initial consonant 222, vowel and vowel sequence 224, word-final consonant 226). The labeling of the markers required for syllable analysis may require manual labeling of the vowels in the trained language before training.

[0061] The affix detection training engine 204 includes a suffix training engine 208, a prefix training engine 210, and human review of a preliminary affix list extracted from the corpus 202. The syllable segmentation marker detection training engine 212 includes word-initial consonant 222, vowel and vowel sequences 224, and word-final consonant 226. Stem-initial consonant clusters or word-initial consonant clusters and stem-final consonant clusters or word-final consonant clusters can be used to determine syllable-initial consonant clusters and syllable-final consonant clusters in the language, because syllables within a word rarely end with a consonant cluster, and such clusters may not end the word. When both the affix detection engine and the syllable detection engine are trained, the affix detection engine may be applied before syllable detection. Syllable detection training may also include human review of word-initial consonant 222, vowel and vowel sequences 224, and word-final consonant 226. The affix detection training engine 204, the syllable segmentation marker detection training engine 212, and the final weighting engine 234 are examples of training processes that can be used to train a language detection model. The final model for a given language can consist of one or more final weighted lists from the final weighted individual language models 250. Multiple weighted lists for various languages ​​can be included. Figure 1 The language model data is stored in repository 120.

[0062] The affix detection training engine 204 can receive multiple words from the corpus 202. The words received from the corpus 202 are the languages ​​that the resulting language detection model will be trained to detect. For example, if the language detection training engine is implemented to train an English language detection model, then the corpus 202 will include multiple English words. Or, if the language detection training engine is implemented to train a German language detection model, then the corpus 202 will include multiple German words.

[0063] The suffix training engine 208 can identify and extract suffixes from words in the corpus 202. When identifying and extracting suffixes, the suffix training engine 208 can search the ends of words in the corpus 202 for n-grams of length 1-N, where N depends on the language of the corpus 202. For example, for a first language, it can be manually determined that the suffix sequence can only have five characters (e.g., letters) or fewer; thus, suffixes from one character to five characters (1-N = 1-5) will be identified and extracted. The suffix training engine 208 can identify character sequences at the ends of words of length 1-N that appear at a threshold frequency in the corpus 202 and mark them as possible suffixes.

[0064] The prefix training engine 210 can identify and extract prefixes from words in the corpus 202. In identifying and extracting prefixes, the prefix training engine 210 can search the beginnings of words in the corpus 202 for n-grams of length 1-N, where N depends on the language of the corpus 202. For example, for a first language, it can be manually determined that prefix sequences can only have six characters (e.g., letters) or fewer; thus, prefixes from one character to six characters (1-N = 1-6) will be identified and extracted. The prefix training engine 210 can identify character sequences of length 1-N that occur at a threshold frequency in the corpus 202 at the beginning of words and mark them as possible prefixes.

[0065] Based on some examples of applying the suffix training engine 208 and / or the prefix training engine 210 to words in the corpus 202, the resulting word stem (e.g., characters minus the identified / extracted prefixes and / or suffixes) must be at least a threshold number of characters long and include at least one vowel. If it is determined that the resulting word stem is not at least a threshold number of characters long and includes at least one vowel, then the word and / or the prefix or suffix already identified for that particular word can be rejected from the training process, and shorter prefixes / suffixes are tested.

[0066] In some examples, the prefixes and suffixes of the identifier can be manually reviewed. That is, someone familiar with the language can manually review the list of prefix candidates (e.g., prefix candidate 220) and discard any prefixes that are not true prefixes of the language for which the model is built / trained. This is illustrated by prefix review 230. Similarly, someone familiar with the language can manually review the list of suffix candidates (e.g., suffix candidate 218) and discard any suffixes that are not true suffixes of the language for which the model is built / trained. This is illustrated by suffix review 228.

[0067] The syllable delimiter marker detection training engine 212 can remove prefixes and suffixes included in the lists associated with prefix candidates 220 and suffix candidates 218, and identify and extract initial and final consonant sequences from the remaining substrings (e.g., stems). The syllable delimiter marker detection training engine 212 can additionally collect and count vowel sequences immediately following an initial consonant sequence or immediately preceding a final consonant sequence. In the example, any initial or final non-vowel string of length zero or greater can be considered a consonant sequence.

[0068] Based on some examples, consonant sequence candidates and vowel sequence candidates (e.g., word-initial consonant 222, vowel and vowel sequence 224, word-final consonant 226) can be manually reviewed, as shown in consonant and vowel sequence review 232. That is, someone familiar with the language can manually review unique consonant and vowel sequences and discard uncommon sequences and any sequences generated by noise in the training data (e.g., proper nouns, foreign nouns). In some examples, monosyllabic vowel sequences can be split across syllables (e.g., "ayo" in "mayor" can be split into "ay" and "o").

[0069] Once the final list of tokens (e.g., suffix candidates 218, prefix candidates 220, word-initial consonants 222, vowel and vowel sequences 224, word-final consonants 226) and the final list of common words 236 are established, the final weight of each token can be assigned by retraining the entire token list on the corpus 202 using the final weighting engine 234. The most common words often have fairly unique structures in the language, so very common words (as defined in the final list of common words 236) can be ignored in the corpus 202 when establishing the weights of other tokens.

[0070] When training the model, prefixes and suffixes in the tag list (e.g., prefix candidate 220 and suffix candidate 218) can be sorted in descending order of length. If a word begins with a prefix or ends with a suffix, it can be determined that the beginning or end of the remaining substring begins or ends with a legal consonant sequence (including empty sequences, if allowed in the language). If the consonant sequence is determined to be illegal, or if the remainder of the word does not contain vowels, the next shortest prefix or suffix matching that word can be identified in the list. For example, in the word "render," training might determine that "re-" is a prefix, but then discard it because "nd" is not in the list of consonant sequences that can legally begin an English word. This step can also be used to avoid incorrectly counting suffixes as "-ion," as in "lion."

[0071] In some examples, additional steps can be performed when processing suffixes. If the suffix begins with a vowel and the preceding consonant sequence cannot end the word, it can be determined whether a valid word-ending / word-initial consonant combination exists. For example, the suffix "-ation" (as in "amalgamation") was found when parsing the word "filtration". After removing "-ation", the remaining substring is "filtr". The suffix might be rejected because the character sequence "tr" cannot legally end an English syllable, but because the suffix begins with a vowel, and because it is possible to create valid sequences that include vowel, word-ending consonant clusters, and word-initial consonant clusters (e.g., i / l / tr), "-ation" can be counted as a suffix in the case of "filtration".

[0072] When this training is performed, the final weighted engine 234 can process the suffix before processing the prefix.

[0073] Once prefixes and suffixes have been removed from a word, the final weighting engine 234 can divide the remaining content into syllables. For example, for the word "fungicide," the previous step performed by the model training engine 234 would identify the suffix "-cide," leaving "fungi." The final weighting engine 234 can add weights (e.g., add 1) to the initial suffix "f" and the vowel sequences "u" and "i." This leaves the sequence "ng." English words cannot legally begin with the sequence "ng," but there are still two legal options. These options are splitting the string into "ng" or "ng-#." The final weighting engine 234 can choose the option with the highest combined weight. For example, if the weight of "ng-#" is 180 and the weight of "ng" is 250, the final weighting engine 234 can choose "ng" and add the weight to "n" as an suffix and the weight to "g" as an initial suffix.

[0074] If the final weighted engine 234 is unable to construct a valid word using the above steps (e.g., the word contains no vowels or begins with an illegal English suffix, such as "kjenne"), then the word can be ignored for training purposes.

[0075] Training performed by the final weighted engine 234 can be accomplished by taking the logarithm of the probabilities of these final counts.

[0076] Figure 3 A computational environment 300 is shown for processing exemplary words from a corpus of a language by means of multiple processing engines included in a language detection training engine for a single language. Although regarding... Figure 3The described processing is shown as being performed by multiple processing engines; however, it should be understood that each illustrated engine may be contained within a single processing engine and applied to the text string via the application of a language detection model. It should be understood that a single processing engine can be used for all languages, with differences between languages ​​contained only in their respective models (e.g., in the final weighted individual language model 250). An exemplary word is “antidisestablishmentary,” prefixed with “anti” and “dis,” and suffixed with “ment,” “ar,” and “y.” Its stem is “establish.”

[0077] The suffix sequence identification engine 308 is applied to the example word 302B to identify the suffix 316 (“y”, followed by “ar”, then “ment”), which is removed from the remaining characters 318 (“antidisestablish”) (e.g., the example subword 302D).

[0078] The prefix sequence identification engine 310 is applied to the exemplary subword 302D, “antidisestablish”, thereby identifying the prefix 311 (“anti” and “dis”), which is removed from the remaining characters 312, which is the stem 302F (“establish”).

[0079] The syllable training engine 320 is applied to the stem 302F “establish” to identify the empty initial consonant sequence 322A (“#”) because “establish” begins with a vowel.

[0080] The vowel sequence analyzer 321 determines that "i" forms a single syllable, and is therefore the best (and only possible) analysis for "i". The best intersyllable consonant sequence analyzer 323 determines which of the three possible intersyllable consonant sequence analyses is optimal. The sequence is "st", and the three possible analyses are "st-", "st", and "-st". "st" is chosen because the probability of syllable-word ending "s" plus the probability of syllable-word beginning "t" is greater than the combined probabilities of the other options. Syllable training iterations are performed to analyze the two remaining syllables in sequence 302H and end with the word-ending syllable 302I "lish" because it is not followed by a vowel.

[0081] It can be made by Figure 3 Each token identified by the engine shown is added to the token list in the language model (e.g., in the final weighted individual language model 250), and the weights of these tokens are normalized once the engine has been applied to the remaining words in the corpus 202.

[0082] Figure 4 The various components of the language detection model 402 are shown, and their relationship with... Figure 2 The final weighted individual language models 250 are the same or similar. The language detection model 402 includes weighted prefixes and prefix sequences 404, weighted suffixes and suffix sequences 406, weighted legal word initial consonant clusters 408, weighted legal word final consonant clusters 410, weighted legal vowel sequences 412, legal characters, code points, punctuation marks 414, and common words 416, which can also be weighted.

[0083] The weighted prefix and prefix sequence 404 include strings of characters identified by the prefix training engine 210, and their weights can be adjusted by applying one or more operations associated with the prefix review 230.

[0084] The weighted suffixes and suffix sequence 406 include strings of characters identified by the suffix training engine 208, and the weights of these character strings can be adjusted by applying one or more operations associated with the suffix review 228.

[0085] The weighted valid word initial consonant clusters 408 include word initial consonant cluster strings identified by the syllable delimiter marker detection training engine 212, and the weights of these word initial consonant cluster strings can be adjusted by applying one or more operations associated with the consonant and vowel sequence review 232.

[0086] The weighted valid word-final consonant clusters 410 include word-final consonant cluster strings identified by the syllable delimiter marker detection training engine 212, the weights of which can be adjusted by applying one or more operations associated with the consonant and vowel sequence review 232.

[0087] The weighted legal vowel sequence 412 includes vowels and vowel sequences identified by the syllable delimiter marker detection training engine 212, and its weights can be adjusted by applying one or more operations associated with the consonant and vowel sequence review 232.

[0088] The list of legal characters, code points, and punctuation marks 414 may include a list of legal characters, code points, and / or punctuation marks that highly indicate a particular language (for example, an inverted question mark may correspond to a 99% probability that the string is Spanish).

[0089] Common Words 416 can include a list of words that are highly indicative of a particular language.

[0090] At runtime (e.g., when a text string input is received), upon receiving an instruction to determine the language associated with the text string containing one or more words, one or more language detection models (such as language detection model 402) for one or more languages ​​can be applied to the text string. The score for a candidate language string can be obtained based on the degree of matching between the string and the frequencies of prefixes, suffixes, and syllables in language detection model 402. The presence of syllables not appearing in the model strongly indicates that the string does not match the model's language. Strings can be tested against multiple candidate language models, and the scores of each model can be compared to obtain a confidence score for the string's language.

[0091] Figure 5A Method 500A is an application of a single syllable-based language detection model to determine the language of a text string and to perform subsequent actions based on that determination. Method 500A begins with a start operation, and the flow moves to operation 502A.

[0092] In Operation 502A, maintain a language detection model for a first language. This language detection model may include: a first list containing the identities of multiple syllable-initial consonants from a corpus of the first language, and a weight for each of these syllable-initial consonants; a second list containing the identities of multiple vowel sequences from the corpus of the first language, and a weight for each of these vowel sequences; and a third list containing the identities of multiple syllable-final consonants from the corpus, and a weight for each of these syllable-final consonants. The weights may correspond to the frequencies of initial and final consonants in the corpus.

[0093] Based on some examples, the preprocessing step (not shown) can quickly accept a word because it is in the list of common words, or it can reject a word because it does not contain a vowel.

[0094] The process continues from operation 502A to operation 504A. In operation 504A, starting from the beginning of the stem or word, a syllable-based language detection model is iteratively applied to each consecutive syllable in the word or stem of the text string. As discussed below, the initial syllable is identified, and then processing moves to the position after the first identified syllable, continuing until the last character of the word or stem.

[0095] Applying a syllable-based language detection model to words can include: determining the syllable-initial consonant cluster score of the first consonant cluster in the remaining (unprocessed) part of the word or stem, where the initial consonant cluster score corresponds to the frequency of the initial consonant cluster in a corpus (e.g., corpus 202). Applying a language detection model to words or stems can also include: determining the vowel cluster score of the first vowel cluster following the first consonant cluster in the remaining unprocessed part of the word or stem. Applying a language detection model to words or stems can also include: determining which analysis of subsequent intersyllable consonant clusters is best (syllable-final clusters entering the preceding syllable and syllable-initial clusters of the following syllable). Applying a language detection model to words can also include: determining the final consonant cluster score of the final consonant clusters included in the word (or each syllable in the word), where the final consonant cluster score corresponds to the frequency of the final consonant cluster in a corpus. Then, the application of the syllable detection model can repeat this process until all syllables in a word or stem have been identified and scored, or until it is determined that no such analysis consistent with the model is possible.

[0096] The process continues from operation 504A to operation 506A. In operation 506A, the language matching score between the text string and the first language is calculated based on the syllable scores of all combinations.

[0097] The process continues from operation 506A to operation 508A. In operation 508A, it is determined that the language matching score meets the threshold.

[0098] The process continues from operation 508A to operation 510A, where, based on the determination that the language matching score meets a threshold, subsequent actions are performed. In this example, subsequent actions may include applying a language processing engine specifically for the first language to the text string. In other examples, performing subsequent actions may include downloading a language pack library for the first language to the computing device where the text string was initially input. The language pack library for the first language may include an embedding library (e.g., the BERT library, the ELMo library).

[0099] The process moves from operation 510A to the end operation, and method 500A ends.

[0100] Figure 5B Another affix-based method, 500B, is used to determine the language of a text string based on a single language detection model, and to perform subsequent actions based on that determination. Method 500B begins with a start operation, and the flow moves to operation 502B.

[0101] In Operation 502B, a language detection model for a first language is maintained. The language detection model may include a first list comprising the identities of multiple prefixes from a first language corpus, and the weight of each of those prefixes. The language detection model may also include a second list comprising the identities of multiple suffixes from the corpus, and the weight of each of those suffixes. The weights may correspond to the frequencies of the prefixes and suffixes in the corpus.

[0102] The process can continue from operation 502B to the preprocessing step (not shown), which can quickly accept the word (e.g., classify the word as corresponding to a language) because the word is in the list of common words, or can reject the word (e.g., not move forward through the application of the language detection model) because the word does not contain vowels.

[0103] According to the examples, the language detection model may additionally include a list containing the identities of multiple initial consonants from the corpus, and the weight of each of those initial consonants. The language detection model may also additionally include a list containing the identities of multiple final consonants from the corpus, and the weight of each of those final consonants. In some examples, the language detection model may additionally include a list of multiple vowel sequences from the corpus, and the weight of each of those sequences.

[0104] The process continues from operation 502B to operation 504B, in which a language detection model is applied to words in the text string. Applying the language detection model to words may include determining prefix scores for prefixes included in the words, where the prefix scores correspond to the frequency of the prefixes in the corpus. Applying the language detection model to words may also include determining suffix scores for suffixes included in the words, where the suffix scores correspond to the frequency of the suffixes in the corpus. In the example, the language detection model may be applied to every word in the text string.

[0105] When the applied model is running, the prefixes and suffixes in the tag list (e.g., weighted prefix and prefix sequence 404 and weighted suffix and suffix sequence 406) can be sorted in descending order of length. If a word begins with a prefix or ends with a suffix, it can be determined that the beginning or end of the remaining substring begins or ends with a legal consonant sequence (including empty sequences, if allowed in the language). If a consonant sequence is determined to be illegal, or if the remainder of the word does not contain vowels, the next shortest prefix or suffix matching that word can be identified in the list. For example, in the word “render”, training might determine that “re-” is a prefix, but then discard it because “nd” is not in the list of consonant sequences that can legally begin an English word. This step can also be used to avoid incorrectly counting “-ion” in “lion” as a suffix.

[0106] The process continues from operation 504B to operation 506B. In operation 506B, at least the language matching score between the text string and the first language is calculated based on the prefix score and the suffix score.

[0107] The process continues from operation 506B to operation 508B. In operation 508B, it is determined that the language matching score meets the threshold.

[0108] The process continues from operation 508B to operation 510B, where, based on the determination that the language matching score meets a threshold, subsequent actions are performed. In this example, subsequent actions may include applying a language processing engine specifically for the first language to the text string. In other examples, performing subsequent actions may include downloading a language pack library for the first language to the computing device where the text string was initially input. The language pack library for the first language may include an embedding library (e.g., the BERT library, the ELMo library).

[0109] The process moves from operation 510B to the end operation, and method 500B ends.

[0110] Figure 5C Method 500C is another method 500C that uses a single language detection model to determine the language of a text string and performs subsequent actions based on that determination. Method 500C will... Figure 5B The affix-based method described in [the text] and Figure 5A This is combined with the syllable-based method described in [the document]. Method 500C begins with the start operation, and the flow moves to operation 502C.

[0111] The process continues from operation 502C to operation 504C. In operation 504C, the preprocessing step can quickly accept a word because it is in the list of common words, or it can reject the word because it does not contain a vowel.

[0112] The process continues from operation 504C to operation 506C. In operation 506C, it will be related to... Figure 5B The described affix-based language identification process is applied to text strings.

[0113] The process continues from operation 506C to operation 508C. In operation 508C, it will be combined with... Figure 5A The syllable-based language detection process described herein is applied to text strings.

[0114] The process continues from operation 508C to operation 510C. In operation 510C, it is determined that the language matching score meets the threshold.

[0115] The process continues from operation 510C to operation 512C, where subsequent actions are performed based on a determination that the language matching score meets a threshold. In one example, the subsequent action may include applying a language processing engine specifically for the first language to the text string. In other examples, performing the subsequent action may include downloading a language pack library for the first language to the computing device where the text string was initially input. The language pack library for the first language may include an embedded library (e.g., the BERT library, the ELMo library).

[0116] The process moves from operation 512C to the end operation, and method 500C ends.

[0117] Figure 5D Method 500D is an application based on multiple language detection models to determine the language of a text string and to perform subsequent actions based on that determination. Method 500D begins with a start operation, and the flow moves to operation 502D.

[0118] In Operation 502D, a first language detection model is maintained for a first language. The first language detection model may include a first list comprising the identities of a first plurality of initial consonants from a first language corpus, and the weight of each of the first plurality of initial consonants. The first language detection model may also include a second list comprising the identities of a first plurality of vowel sequences from the first language corpus, and the weight of each of the first plurality of final consonants. The first language detection model may also include a third list comprising the identities of a first plurality of final consonants from the first language corpus, and the weight of each of the first plurality of final consonants. In some examples, the first language detection model may also include a list of prefixes and suffixes from the first language corpus, and the weight of each of these prefixes and suffixes. In some examples, the first language detection model may also include a list of common words from the first language corpus, and the weight of each of these common words. The weights may correspond to the frequency of each text unit (tag) in the first language corpus.

[0119] The process continues from operation 502D to operation 504D, in which a second language detection model for a second language is maintained. The second language detection model may include a fourth list containing the identities of a second plurality of initial consonants from the second language corpus, and the weight of each of these initial consonants. The second language detection model may also include a fifth list containing the identities of a second plurality of vowel sequences from the second language corpus, and the weight of each of these final consonants. The second language detection model may also include a sixth list containing the identities of a second plurality of final consonants from the second language corpus, and the weight of each of these final consonants. In some examples, the second language detection model may also include a list of prefixes and suffixes from the second language corpus, and the weight of each of these prefixes and suffixes. In some examples, the first language detection model may also include a list of common words from the first language corpus, and the weight of each of these common words. The weights can correspond to the frequency of each text unit (tag) in the second language corpus.

[0120] The process continues from operation 504D to operation 506D, where the first language detection model is applied to words in the text string. In the example, the first language detection model can be applied to every word in the text string.

[0121] The process continues from Operation 506D to Operation 508D. In Operation 508D, a first language matching score is calculated between the text string and the first language, based on applying the first language detection model to the text string. The first language matching score can be calculated at least in part based on the first language initial consonant score determined for initial consonant clusters included in words and the first language final consonant score determined for final consonant clusters included in words.

[0122] The process continues from operation 508D to operation 510D, where the second language detection model is applied to the text string. In the example, the second language detection model can be applied to each word in the text string.

[0123] The process continues from Operation 510D to Operation 512D. In Operation 512D, a second language matching score is calculated between the second string and the second language, based on applying the second language detection model to the text string. The second language matching score can be calculated at least in part based on the initial consonant score of the second language determined for initial consonant clusters included in words and the final consonant score of the second language determined for final consonant clusters included in words.

[0124] The process continues from operation 512D to operation 514D. In operation 514D, it is determined that the first language matching score is greater than the second language matching score.

[0125] The process continues from operation 514D to operation 516D, in which, based on the first language matching score being greater than the second language matching score, a subsequent action associated with the first language is performed. In one example, the subsequent action may include applying a language processing engine dedicated to the first language to the text string. In other examples, performing the subsequent action may include downloading a language pack library for the first language to the computing device where the text string was initially input. The language pack library for the first language may include an embedding library (e.g., the BERT library, the ELMo library).

[0126] The process moves from operation 516D to the end operation, and method 500D ends.

[0127] Figure 6 and Figure 7 A mobile computing device 600 is illustrated, such as a mobile phone, smartphone, wearable computer, tablet computer, e-reader, laptop computer, AR-compatible computing device, or VR computing device, which can be used to practice embodiments of this disclosure. (See also...) Figure 6This illustration shows one aspect of a mobile computing device 600 used to implement these aspects. In a basic configuration, the mobile computing device 600 is a handheld computer with input and output elements. The mobile computing device 600 typically includes a display 605 and one or more input buttons 610 that allow a user to input information into the mobile computing device 600. The display 605 of the mobile computing device 600 can also be used as an input device (e.g., a touchscreen display). Optional side input elements 615 (if included) allow for further user input. The side input elements 615 can be rotary switches, buttons, or any other type of manual input element. Alternatively, the mobile computing device 600 can include more or fewer input elements. For example, in some embodiments, the display 605 may not be a touchscreen. In yet another alternative embodiment, the mobile computing device 600 is a portable telephone system, such as a cellular phone. The mobile computing device 600 may also include an optional keypad 635. The optional keypad 635 can be a physical keypad or a “soft” keypad generated on a touchscreen display. In various embodiments, output elements include a display 605 for displaying a graphical user interface (GUI), a visual indicator 620 (e.g., a light-emitting diode), and / or an audio transducer 625 (e.g., a speaker) for displaying a graphical user interface (GUI). In some aspects, the mobile computing device 600 incorporates a vibration transducer for providing haptic feedback to a user. In another aspect, the mobile computing device 600 includes input and / or output ports, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0128] Figure 7 This is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, the mobile computing device 700 can be combined with a system (e.g., architecture) 702 to implement some aspects. In one embodiment, the system 702 is implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 702 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0129] One or more applications 766 may be loaded into memory 762 and run on or associated with operating system 764. Examples of applications include telephone dialer programs, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, and so on. System 702 also includes a non-volatile storage area 768 within memory 762. The non-volatile storage area 768 can be used to store permanent information that should not be lost if system 702 is powered off. Applications 766 can use information and store it in the non-volatile storage area 768, such as emails or other messages used by email applications. A synchronization application (not shown) also resides on system 702 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 768 synchronized with the corresponding information stored on the host computer. It should be understood that other applications may be loaded into memory 762 and run on mobile computing device 700, including instructions for providing and operating the digital assistant computing platform.

[0130] System 702 has a power supply 770, which can be implemented as one or more batteries. The power supply 770 may also include an external power source, such as an AC adapter or an electrically operated docking station for replenishing or charging the batteries.

[0131] System 702 may also include a radio interface layer 772, which performs the functions of transmitting and receiving radio frequency communications. Radio interface layer 772 facilitates wireless connectivity between system 702 and the "external world" via a communications operator or service provider. Transmissions to and from radio interface layer 772 are conducted under the control of operating system 764. In other words, communications received by radio interface layer 772 can be propagated to application 766 via operating system 764, and vice versa.

[0132] A visual indicator 620 can be used to provide visual notifications, and / or an audio interface 774 can be used to generate auditory notifications via an audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light-emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to a power supply 770 such that when activated, they remain on for the duration indicated by the notification mechanism, even if the processor 760 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 774 is used to provide and receive auditory signals from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 774 can also be coupled to a microphone to receive auditory input, such as for facilitating telephone conversations. According to embodiments of this disclosure, the microphone can also be used as an audio sensor to facilitate control notifications, as described below. System 702 may also include a video interface 776, which enables the onboard camera 630 to record still images, video streams, etc.

[0133] The mobile computing device 700 implementing system 702 may have additional features or functions. For example, the mobile computing device 700 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 7 The non-volatile storage region 768 is shown in the middle.

[0134] Data / information generated or captured by mobile computing device 700 and stored via system 702 can be locally stored on mobile computing device 700, as described above, or the data can be stored on any number of storage media accessible by the device via radio interface layer 772 or via a wired connection between mobile computing device 700 and an independent computing device associated with mobile computing device 700, such as a server computer in a distributed computing network (like the Internet). It should be understood that such data / information can be accessed via mobile computing device 700 via radio interface layer 772 or via a distributed computing network. Similarly, according to well-known data / information transmission and storage methods, including email and collaborative data / information sharing systems, such data / information can be easily transferred between computing devices for storage and use.

[0135] Figure 8This is a block diagram of the physical components (e.g., hardware) of a computing device 800 that can be used to practice various aspects of this disclosure. The computing device components described below may have computer-executable instructions for training or executing one or more language detection models. In a basic configuration, the computing device 800 may include at least one processing unit 802 and system memory 804. Depending on the configuration and type of the computing device, system memory 804 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of these memories. System memory 804 may include an operating system 805 adapted to run one or more linguistics and / or natural language processing programs. For example, operating system 805 may be adapted to control the operation of computing device 800. Furthermore, embodiments of this disclosure can be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration in Figure 8 The components within the dashed line 808 are shown. The computing device 800 may have additional features or functions. For example, the computing device 800 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 8 The image is shown by a removable storage device 809 and a non-removable storage device 810.

[0136] As described above, multiple program modules and data files can be stored in system memory 804. When executed on processing unit 802, program module 806 (e.g., language detection application 820) can perform processes including, but not limited to, those described herein. Language detection application 820 may include language detection module 124 / 500C, which can read any or all language models described as final weighted individual language models 250 or 402. Language model data 120 may include one or more such models, which can be as follows: Figure 2 and Figure 3 It can be trained as described in the text, and can be trained as described in the text. Figure 5A , Figure 5B , Figure 5C and / or Figure 5D It was deployed as described in the text.

[0137] Furthermore, embodiments of this disclosure can be practiced in circuits including discrete electronic components, packages containing logic gates or integrated electronic chips, circuits utilizing microprocessors, or on a single chip including electronic components or a microprocessor. For example, embodiments of this disclosure can be practiced via a System-on-a-Chip (SOC), wherein... Figure 8Each or many of the components shown can be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operating via the SoC, the capabilities described herein regarding the client switching protocol can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 800. Embodiments of this disclosure can also be practiced using other techniques capable of performing logical operations such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, embodiments of this disclosure can be practiced in general-purpose computers or any other circuit or system.

[0138] The computing device 800 may also have one or more input devices 812, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. Multiple output devices 814, such as a monitor, speaker, printer, etc., may also be included within such devices. The above devices are examples, and other devices may be used. The computing device 800 may include one or more communication connections 816 that allow communication with other computing devices 850. Examples of suitable communication connections 816 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; Universal Serial Bus (USB); parallel and / or serial ports.

[0139] As used herein, the term "computer-readable medium" can include computer storage media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 804, removable storage device 809, and non-removable storage device 810 are examples of computer storage media (e.g., memory storage). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other storage technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and can be accessed by computing device 800. Any such computer storage medium can be part of computing device 800. Computer storage media does not include carrier waves or other propagated or modulated data signals. Computer-readable storage devices do not include carrier waves or other propagated or modulated data signals.

[0140] Communication media can be embodied in computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transmission mechanisms, and include any information delivery medium. The term "modulated data signal" can describe a signal having one or more characteristics set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media, such as wired networks or direct-line connections, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0141] Figure 9 One aspect of a system architecture for processing data received by a computing system from a remote source is illustrated, such as a personal / general purpose computer 904, a tablet computing device 906, or a mobile computing device 908 as described above. Content displayed at server device 902 can be stored in different communication channels or other storage device types. For example, various documents can be stored using a directory service 922, a web portal 924, an email service 926, an instant messaging repository 928, or a social networking site 930. Program module 806 can be used by a client communicating with server device 902, and / or program module 806 can be used by server device 902. Server device 902 can provide data to and from client computing devices such as personal / general purpose computers 904, tablet computing devices 906, and / or mobile computing devices 908 (e.g., smartphones) via network 915. For example, the above references... Figures 6-8 The described computer system may be embodied in a personal / general purpose computer 904, a tablet computing device 906, and / or a mobile computing device 908 (e.g., a smartphone). In addition to receiving graphics data that can be used for preprocessing at the graphics originating system or post-processing at the receiving computing system, any of these embodiments of the computing device may obtain content from a repository 916.

[0142] For example, aspects of this disclosure have been described above with reference to block diagrams and / or operational diagrams of methods, systems, and computer program products according to various aspects of this disclosure. The functions / actions marked in the boxes may not occur in the order shown in any flowchart. For example, two boxes shown consecutively may actually be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order, depending on the functions / actions involved.

[0143] The descriptions and illustrations of one or more aspects provided in this application are not intended to limit or restrict the scope of this disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey ownership and enable others to make and use the claimed disclosure as the best mode. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Whether shown and described in combination or separately, various features (structural and methodological) are intended to be selectively included or omitted to produce embodiments having a particular set of features. Given the descriptions and illustrations provided in this disclosure, those skilled in the art can contemplate variations, modifications, and alternatives falling within the spirit of the broader aspects of the overall inventive concept embodied in this application without departing from the broader scope of the claimed disclosure.

[0144] The various embodiments described above are for illustrative purposes only and should not be construed as limiting the appended claims. Those skilled in the art will readily recognize that various modifications and variations can be made without following the exemplary embodiments and applications shown and described herein, and without departing from the true spirit and scope of the appended claims.

Claims

1. A computer-implemented method for determining the language of a text string, the computer-implemented method comprising: Maintain a language detection model for syllables of a first language, the language detection model comprising: The first list includes the identities of multiple initial consonants from a corpus of the first language, and the weights for each initial consonant among the multiple initial consonants. The second list includes the identities of multiple vowel sequences from the corpus of the first language, and the weights for each vowel sequence among the multiple vowel sequences, and The third list includes the identities of multiple word-final consonants from the corpus, and the weight of each of the multiple word-final consonants; Applying the language detection model to words in the text string, wherein the application includes: Iteratively identify all consecutive syllables within the word; Determine the initial consonant cluster score of the initial consonant clusters included in the identified syllables of the word, wherein the initial consonant cluster score corresponds to the frequency of the initial consonant cluster in the corpus. Determine the vowel sequence score of the vowel sequence included in the identified syllables of the word, wherein the vowel sequence score corresponds to the frequency of the vowel sequence in the corpus, and Determine the word-final consonant cluster score of the word-final consonant clusters included in the identified syllables of the word, wherein the word-final consonant cluster score corresponds to the frequency of the word-final consonant cluster in the corpus; At least based on the initial consonant cluster score and the final consonant cluster score, calculate the language matching score between the text string and the first language; Determine that the language matching score meets the threshold; and Based on the determination that the language matching score meets the threshold, subsequent actions are performed.

2. The computer-implemented method according to claim 1, wherein the language detection model further comprises: The fourth list includes the identities of multiple prefixes from the corpus, and the weights for each of the multiple prefixes.

3. The computer-implemented method according to claim 2 further includes: Determine the prefix score of the prefix included in the word, wherein the prefix score corresponds to the frequency of the prefix in the corpus; as well as The language matching score between the text string and the first language is further calculated based on the prefix score.

4. The computer-implemented method according to claim 1, wherein the language detection model further comprises: The third list includes the identities of multiple suffixes from the corpus, and the weights for each of the multiple suffixes.

5. The computer-implemented method according to claim 4, further comprising: Determine the suffix score of the suffixes included in the word, wherein the suffix score corresponds to the frequency of the suffix in the corpus; as well as The language matching score between the text string and the first language is further calculated based on the suffix score.

6. The computer-implemented method according to claim 2, further comprising: Determine the prefix score of the prefix sequence included in the word, wherein the prefix score corresponds to the frequency of the prefix sequence in the corpus; as well as The language matching score between the text string and the first language is further calculated based on the prefix score.

7. The computer-implemented method according to claim 4, further comprising: Determine the suffix scores of the suffix sequences included in the words, wherein the suffix scores correspond to the frequency of the suffix sequences in the corpus; and The language matching score between the text string and the first language is further calculated based on the suffix score.

8. The computer-implemented method according to claim 1, wherein performing the subsequent action comprises: A language processing engine specific to the first language is applied to the text string.

9. The computer-implemented method according to claim 1, wherein performing the subsequent action comprises: The language pack library for the first language is downloaded to the computing device where the text string was initially input.

10. The computer-implemented method of claim 9, wherein the language package library for the first language includes a language embedding library.

11. The computer-implemented method according to claim 1, further comprising: Determine the cross-marking score between the suffix and the prefix of the word; and in The language matching score between the text string and the first language is further calculated based on the cross-marking score.

12. A system for determining the language of a text string, comprising: Memory, used to store executable program code; as well as A processor, functionally coupled to the memory, the processor responding to computer-executable instructions contained in the program code, and capable of operating to: Maintain a first language detection model for the first language, the first language detection model including: The first list includes the identities of the first plurality of syllable-word initial consonants from the corpus of the first language, and the weights for each of the first plurality of syllable-word initial consonants. The second list includes the identities of a first plurality of vowel sequences from a corpus of the first language, and the weights for each vowel sequence within the first plurality of vowel sequences, and The third list includes the identity of the first plurality of syllable-final consonants from the corpus of the first language, and the weight of each syllable-final consonant in the first plurality of syllable-final consonants. Maintain a second language detection model for a second language, the second language detection model including: The fourth list includes the identity of the second plurality of syllable-word initial consonants from the corpus of the second language, and the weight of each syllable-word initial consonant in the second plurality of syllable-word initial consonants, and The fifth list includes the identities of a second plurality of vowel sequences from a corpus of the second language, and the weights for each vowel sequence within the second plurality of vowel sequences, and The sixth list includes the identity of the second plurality of syllable-final consonants from the corpus of the second language, and the weight of each syllable-final consonant in the second plurality of syllable-final consonants; The first language detection model is iteratively applied to the words in the text string; Based on applying the first language detection model to the text string, a first language matching score is calculated between the text string and the first language; The second language detection model is iteratively applied to the words in the text string; Based on applying the second language detection model to the text string, a second language matching score is calculated between the text string and the second language; Determine that the first language matching score is greater than the second language matching score and is within a threshold; and If the first language matching score is greater than the second language matching score and is within the threshold, then perform the subsequent action associated with the first language.

13. The system according to claim 12, wherein the first language detection model further comprises: The seventh list includes the identities of multiple prefixes from the corpus of the first language, and the weights for each of the multiple prefixes; as well as The eighth list includes the identities of multiple suffixes from the corpus of the first language, and a weight for each of the multiple suffixes.

14. The system of claim 13, wherein the second language detection model further comprises: The ninth list includes the identities of a second plurality of prefixes from the corpus of the second language, and the weights for each of the second plurality of prefixes; as well as The tenth list includes the identities of a second plurality of suffixes from the corpus of the second language, and the weights for each of the second plurality of suffixes.

15. A computer-readable storage device including executable instructions that, when executed by a processor, help determine the language of a text string, the computer-readable storage device including instructions executable by the processor for: Maintain a language detection model for the first language, the language detection model comprising: The first list includes the identities of multiple prefixes from a corpus of the first language, and the weights for each of the multiple prefixes, and The second list includes the identities of multiple suffixes from the corpus, and the weight for each of the multiple suffixes; Applying the language detection model to words in the text string, wherein the application includes: Determine the prefix scores of the prefixes included in the words, wherein the prefix scores correspond to the frequency of the prefixes in the corpus, and Determine the suffix score of the suffixes included in the word, wherein the suffix score corresponds to the frequency of the suffix in the corpus; Calculate the language matching score between the text string and the first language based at least on the prefix score and / or the suffix score; Determine that the language matching score meets the threshold; and Based on the determination that the language matching score meets the threshold, subsequent actions are performed.

16. The computer-readable storage device of claim 15, wherein the language detection model further comprises: The third list includes the identities of multiple initial consonant sequences from the corpus, and the weights for each initial consonant sequence among the multiple initial consonant sequences; The fourth list includes the identities of multiple vowel sequences from the corpus, and the weights for each vowel sequence among the multiple vowel sequences.

17. The computer-readable storage device of claim 16, wherein the instructions are further executable by the processor for: Remove optional prefixes or optional suffixes from the words; and Iterate through the syllables in the stem of the word.

18. The computer-readable storage device of claim 15, wherein the language detection model further comprises: The third list includes the identities of multiple word-final consonants from the corpus, and the weights for each of the multiple word-final consonants.

19. The computer-readable storage device of claim 18, wherein the instructions are further executable by the processor for: Determine the final consonant scores of the final consonants included in the word, wherein the final consonant scores correspond to the frequencies of the final consonants in the corpus; and in, The language matching score between the text string and the first language is further calculated based on the word-final consonant score.

20. The computer-readable storage device of claim 15, wherein, when performing a subsequent action, the instructions are further executable by the processor for: The language pack library for the first language is downloaded to the computing device where the text string was initially input.

Images