Language recognition method, device, electronic device and readable storage medium

Abstract

The present application relates to the field of natural language processing technology of artificial intelligence, and discloses a language identification method, device, electronic device and readable storage medium. The language identification method includes: obtaining a language text to be identified, determining the language category corresponding to the language text; matching the language rule information corresponding to the language category with the language text; the language rule information includes the language rule information corresponding to at least one language in the language category; if the corresponding language rule information is not matched, determining the language classification model corresponding to the language category; classifying the language text based on the language classification model, and determining the language of the language text. The language identification method provided by the present application can improve the accuracy of language identification.

Description

Technical Field

[0001] The present application relates to the field of language processing technology. Specifically, the present application relates to a language recognition method, device, electronic device and readable storage medium. Background Art

[0002] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0003] In many natural language processing scenarios, language identification is required. Language identification (LI) is to identify the language of natural language written in the form of strings in computers. There are many related works. Currently, language identification is also used as part of many text processing tasks. For example, language text is first identified and then translated after the language is identified. However, in many cases, the language of the text is identified by statistical methods, which has a low accuracy rate. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above technical defects, and the following technical solutions are proposed:

[0005] In a first aspect, a language recognition method is provided, comprising:

[0006] Obtaining a language text to be recognized and determining a language category corresponding to the language text;

[0007] Matching the language rule information corresponding to the language family category with the language text; the language rule information includes the language rule information corresponding to at least one language in the language family category;

[0008] If no corresponding language rule information is matched, a language classification model corresponding to the language family category is determined;

[0009] The language text is classified based on the language classification model to obtain the language of the language text.

[0010] In an optional embodiment of the first aspect, determining the language family category corresponding to the language text includes:

[0011] Using a preset encoding method to obtain the encoding of the language text;

[0012] The coding interval of the code is determined, and the language category corresponding to the language text is determined based on the coding interval.

[0013] In an optional embodiment of the first aspect, before determining the language classification model corresponding to the language family category, the method further includes:

[0014] Determine the number of languages ​​included in each of the multiple language categories;

[0015] Based on the determined number of languages, initial language classification models corresponding to the multiple language categories are respectively obtained, and the number of training texts corresponding to the multiple language categories is determined;

[0016] A corresponding number of training texts are obtained, and the corresponding initial language classification models are trained respectively to obtain language classification models corresponding to the various language categories.

[0017] In an optional embodiment of the first aspect, classifying the language text based on the language classification model to obtain the language of the language text includes:

[0018] Extract semantic vectors from language texts;

[0019] The semantic vector is classified based on the language classification model to obtain the language of the language text.

[0020] In an optional embodiment of the first aspect, extracting a semantic vector of a language text includes:

[0021] splitting the language text into at least one text unit;

[0022] Obtaining a text unit vector corresponding to each text unit, and determining a position vector corresponding to each text unit;

[0023] A semantic vector is obtained based on the text unit vector and the position vector corresponding to each text unit.

[0024] In an optional embodiment of the first aspect, determining a position vector corresponding to each text unit includes:

[0025] For each text unit, determining the position of the text unit in the language text;

[0026] A position vector of the text unit is determined based on the position of the text unit in the language text.

[0027] In an optional embodiment of the first aspect, acquiring a semantic vector based on a text unit vector and a position vector corresponding to each text unit includes:

[0028] For each text unit, add the text unit vector and the position vector of the text unit to obtain the text unit representation corresponding to the text unit;

[0029] The text unit representation of each text unit corresponding to the language text is fused to obtain a semantic vector.

[0030] In a second aspect, a language recognition device is provided, comprising:

[0031] A determination module, used to obtain a language text to be identified and determine a language category corresponding to the language text;

[0032] A matching module, used to match the language rule information corresponding to the language family category with the language text; the language rule information includes the language rule information corresponding to at least one language in the language family category;

[0033] An acquisition module, for determining a language classification model corresponding to the language family category if no corresponding language rule information is matched;

[0034] The recognition module is used to classify the language text based on the language classification model to obtain the language of the language text.

[0035] In an optional embodiment of the second aspect, when determining the language family category corresponding to the language text, the determination module is specifically configured to:

[0036] Using a preset encoding method to obtain the encoding of the language text;

[0037] The coding interval of the code is determined, and the language category corresponding to the language text is determined based on the coding interval.

[0038] In an optional embodiment of the second aspect, a training module is further included, which is used to:

[0039] Determine the number of languages ​​included in each of the multiple language categories;

[0040] Based on the determined number of languages, initial language classification models corresponding to the multiple language categories are respectively obtained, and the number of training texts corresponding to the multiple language categories is determined;

[0041] A corresponding number of training texts are obtained, and the corresponding initial language classification models are trained respectively to obtain language classification models corresponding to the various language categories.

[0042] In an optional embodiment of the second aspect, when the recognition module classifies the language text based on the language classification model to obtain the language of the language text, it is specifically used to:

[0043] Extract semantic vectors from language texts;

[0044] The semantic vectors are classified based on the language classification model to identify the language of the text.

[0045] In an optional embodiment of the second aspect, when extracting the semantic vector of the language text, the recognition module is specifically used to:

[0046] splitting the language text into at least one text unit;

[0047] Obtaining a text unit vector corresponding to each text unit, and determining a position vector corresponding to each text unit;

[0048] A semantic vector is obtained based on the text unit vector and the position vector corresponding to each text unit.

[0049] In an optional embodiment of the second aspect, when determining the position vector corresponding to each text unit, the recognition module is specifically configured to:

[0050] For each text unit, determining the position of the text unit in the language text;

[0051] A position vector of the text unit is determined based on the position of the text unit in the language text.

[0052] In an optional embodiment of the second aspect, when the recognition module obtains the semantic vector based on the text unit vector and the position vector corresponding to each text unit, it is specifically used to:

[0053] For each text unit, add the text unit vector and the position vector of the text unit to obtain the text unit representation corresponding to the text unit;

[0054] The text unit representation of each text unit corresponding to the language text is fused to obtain a semantic vector.

[0055] In a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the language recognition method shown in the first aspect of the present application is implemented.

[0056] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the language recognition method shown in the first aspect of the present application is implemented.

[0057] The beneficial effects of the technical solution provided by this application are:

[0058] The language family category of the language text is first determined, and the language rule information of the language family category is matched with the language text to determine the language of the language text. If the corresponding language rule information is not matched, the language of the language text is determined based on the language classification model corresponding to the language family category. Firstly, determining the language family category can narrow the scope of the languages ​​to be determined, and then matching is performed based on the language rule information. If the language can be directly identified, the computational complexity of language recognition can be reduced. If the language cannot be directly identified, further identification is performed through the language classification model to improve the accuracy of language recognition.

[0059] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0061] Figure 1 An application environment diagram of a language recognition method provided in an embodiment of the present application;

[0062] Figure 2 A flow chart of a language recognition method provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a language recognition method provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of a solution for obtaining a language classification model in the example provided in the embodiment of the present application;

[0065] Figure 5 A flow chart of a language recognition method provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of a method for obtaining a semantic vector in an example provided in an embodiment of the present application;

[0067] Figure 7 A flowchart of a language recognition method in an example provided in an embodiment of the present application;

[0068] Figure 8 A flowchart of a language recognition method in an example provided in an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of the structure of a language recognition device provided in an embodiment of the present application;

[0070] Fig.10A schematic diagram of the structure of an electronic device for language recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0072] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0073] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0074] The language recognition method, device, electronic device and computer-readable storage medium provided in this application are intended to solve the above technical problems in the prior art.

[0075] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0076] The language recognition method provided in this application can be applied to the following translation application scenarios, which may specifically include: a user inputs a language text to be recognized, that is, Figure 1 For the source sentence in the language text to be recognized, the language recognition is performed; if the language is recognized, the corresponding language translation is performed, such as Figure 1 As shown, if the language is Uyghur and the target language for translation is Chinese, Uyghur-Chinese translation is performed; if the language is Chinese and the target language for translation is Uyghur, Chinese-Uyghur translation can be performed; if the language is Tibetan and the target language for translation is Chinese, Tibetan-Chinese translation is performed.

[0077] The above-mentioned language recognition method can be performed in a terminal, and can also be applied to a server.

[0078] Technicians in this technical field can understand that the "terminal" used here can be a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a MID (Mobile Internet Device), etc.; the "server" can be implemented by an independent server or a server cluster consisting of multiple servers.

[0079] A possible implementation method is provided in the embodiment of the present application, such as Figure 2 As shown, a language recognition method is provided, which can be applied to a terminal or a server and can include the following steps:

[0080] Step S201, obtaining a language text to be recognized, and determining a language category corresponding to the language text.

[0081] In the present application, language categories may be divided according to the alphabetic system of the language text, or the correspondence between certain pronunciations, vocabulary, grammatical rules, and text units. Languages ​​of the same language category have similarities.

[0082] It should be noted that the language family categories in this application are not limited to the current nine language families, namely the Sino-Tibetan, Indo-European, Altaic, Semitic, Uralic, Caucasian (Iberian-Caucasian), Austronesian (Malayo-Polynesian), South Asian and Dravidian languages, which are divided according to phonetics, vocabulary and grammatical rules. They can also be divided according to other similarities, for example, according to the encoding rules of text units.

[0083] In one implementation, the language family category is divided according to common language family division rule information such as vocabulary and grammar rules, and the step S201 of determining the language family category corresponding to the language text may include:

[0084] The preset language family rule classification information is matched with the language text to determine the language family category corresponding to the language text.

[0085] Specifically, if the language category is divided according to general language division rule information, the language category is directly determined according to the language division rule information.

[0086] In another embodiment, the language family category is divided according to the alphabetic system of the language text, that is, according to the encoding method; then the step S201 of determining the language family category corresponding to the language text may include:

[0087] (1) Using a preset encoding method to obtain the encoding of the language text;

[0088] (2) Determine the coding range in which the code is located, and determine the language category corresponding to the language text based on the coding range.

[0089] The encoding method may be an encoding specification formulated when establishing the relationship between a text unit and a binary system. For example, the encoding method may include ASCII (American Standard Code for Information Interchange), Unicode symbol set, and the like.

[0090] Specifically, the language text is encoded, and different language categories correspond to different coding intervals. The corresponding language category can be determined according to the coding interval where the code is located.

[0091] Step S202, matching the language rule information corresponding to the language family category with the language text; the language rule information includes language rule information corresponding to at least one language in the language family category.

[0092] Specifically, the language rule information corresponding to the language family category is determined, and the language rule information corresponding to each language belonging to the language family category is matched with the language text.

[0093] The language rule information may include the writing order of letters corresponding to each language.

[0094] For example, Uyghur and Kazakh languages ​​both belong to the Arabic alphabet system. Although most of the letters are overlapping, the writing order of some of their letters is clearly different.

[0095] Specifically, the preset language rule information of each language, that is, the letter writing order, can be matched with the language text one by one to determine whether the language text conforms to the language rule information.

[0096] Step S203: if the corresponding language rule information is not matched, then determine the language classification model corresponding to the language family category.

[0097] In the specific implementation process, if the corresponding language rule information is matched, the language can be determined directly based on the matched language rule information. If the language rule information is not matched, it is necessary to further determine the language of the language text based on the language classification model.

[0098] Specifically, for each language category, a corresponding language classification model can be set. Each language classification model is trained based on the training text corresponding to the language category, wherein the training text corresponding to the language category may include training text corresponding to at least one language. The specific method of obtaining the language classification model will be explained in detail below.

[0099] Step S204: classify the language text based on the language classification model to identify the language of the language text.

[0100] Specifically, the language text is input into a language classification model corresponding to the language family category to determine the language of the language text.

[0101] like Figure 3 As shown, the language category corresponding to the language text can be determined first, and the language rule information of the language category can be matched with the language text. If the language cannot be identified based on the language classification information, the language can be identified based on the language classification model corresponding to the language category.

[0102] The language identification method provided in this embodiment first determines the language family category of the language text, and then matches the language rule information of the language family category with the language text to determine the language of the language text. If the corresponding language rule information is not matched, the language of the language text is determined based on a language classification model corresponding to the language family category. Firstly, determining the language family category can narrow the range of languages ​​to be determined, and then matching is performed based on the language rule information. If the language can be directly identified, the amount of calculation for language recognition can be reduced. If the language cannot be directly identified, further identification is performed through the language classification model to improve the accuracy of language recognition.

[0103] The following will further explain the process of acquiring the language classification model in conjunction with a specific implementation method.

[0104] The present application provides a possible implementation method. Before determining the language classification model corresponding to the language family category in step S203, the following may also be included:

[0105] (1) Determine the number of languages ​​included in each of the multiple language categories.

[0106] Among them, a language category may include at least one language. For example, if the language category is Arabic, the language may include Uyghur and Kazakh.

[0107] (2) Based on the determined number of languages, initial language classification models corresponding to the multiple language categories are respectively obtained, and the number of training texts corresponding to the multiple language categories is determined.

[0108] Specifically, the more languages ​​the same language family category contains, the greater the difficulty of classification; the more languages ​​the more languages ​​the larger the size of the initial language classification model, that is, the number of languages ​​the language family category contains is positively correlated with the size of the initial language classification model; the more languages ​​the same language family category contains, the more training texts are required, that is, the number of languages ​​the language family category contains is positively correlated with the number of training texts.

[0109] Specifically, there are many options for language classification models, such as language classification models based on Bi-LSTM (Bi-directional Long Short-Term Memory), CNN (Convolutional Neural Networks) or Self-attention.

[0110] Specifically, considering the factors of online, lightweight and speed, FastText (Fast Text Classification Algorithm) can also be used as a language classification model. FastText has the characteristics of fast, efficient and simple.

[0111] (3) Obtain a corresponding number of training texts, train the corresponding initial language classification models respectively, and obtain language classification models corresponding to the various language categories.

[0112] Specifically, if the training text is in a known language, the known language can be set as the output label of the language classification model, and the training text can be converted into a corresponding vector representation and input into the initial language classification model corresponding to the language category. The model parameters are trained, and the model capability and degree of fit are adjusted by setting parameters such as the network size and the number of training rounds to obtain the final language classification model.

[0113] For example, for Chinese, Cantonese, English and Zhuang, the amount of training text data can be 2.4M (million lines), and the amount of training text data for Uyghur and Kazakh can be about 2.3M (million lines). Since there are only two language categories that need to be classified for a single alphabet system, it is relatively simple for the classification task, so the language classification model used is relatively small, and its parameters are as follows: the vocabulary size of the three models (that is, the number of different words or tokens in the training text) is 1.9M, 26k, and 12k for Chinese and Cantonese respectively; the maximum length of training data is 120 words, the network hidden layer size of the language classification model is 100, the word embedding size of the language classification model is 100, the n-gram (n-gram) is 2, the dropout size is 0.5, the batch size is 100 during training, the initial learning rate is 1e-5, and a total of 20 rounds of training are used.

[0114] like Figure 4 As shown, the number of languages ​​included in language category 1 to language category N can be determined respectively, that is, the number of languages ​​1 included in language category 1, the number of languages ​​2 included in language category 2, until the number of languages ​​N included in language category N is determined, and the corresponding initial language classification models 1 to initial language classification models N are determined according to the number of languages ​​1 to the number of languages ​​N, and the number of training texts 1 to training texts N is determined, and the initial language classification models 1 to initial language classification models N are trained based on the training texts 1 to training texts N, so as to obtain language classification models 1 to language classification models N corresponding to language category 1 to language category N.

[0115] In the above embodiment, the number of languages ​​included in the language family category is positively correlated with the size of the initial language classification model, and the number of languages ​​included in the language family category is positively correlated with the number of training texts. This can save data resources in the training process while ensuring the accuracy of language recognition and improve the computational efficiency of the classification process.

[0116] The above embodiment describes the process of acquiring a language classification model. The following will further describe the process of classifying a language text to identify the language in conjunction with the embodiment.

[0117] A possible implementation method is provided in the embodiment of the present application, such as Figure 5 As shown, the step S204 of classifying the language text based on the language classification model to identify the language of the language text may include:

[0118] Step S410: extracting the semantic vector of the language text.

[0119] The language text may include at least one text unit, and the text unit may be the smallest component form of the language text. For example, if the language text is in English, the text unit is a letter; if the language text is in Chinese, the text unit may be a Chinese character.

[0120] Specifically, the semantic vector of the language text may be extracted according to the text units contained in the language text.

[0121] Step S420, classifying the semantic vector based on the language classification model to obtain the language of the language text.

[0122] Specifically, Figure 6 As shown in the figure, the language classification model can include a network hidden layer and an output layer, and the text unit vector (shown as w in the figure) 1 tow n ) and the corresponding position vector (shown as p in the figure 1 to p n ) is input into the hidden layer of the network to obtain the semantic vector, and the semantic vector is classified to obtain the language output by the output layer.

[0123] In the above embodiment, the semantic vector of the text is determined by combining the position vector and the text unit vector, and the language of the text is determined by classifying the semantic vector, so that the accuracy of language recognition can be improved.

[0124] A possible implementation method is provided in the embodiment of the present application. Extracting the semantic vector of the language text in step S410 may include:

[0125] (1) Split a language text into at least one text unit.

[0126] The text unit may be the smallest component form of the language text. For example, if the language text is English, the text unit is a letter; if the language text is Chinese, the text unit may be a Chinese character.

[0127] (2) Obtain the text unit vector corresponding to each text unit.

[0128] Specifically, the text unit may be input into a preset neural network to obtain a corresponding text unit vector; or the correspondence between multiple text units and text unit vectors may be preset, and for each text unit, the corresponding text unit vector may be directly queried.

[0129] (3) Based on the text unit vector corresponding to each text unit, the semantic vector of the language text is obtained.

[0130] Specifically, a corresponding weight may be set for each text unit, and a semantic vector may be calculated based on the weight of each text unit and the text unit vector. For example, a weighted sum of each text unit vector may be calculated to obtain the semantic vector.

[0131] In the above implementation, a semantic vector is obtained based on multiple text units of multiple language texts and the weight corresponding to each text unit. In another implementation, the semantic vector may also be calculated by considering the position of each text unit.

[0132] A possible implementation method is provided in the embodiment of the present application. Extracting the semantic vector of the language text in step S410 may include:

[0133] (1) Split a language text into at least one text unit.

[0134] The text unit may be the smallest component form of the language text. For example, if the language text is English, the text unit is a letter; if the language text is Chinese, the text unit may be a Chinese character.

[0135] (2) Obtain the text unit vector corresponding to each text unit, and determine the position vector corresponding to each text unit.

[0136] Specifically, the text unit can be input into a preset neural network to obtain the corresponding text unit vector; alternatively, the corresponding relationship between multiple text units and text unit vectors can be preset, and for each text unit, the corresponding text unit vector can be directly queried.

[0137] Specifically, the position vector corresponding to the text unit can be determined by the position of the text unit in the language text, and the specific determination process of the position vector will be elaborated in detail below.

[0138] (3) Based on the text unit vector and position vector corresponding to each text unit, obtain the semantic vector.

[0139] Specifically, for each text unit, the corresponding text unit vector and position vector can be fused to obtain the semantic vector.

[0140] Specifically, the text unit vector and position vector can be concatenated, or the text unit vector and position vector can be added. The specific process of obtaining the semantic vector will be elaborated in detail below.

[0141] In the above embodiments, by combining the position vector and the text unit vector to determine the semantic vector of the text and classifying the semantic vector to determine the language type of the language text, the accuracy of language type recognition can be improved.

[0142] The process of obtaining the position vector will be elaborated below in conjunction with specific embodiments.

[0143] In the specific implementation process, determining the position vector corresponding to each text unit may include:

[0144] a. For each text unit, determine the position of the text unit in the language text;

[0145] b. Based on the position of the text unit in the language text, determine the position vector of the text unit.

[0146] Specifically, the language text can be first converted as a whole into the corresponding initial position vector, and then the initial position vector can be split in sequence to obtain the position vector of each text unit.

[0147] For example, "Today is Friday" can be converted into [w 1 , w 2 , w 3 , w 4 , w 5 , w 6 according to the position of each character. Then, the position vector corresponding to "今" is [w 1 , the position vector corresponding to "天" is [w 2 , and the position vector corresponding to "是" is [w 3], the position vector corresponding to the “star” is [w 4 ], the position vector corresponding to “period” is [w 5 ], the position vector corresponding to "five" is [w 6 ].

[0148] An embodiment of the present application provides a possible implementation method, which may include: obtaining a semantic vector based on a text unit vector and a position vector corresponding to each text unit;

[0149] a. For each text unit, add the text unit vector and the position vector of the text unit to obtain the text unit representation corresponding to the text unit.

[0150] Specifically, in other implementations, the text unit vector and the position vector may also be concatenated to determine the text unit representation.

[0151] b. Fuse the text unit representation of each text unit corresponding to the language text to obtain a semantic vector.

[0152] Specifically, the text unit representation of each text unit corresponding to the language text is fused by determining a weight coefficient corresponding to each text unit, and calculating the weighted sum between the text unit representations of each text unit according to the weight coefficient corresponding to each text unit to obtain a semantic vector.

[0153] In order to better understand the above language recognition method, Figure 7 As shown, an example of the language recognition method of the present invention is described in detail below:

[0154] In one example, the language identification method provided by the present application may include the following steps:

[0155] Step S700, obtaining a language text to be recognized;

[0156] Step S701, determining the language category corresponding to the language text;

[0157] Step S702, obtaining language rule information corresponding to the determined language category;

[0158] Step S703, determining whether the language text matches the language rule information; if so, executing step S704; if not, executing step S705;

[0159] Step S704, determining the language of the language text based on the matched language rule information;

[0160] Step S705, determining a language classification model corresponding to the language family category;

[0161] Step S706, splitting the language text into at least one text unit;

[0162] Step S707, obtaining a text unit vector corresponding to each text unit, and determining a position vector corresponding to each text unit;

[0163] Step S708, obtaining a semantic vector based on the text unit vector and the position vector corresponding to each text unit;

[0164] Step S709: classify the semantic vector based on the language classification model to identify the language of the language text.

[0165] In order to better understand the above language recognition method, Figure 8 As shown, an example of the language recognition method of the present invention is described in detail below:

[0166] In one example, the language of the language text to be recognized may include domestic minority languages ​​other than Chinese, Cantonese and English, such as Uyghur, Kazakh, Mongolian, Tibetan, Korean, Zhuang and Yi (here Cantonese is a branch of Chinese and can be understood as its dialect), as shown in the following Table 1, which shows all the above languages ​​and their corresponding language symbols:

[0167] Table 1 Languages ​​and their language symbols

[0168] Serial number Language symbol 1 Uyghur ug 2 Kazakh kk 3 Mongolian mn 4 Tibetan language tb 5 Cantonese cn 6 Chinese ca 7 Korean language kr 8 Zhuang za 9 English en 10 Yi language yi 11 Cyrillic Mongolian cmn 12 Cyrillic Kazakh ckk

[0169] The above languages ​​can be classified into Arabic letters, Mongolian letters, Tibetan letters, Chinese letters, Korean letters, Latin letters, Yi letters and Cyrillic letters; according to the already classified language categories, further classification is carried out according to the language classification model based on neural network or language rules to obtain the final language.

[0170] Specifically, when languages ​​are classified, they can be classified into various alphabetic systems (ie, language categories) according to the Unicode encoding table, wherein the encoding range of each alphabetic system is shown in Table 2.

[0171] Table 2 Alphabet system and its coding interval table

[0172]

[0173] Specifically, the coding ranges of different alphabetic systems are divided according to the Unicode coding table to obtain the alphabetic system and its coding range table shown in Table 2; then the Unicode coding method is used to determine the encoding of the language text, and then the coding range in which the encoding is located is determined. For example, if the obtained code is 0x4dff, it can be determined that the coding range in which the code is located is 0x4dff-0x9fff, thereby determining that the language category of the language text is Latin letters.

[0174] like Figure 8 As shown, the language category of the input language text is determined and distinguished according to the language rule information. For example, Uyghur and Kazakh, which both belong to the Arabic alphabet system, have most of the same letters, but the writing order and letter position rules of some letters are clearly distinguished. Among them, the letter position rule means that for a language text of a certain language, a specific letter can only appear in a specific position of the language text, such as only at the beginning or end of a sentence. Therefore, if these language rule information are sufficient to distinguish languages ​​belonging to the same alphabet system (i.e., language category), it can also avoid redundant calculation neural network operations, thereby improving recognition speed and accuracy while reducing the consumption of computing resources. If the language cannot be identified according to the language rule information, further identification can be performed based on the language classification model.

[0175] In order to illustrate the effect of the language recognition method of the present application, further description will be given below in combination with experimental data.

[0176] This application selects monolingual corpus as the test set for each language, and the amount of data ranges from 1,000 to 10,000 lines. For languages ​​with relatively small ambiguity such as Mongolian, Korean, and Yi, they can be directly recognized through part of the rules, and the recognition rate is generally high, so a small amount of monolingual data is selected as the test set. For the recognition of Uyghur, Kazakh, Zhuang, English, Chinese, and Cantonese, a language classification model is used, so more monolingual data is used as the test set to ensure the objectivity of the test results.

[0177] Table 3 is the test result of the present invention. For better comparative performance, open source LangID and LangDetect models are introduced. Since the default LangID and LangDetect do not support certain languages, the present invention utilizes a model (LangID-new) trained again on the same training set by the LangID model. From the experimental results, it can be seen that for the language classification model, the binary classification task is more accurate than the multi-classification task, and the classification based on rules simplifies the difficulty of the subsequent language classification model. In addition, it can be seen from LangID-new that since the similarity between Uygur and Kazakhstan and Chinese and Guangdong is very high, the performance will be affected by each other after these languages ​​are introduced (such as the high recognition accuracy of Kazakhstan, but it affects the recognition of Uyghur). The main reason why English and Zhuang are not affected is that although both English and Zhuang use Latin letters, the composition of their words is still very different and is better distinguished.

[0178] Table 3 Language recognition effects of different systems

[0179]

[0180]

[0181] The above-mentioned language identification method first determines the language family category of the language text, and matches the language rule information of the language family category with the language text to determine the language of the language text. If the corresponding language rule information is not matched, the language of the language text is determined based on the language classification model corresponding to the language family category. First, determining the language family category can narrow the scope of the language to be determined, and then matching is performed based on the language rule information. If the language can be directly identified, the calculation amount of language recognition can be reduced. If the language cannot be directly identified, it is further identified through the language classification model to improve the accuracy of language recognition.

[0182] Furthermore, the number of languages ​​included in the language family category is positively correlated with the size of the initial language classification model, and the number of languages ​​included in the language family category is positively correlated with the number of training texts. This can save data resources in the training process and improve the computational efficiency of the classification process while ensuring the accuracy of language recognition.

[0183] Furthermore, by combining the position vector and the text unit vector to determine the semantic vector of the text, and classifying the semantic vector to determine the language of the text, the accuracy of language recognition can be improved.

[0184] A possible implementation method is provided in the embodiment of the present application, such as Fig. 9 As shown, a language identification device 90 is provided. The language identification device 90 may include: a determination module 901, a matching module 902, an acquisition module 903 and a recognition module 904, wherein:

[0185] A determination module 901 is used to obtain a language text to be recognized and determine a language category corresponding to the language text;

[0186] A matching module 902 is used to match the language rule information corresponding to the language family category with the language text; the language rule information includes the language rule information corresponding to at least one language in the language family category;

[0187] The acquisition module 903 is used to determine the language classification model corresponding to the language family category if the corresponding language rule information is not matched;

[0188] The recognition module 904 is used to classify the language text based on the language classification model to obtain the language of the language text.

[0189] A possible implementation method is provided in an embodiment of the present application. When determining the language category corresponding to the language text, the determination module 901 is specifically configured to:

[0190] Using a preset encoding method to obtain the encoding of the language text;

[0191] The coding interval of the code is determined, and the language category corresponding to the language text is determined based on the coding interval.

[0192] In an optional embodiment of the second aspect, a training module is further included, which is used to:

[0193] Determine the number of languages ​​included in each of the multiple language categories;

[0194] Based on the determined number of languages, initial language classification models corresponding to the multiple language categories are respectively obtained, and the number of training texts corresponding to the multiple language categories is determined;

[0195] A corresponding number of training texts are obtained, and the corresponding initial language classification models are trained respectively to obtain language classification models corresponding to the various language categories.

[0196] In an optional embodiment of the second aspect, when the recognition module 904 classifies the language text based on the language classification model to obtain the language of the language text, it is specifically used to:

[0197] Extract semantic vectors from language texts;

[0198] The semantic vectors are classified based on the language classification model to identify the language of the text.

[0199] In an optional embodiment of the second aspect, when extracting the semantic vector of the language text, the recognition module 904 is specifically used to:

[0200] splitting the language text into at least one text unit;

[0201] Obtaining a text unit vector corresponding to each text unit, and determining a position vector corresponding to each text unit;

[0202] A semantic vector is obtained based on the text unit vector and the position vector corresponding to each text unit.

[0203] In an optional embodiment of the second aspect, when determining the position vector corresponding to each text unit, the recognition module 904 is specifically configured to:

[0204] For each text unit, determining the position of the text unit in the language text;

[0205] A position vector of the text unit is determined based on the position of the text unit in the language text.

[0206] In an optional embodiment of the second aspect, when the recognition module 904 obtains the semantic vector based on the text unit vector and the position vector corresponding to each text unit, it is specifically configured to:

[0207] For each text unit, add the text unit vector and the position vector of the text unit to obtain the text unit representation corresponding to the text unit;

[0208] The text unit representation of each text unit corresponding to the language text is fused to obtain a semantic vector.

[0209] The above-mentioned language identification device first determines the language family category of the language text, and matches the language rule information of the language family category with the language text to determine the language of the language text. If the corresponding language rule information is not matched, the language of the language text is determined based on the language classification model corresponding to the language family category. Firstly, determining the language family category can narrow the scope of the language to be determined, and then matching is performed based on the language rule information. If the language can be directly identified, the calculation amount of language recognition can be reduced. If the language cannot be directly identified, it is further identified through the language classification model to improve the accuracy of language recognition.

[0210] Furthermore, the number of languages ​​included in the language family category is positively correlated with the size of the initial language classification model, and the number of languages ​​included in the language family category is positively correlated with the number of training texts. This can save data resources in the training process and improve the computational efficiency of the classification process while ensuring the accuracy of language recognition.

[0211] Furthermore, by combining the position vector and the text unit vector to determine the semantic vector of the text, and classifying the semantic vector to determine the language of the text, the accuracy of language recognition can be improved.

[0212] The language recognition device for an image in the embodiments of the present disclosure can execute a language recognition method for an image provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the language recognition device for an image in each embodiment of the present disclosure correspond to the steps in the language recognition method for an image in each embodiment of the present disclosure. For the detailed functional description of each module in the language recognition device for an image, please refer to the description in the corresponding language recognition method for an image shown in the previous text, which will not be repeated here.

[0213] Based on the same principle as the method shown in the embodiment of the present disclosure, an electronic device is also provided in the embodiment of the present disclosure, which may include but is not limited to: a processor and a memory; the memory is used to store computer operation instructions; the processor is used to execute the language recognition method shown in the embodiment by calling the computer operation instructions. Compared with the prior art, the language recognition method in the present application can improve the accuracy of language recognition.

[0214] In an alternative embodiment, an electronic device is provided, such as Fig.10 As shown, Fig.10The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0215] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0216] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0217] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0218] The memory 4003 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the contents shown in the above method embodiment.

[0219] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0220] The embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer can execute the corresponding content of the above method embodiment. Compared with the prior art, the language recognition method in the present application can improve the accuracy of language recognition.

[0221] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0222] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0223] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0224] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0225] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that when the computer device executes the computer instructions, the following conditions are achieved:

[0226] Obtaining a language text to be recognized and determining a language category corresponding to the language text;

[0227] Matching the language rule information corresponding to the language family category with the language text; the language rule information includes the language rule information corresponding to at least one language in the language family category;

[0228] If no corresponding language rule information is matched, a language classification model corresponding to the language family category is determined;

[0229] The language text is classified based on the language classification model to obtain the language of the language text.

[0230] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0231] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0232] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, a recognition module may also be described as a "module for identifying the category of a language".

[0233] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A language recognition method, characterized in that: include: Determine the number of languages ​​included in each of the multiple language categories; Based on the determined number of languages, respectively obtaining initial language classification models corresponding to the multiple language categories, and determining the number of training texts corresponding to the multiple language categories; Obtain a corresponding number of training texts, train the corresponding initial language classification models respectively, and obtain language classification models corresponding to the multiple language categories; Acquire a language text to be recognized, determine a language category corresponding to the language text, the language category includes a language category divided according to a coding rule of a text unit, acquire the encoding of the language text using a preset encoding method, determine a coding interval where the encoding is located, and determine the language category corresponding to the language text based on the coding interval; Matching the language rule information corresponding to the language family category with the language text; the language rule information includes the language rule information corresponding to each language in the language family category, and the language rule information includes the writing order of letters corresponding to each language; If the corresponding language rule information is not matched, a language classification model corresponding to the language family category is determined, and for each language family category, a corresponding language classification model is set, and each language classification model is trained based on the training text corresponding to the language family category, and the language classification model includes a binary classification task model, and the number of languages ​​included in the language family category is positively correlated with the size of the initial language classification model, and the number of languages ​​included in the language family category is positively correlated with the number of training texts; Classifying the language text based on the language classification model to obtain the language of the language text; If the corresponding language rule information is matched, the language is directly determined according to the matched language rule information.

2. The language identification method according to claim 1, characterized in that: The classifying the language text based on the language classification model to obtain the language of the language text includes: Extracting a semantic vector of the language text; The semantic vector is classified based on the language classification model to obtain the language of the language text.

3. The language identification method according to claim 2, characterized in that: The step of extracting the semantic vector of the language text comprises: Splitting the language text into at least one text unit; Obtaining a text unit vector corresponding to each text unit, and determining a position vector corresponding to each text unit; The semantic vector is obtained based on the text unit vector and the position vector corresponding to each text unit.

4. The language identification method according to claim 3, characterized in that: The step of determining the position vector corresponding to each text unit includes: For each text unit, determining the position of the text unit in the language text; A position vector of the text unit is determined based on the position of the text unit in the language text.

5. The language identification method according to claim 3, characterized in that: The acquiring the semantic vector based on the text unit vector and the position vector corresponding to each text unit includes: For each text unit, add the text unit vector and the position vector of the text unit to obtain the text unit representation corresponding to the text unit; The text unit representation of each text unit corresponding to the language text is fused to obtain the semantic vector.

6. A language recognition device, characterized in that: include: A training module, used to determine the number of languages ​​contained in each of the multiple language family categories; Based on the determined number of languages, initial language classification models corresponding to the multiple language categories are respectively obtained, and the number of training texts corresponding to the multiple language categories is determined; a corresponding number of training texts are obtained, and the corresponding initial language classification models are respectively trained to obtain language classification models corresponding to the multiple language categories; A determination module, used to obtain a language text to be identified, determine a language category corresponding to the language text, wherein the language category includes a language category divided according to a coding rule of a text unit, obtain a code of the language text using a preset coding method, determine a coding interval where the code is located, and determine a language category corresponding to the language text based on the coding interval; A matching module, used to match the language rule information corresponding to the language category with the language text; the language rule information includes the language rule information corresponding to each language in the language category, and the language rule information includes the writing order of letters corresponding to each language; An acquisition module, for determining a language classification model corresponding to the language family category if the corresponding language rule information is not matched, wherein a corresponding language classification model is set for each language family category, each language classification model is trained based on the training text corresponding to the language family category, the language classification model includes a binary classification task model, the number of languages ​​included in the language family category is positively correlated with the size of the initial language classification model, and the number of languages ​​included in the language family category is positively correlated with the number of training texts; The recognition module is used to classify the language text based on the language classification model to identify the language of the language text. If the corresponding language rule information is matched, the language is directly determined according to the matched language rule information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the language recognition method described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the language recognition method according to any one of claims 1 to 5.

Images