Method and device for identifying word types
Through mixed language model analysis and language model selection, the problem of low accuracy in identifying new vocabulary and polysemous vocabulary is solved, and the recognition accuracy of multilingual texts is improved.
Patent Information
- Application Number
- CN202010002031.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-01-02
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-01-02
AI Technical Summary
Existing translation models have low accuracy in identifying new and polysemous vocabulary, and cannot effectively process multilingual mixed language texts.
By obtaining texts containing two different languages, using a mixed language model to analyze word segmentation, calculate the probability that each word segmentation belongs to the target language, determine its language model, and recognize it based on the language model to improve the accuracy of text recognition.
It improves the recognition accuracy of new vocabulary and polysense, reduces the dependence on parallel corpus, and enhances the recognition ability of multilingual texts.
Smart Images

Figure CN113065333B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent recognition, and more specifically, to a method and device for identifying a segmented word type. Background Art
[0002] With the development of the Internet and the integration of global data and information, language expression is no longer limited to one language. For example, it can be expressed in one main language and other auxiliary languages (mainly Chinese and supplemented by English terms), or in a close combination of two languages (Hindi and English). In spoken expression, Chinese people are increasingly incorporating English vocabulary, phrases, and whole sentence quotations, which poses new challenges to language modeling. The mixing of multiple languages, also known as code switching or code mixing, has become an important research issue in natural language processing.
[0003] In the related art, a machine translation component that translates between one or more languages is used to generate a process that can accept linguistic input in one or more languages. It relies on an existing translation model of a specific language orientation to translate the input from its language into an anchor language. This method relies on a translation model and requires the use of parallel corpora to train the translation model in advance, and cannot be used in a language orientation without parallel corpora. At the same time, because there are differences in the data of the translation model and the target language modeling, this method is less efficient in processing new vocabulary (vocabulary that the translation model cannot translate) and polysemous vocabulary (a word has multiple meanings in different contexts, such as whether an apple is a fruit or a company).
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a method and device for identifying word types, so as to at least solve the technical problem that the existing translation model has low accuracy in identifying new words and polysemous words.
[0006] According to one aspect of an embodiment of the present application, a method for identifying the language of a segmented word is provided, comprising: obtaining a text to be detected, wherein the text to be detected includes corpora in at least two different languages; inputting multiple segmented words of the text to be detected into a mixed language model for analysis, and obtaining a first scoring indicator for at least one segmented word in the text to be detected, wherein the first scoring indicator is used to calculate the probability that the at least one segmented word belongs to the target language.
[0007] Based on the first scoring index, it is judged whether the probability that at least one segmentation word belongs to the first language is greater than the probability that at least one segmentation word belongs to the second language, wherein the second language includes any language other than the first language; the language to which the specified segmentation word in the to-be-detected text belongs is determined, wherein the specified segmentation word includes a segmentation word in the to-be-detected text that is different from the at least one segmentation word; the language model of the at least one segmentation word is determined according to the judgment result and the language to which the specified segmentation word belongs; based on the determined language model, a second scoring index of the at least one segmentation word is determined, wherein the second scoring index is used to indicate the probability that the at least one segmentation word belongs to the target language; and the language to which the at least one segmentation word belongs is determined based on the second scoring index.
[0008] According to another aspect of an embodiment of the present application, a method for identifying the language of a segmented word is provided, comprising: obtaining a text to be detected, wherein the text to be detected includes corpora in at least two different languages; inputting multiple segmented words of the text to be detected into a mixed language model for analysis to obtain a first scoring indicator for at least one segmented word in the text to be detected, wherein the first scoring indicator is used to indicate the probability that the at least one segmented word belongs to the target language; determining a language model of the at least one segmented word based on the first scoring indicator; and identifying the at least one segmented word based on the determined language model.
[0009] According to another aspect of an embodiment of the present application, a device for identifying the language of a segmented word is also provided, comprising: an acquisition module, used to acquire a text to be detected, wherein the text to be detected includes corpora of at least two different languages; an analysis module, used to input multiple segmented words of the text to be detected into a hybrid language model for analysis, and obtain a first scoring indicator of at least one segmented word in the text to be detected, wherein the first scoring indicator is used to indicate the probability that the at least one segmented word belongs to the target language; a judgment module, used to judge whether the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language based on the first scoring indicator; a first determination module, used to determine the language to which the previous segmented word of at least one segmented word in the text to be detected belongs; a second determination module, used to determine the language model of at least one segmented word based on the judgment result and the language to which the previous segmented word belongs; and a third determination module, used to determine the second scoring indicator of at least one segmented word based on the determined language model.
[0010] According to another aspect of an embodiment of the present application, a method for determining a segmentation language is also provided, comprising: displaying a first control for triggering uploading a text to be detected in a display interface, wherein the text to be detected includes at least two corpora in different languages; outputting a first scoring indicator of at least one segmentation in the text to be detected obtained after inputting multiple segmentations of the text to be detected into a hybrid language model for analysis, wherein the first scoring indicator is used to calculate the probability that the at least one segmentation belongs to a target language; outputting at least one of the following: a probability that at least one segmentation belongs to a different language determined based on the first scoring indicator; a language to which a specified segmentation in the text to be detected belongs, wherein the specified segmentation includes a segmentation in the text to be detected that is different from the at least one segmentation; wherein the probability and the language to which the specified segmentation belongs are used to determine a language model corresponding to the at least one segmentation; outputting a second scoring indicator of at least one segmentation determined based on the language model, and outputting the language to which the at least one segmentation determined based on the second scoring indicator belongs, wherein the second scoring indicator is used to indicate the probability that the at least one segmentation belongs to the target language.
[0011] According to another aspect of an embodiment of the present application, a method for identifying the language of a segmented word is also provided, comprising: displaying a third control for uploading a text to be detected in a display interface of a user terminal device, wherein the text to be detected includes at least two corpora in different languages; displaying multiple segmented words of the text to be detected output by a background device in the display interface; displaying a first scoring indicator obtained after analyzing at least one segmented word according to a hybrid language model in the display interface, wherein the first scoring indicator is used to calculate the probability that the at least one segmented word belongs to a target language; displaying the probabilities of different languages determined based on the first scoring indicator and the language to which a specified segmented word in the text to be detected belongs in the display interface, wherein the specified segmented word includes a segmented word in the text to be detected that is different from the at least one segmented word; wherein the probability and the language to which the specified segmented word belongs are used to determine the language model corresponding to the at least one segmented word; displaying a second scoring indicator of at least one segmented word determined based on the language model and the language to which the at least one segmented word belongs determined based on the second scoring indicator in the display interface, wherein the second scoring indicator is used to indicate the probability that the at least one segmented word belongs to the target language.
[0012] According to another aspect of an embodiment of the present application, a method for identifying the language of a segmented word is provided, comprising: obtaining a text to be detected, wherein the text to be detected includes corpora of at least two different languages; obtaining the probability of occurrence of each language in the text to be detected; determining a language model corresponding to at least one segmented word in the text to be detected based on the probability; and evaluating at least one segmented word based on the language model to obtain a scoring index, wherein the scoring index is used to indicate the probability that the at least one segmented word belongs to the target language.
[0013] According to another aspect of an embodiment of the present application, a model training method is provided, including: acquiring monolingual corpora in multiple languages; determining a target monolingual corpus in which a proportion of specified characters in the monolingual corpora in the multiple languages is greater than a preset threshold, wherein the specified character proportion is a proportion of characters in a second language different from the first language in the monolingual corpus, wherein the proportion of the corpus in the first language in the monolingual corpus is greater than the proportion of characters in the second language in the monolingual corpus; performing preprocessing based on the target monolingual corpus and the monolingual corpora in multiple languages to obtain a target mixed corpus; training a monolingual language model using the monolingual corpora in the target mixed corpus to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language; training using the mixed corpus in the target mixed corpus to obtain a mixed language model, wherein the mixed corpus includes the corpus in the first language and the corpus in the second language.
[0014] In an embodiment of the present application, a text to be detected is obtained, wherein the text to be detected includes corpora of at least two different languages; multiple segmentations of the text to be detected are input into a hybrid language model for analysis to obtain a first scoring index of at least one segmentation in the text to be detected, wherein the first scoring index is used to indicate the probability that the at least one segmentation belongs to the target language; based on the first scoring index, it is determined whether the probability that the at least one segmentation belongs to the first language is greater than the probability that the at least one segmentation belongs to the second language; the language to which a specified segmentation of at least one segmentation in the text to be detected belongs is determined; the language model of at least one segmentation is determined based on the judgment result and the language to which the specified segmentation belongs; and the second scoring index of at least one segmentation is determined based on the determined language model. By inputting multiple segmentations of the text to be detected into a hybrid language model for analysis, the first scoring index of at least one segmentation in the text to be detected is obtained, and the language model corresponding to each segmentation is determined according to the first scoring index, thereby achieving the technical effect of improving the recognition accuracy of the text based on the selection of the hybrid language model and the language model corresponding to the at least one segmentation, thereby solving the technical problem that the existing translation model has low accuracy in recognizing new words and polysemous words. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0016] Figure 1 is a hardware structure block diagram of a computing device according to an embodiment of the present application;
[0017] Figure 2a is a flow chart of an optional method for identifying a segmentation genre according to an embodiment of the present application;
[0018] Figure 2b This is a schematic diagram of an application scenario of an optional method for identifying a word segmentation genre according to an embodiment of the present application;
[0019] Figure 3 is a flow chart of an optional method for identifying a segmentation genre according to the first embodiment of the present application;
[0020] Figure 4 is a flow chart of an optional method for identifying a segmentation genre according to an embodiment of the present application;
[0021] Figure 5 is a schematic structural diagram of an optional device for identifying a word type according to an embodiment of the present application;
[0022] Figure 6 is a schematic diagram of the structure of an optional computer device according to an embodiment of the present application;
[0023] Figure 7 is a flow chart of a method for determining a segmentation genre according to an embodiment of the present application;
[0024] Figure 8 is a flow chart of another method for identifying a segmentation genre according to an embodiment of the present application;
[0025] Fig. 9 It is a flowchart of a model training method according to an embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:
[0029] Language Model (LM): A language model is a machine learning model used to model the probability distribution of a continuous sequence.
[0030] Code-Switch (CS): The conversion of text from one language form to another.
[0031] Code-Mix (CM): The mixing of two or even multiple language variants and codes.
[0032] Byte-pair Encoding (BPE): Using the BPE algorithm to jointly encode bilingual vocabulary can obtain a smaller vocabulary.
[0033] Example 1
[0034] According to an embodiment of the present application, an embodiment of a method for identifying a word type is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0035] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computing device or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device (or mobile device) for implementing a method for identifying a segmented word type. Figure 1 As shown, the computing device 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0036] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computing device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0037] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for identifying the word segmentation type in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for identifying the word segmentation type of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computing device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0038] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of computing device 10 (or mobile device).
[0040] Under the above operating environment, this application provides Figure 2a The identification method of the segmented word type is shown. Figure 2a : is a flowchart of a method for identifying a word segmentation type according to the first embodiment of the present application, the method comprising the following processing flow:
[0041] Step S202, obtaining a text to be detected, wherein the text to be detected includes corpora in at least two different languages.
[0042] Optionally, the text to be detected is a mixed text T=(t_1, t_2, ... t_n) predicted by a language model: the text T may be composed of n BPE segments. The mixed text is a text including corpora of at least two different languages.
[0043] Step S204: input multiple word segments of the text to be detected into the hybrid language model for analysis to obtain a first scoring index of at least one word segment in the text to be detected, wherein the first scoring index is used to calculate the probability that the at least one word segment belongs to the target language.
[0044] Optionally, the hybrid language model can be Figure 3 M_ab in step S314: Use M_ab to perform language model score P(t_i|t_i-1) on the t-th BPE segment in the text, where the score P(t_i|t_i-1) is the first evaluation indicator.
[0045] Step S206, based on the first scoring index, determines whether the probability that at least one segmented word belongs to the first language is greater than the probability that at least one segmented word belongs to the second language, wherein the second language includes any language other than the first language. In one embodiment, the text to be detected may include multiple languages, for example: "I have a joli and lovely doll" includes three languages, wherein "I have a" is in Chinese, "joli" is in French, and "lovely doll" is in English. That is to say, when the first language is Chinese, the second language may be French or English.
[0046] For example: Figure 3 In step S316, it is determined whether the probability of language a in the language model probability is greater than that of language b; determining whether the probability of language a in the language model probability is greater than that of language b can be: determining whether the probability of each word belonging to language a is greater than the probability of belonging to language b.
[0047] Step S208, determining the language to which the specified participle in the text to be detected belongs, wherein the above-mentioned specified participle includes but is not limited to a participle in the file to be detected that is different from the above-mentioned at least one participle, for example, the specified participle includes: the previous participle of the at least one participle in the text to be detected, wherein the previous participle refers to a participle that is located before the current participle (that is, the above-mentioned at least one participle) among multiple participles arranged in a preset arrangement order in the text to be detected; for another example, the above-mentioned specified participle may also include: the Nth participle in front of the at least one participle, where N is greater than 1.
[0048] Step S210, determining a language model of at least one word segmentation according to the judgment result and the language to which the previous word segmentation belongs.
[0049] Step S212, determining a second scoring indicator for at least one word segment based on the determined language model, wherein the second scoring indicator is used to indicate the probability that the at least one word segment belongs to the target language; and determining the language to which the at least one word segment belongs based on the second scoring indicator.
[0050] Before inputting multiple word segments of the text to be detected into the mixed language model for analysis, the above method also needs to perform the following process: obtain monolingual corpora in multiple languages; obtain target mixed corpora in multiple languages; use the monolingual corpora in the target mixed corpora to train the monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language; use the target mixed corpus to train the preset mixed language model to obtain a mixed language model, wherein the target mixed corpus includes the corpus of the first language and the corpus of the second language. Specifically, the above process can be expressed as the implementation process shown in steps S2042 to S2048:
[0051] Step S2042, obtaining monolingual corpora in multiple languages.
[0052] Optionally, obtain monolingual corpora D_a and D_b in multiple languages, for example, obtain Chinese and English corpora respectively. Chinese data such as Baidu Encyclopedia, People's Daily News, etc. are used as D_a data. English data such as data in Wikipedia, Associated Press News, movie subtitles, etc. are used as D_b data.
[0053] For example Figure 3 Step S302 in the above example obtains the multilingual monolingual corpora D_a and D_b.<src,tgt=””>< / src,> .
[0054] Step S2044: determining a target monolingual corpus in which the proportion of designated characters in the monolingual corpora of multiple languages is greater than a preset threshold, wherein the designated character proportion is the proportion of characters in a second language different from the first language in the monolingual corpus, wherein the proportion of the corpus in the first language in the monolingual corpus is greater than the proportion of characters in the second language in the monolingual corpus; extracting a multilingual portion of the corpus from the target monolingual corpus, and determining a target mixed corpus based on the multilingual portion of the corpus, wherein the multilingual portion of the corpus includes at least corpora of multiple languages.
[0055] Based on the above process, it can be seen that the acquired monolingual corpus may contain other languages. The first language may be Chinese, and the monolingual corpus corresponding to Chinese may contain other languages. When the proportion of characters of the second language in the monolingual corpus is greater than a preset threshold, the corpus containing the first language and the second language is confirmed to be the target monolingual corpus.
[0056] For example Figure 3 In step S304, a multilingual mixed corpus D_ab is obtained, and the D_ab is the target monolingual corpus.
[0057] For example, the part of the acquired monolingual corpus that contains multilingual input is filtered according to the character set and proportion. For example, when the preset threshold is 20%, if the proportion of the specified English characters in the monolingual corpus corresponding to Chinese is greater than 20%, the monolingual corpus containing the English characters is used as the target monolingual corpus.
[0058] Step S2046, preprocessing is performed based on the target monolingual corpus and the monolingual corpora of multiple languages to obtain a target mixed corpus; optionally, preprocessing is performed based on the target monolingual corpus and the monolingual corpora of multiple languages to obtain a target mixed corpus can be achieved in the following ways: performing word segmentation processing on the corpus of the multilingual part to obtain multiple word segments; processing the multiple word segments using a byte pair encoding algorithm to obtain high-frequency sub-words; determining a common vocabulary of multiple languages based on the high-frequency sub-words, and using the common vocabulary as the target mixed corpus.
[0059] like Figure 3 In step S306, the corpus is preprocessed to obtain the processed target mixed corpus D_a', D_b', D_ab' and the BPE dictionary Dic, wherein D_a', D_b', D_ab' are the corpus obtained after preprocessing D_a, D_b, and D_ab respectively.
[0060] Step S2048: Using the monolingual corpus in the target mixed corpus to train the monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language; using the mixed corpus in the target mixed corpus to train to obtain the mixed language model, wherein the mixed corpus includes the corpus of the first language and the corpus of the second language.
[0061] Specifically, the first monolingual language model, the second monolingual language model, and the mixed language model can all be ngram or RNNLM language models.
[0062] Optionally, a monolingual corpus in the target mixed corpus is used to train a monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language. Figure 3 In step S308, the monolingual corpus is trained to obtain a monolingual corpus model M_a and a model M_b, wherein the model M_a may be a first monolingual language model, and the model M_b may be a second monolingual language model.
[0063] Optionally, the above D_a' is used to train the model M_a, and D_b' is used to train the model M_b.
[0064] Optionally, a mixed corpus in the target mixed corpus is used for training to obtain a mixed language model such as Figure 3 In step S310, the mixed language model M_ab is obtained by training the mixed corpus.
[0065] Optionally, D_ab' is used to train the model M_ab.
[0066] In some optional embodiments of the present application, determining the language model of at least one word segmentation according to the judgment result and the language to which the previous word segmentation belongs can be implemented in the following manner:
[0067] When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language, and the previous segmented word of the at least one segmented word belongs to the first language, determining to use the first monolingual language model to calculate the second scoring indicator of the at least one segmented word;
[0068] When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language, and the previous segmented word of the at least one segmented word belongs to the second language, determining to use the second monolingual language model and the hybrid language model to calculate the second scoring indicator for the at least one segmented word;
[0069] When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is less than the probability that the at least one segmented word belongs to the second language, and the previous segmented word belongs to the first language, determining to use the first monolingual language model and the hybrid language model to calculate the second scoring indicator for the at least one segmented word;
[0070] When the judgment result indicates that the probability of at least one segmented word belonging to the first language is less than the probability of belonging to the second language, and the previous segmented word belongs to the second language, it is determined to use the second monolingual language model to calculate the second scoring indicator of the at least one segmented word.
[0071] Optionally, determining to use the second monolingual language model and the mixed language model to calculate the second scoring index for at least one word segmentation can be achieved in the following manner: respectively using the second monolingual language model and the mixed language model to calculate an initial scoring index for at least one word segmentation; and performing a weighted operation on the initial scoring indexes calculated using the second monolingual language model and the mixed language model to obtain a second scoring index.
[0072] Alternatively, if Figure 3 As shown in steps S316, S318, and S320, the first language can be language a, and the second language can be language b. When the probability of language a in the language model probability is greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If so, M_a is used as the language model score.
[0073] Alternatively, if Figure 3 As shown in steps S316, S318 and S322, the first language may be language a and the second language may be language b. When the probability of language a in the language model probability is greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If not, the weighted score of M_b and M_ab is used as the language model score for the word.
[0074] Alternatively, if Figure 3 As shown in steps S316, S324 and S326, the first language may be language a and the second language may be language b. When the probability of language a in the language model probability is not greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If so, the weighted score of Ma_a and Ma_ab is used as the language model score for the word.
[0075] Alternatively, if Figure 3 As shown in steps S316, S324 and S328, the first language can be language a and the second language can be language b. When the probability of language a in the language model probability is not greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If not, M_b is used as the language model for scoring.
[0076] Optionally, before inputting multiple word segments of the text to be detected into the hybrid language model for analysis, the above method needs to perform the following steps: using the BPE algorithm to perform word segmentation on the text to be detected to obtain subwords, and using the subwords as word segments input into the hybrid language model.
[0077] The solution of the present application reduces the dependence of the training translation model on data by training the language models of monolingual and mixed languages respectively. At the same time, the language model vocabulary obtained can cover the vocabulary not covered in the parallel corpus, and the language model prediction for this part is more accurate. Based on easily accessible monolingual corpora, a mixed corpus for training a multilingual language model is constructed in a flexible manner. The vocabulary constructed using the BPE algorithm better covers the vocabulary of the two languages, reduces the OOV (out-of-vocabulary) situation, and can obtain subwords with semantic similarity for similar languages. Scoring of the multilingual mixed language model is used as a benchmark to select a suitable mixing method to obtain a more accurate language model score. Thereby achieving the technical effect of improving the recognition accuracy of the text based on the selection of a language model corresponding to a mixed language model and at least one word segmentation.
[0078] In order to better understand the above embodiment, an application scenario is described in detail below:
[0079] like Figure 2bAs shown, the user terminal provides an application for translation, which has Figure 2b The translation interface shown in the figure has a text input box and a translation result box for displaying the translation result. After inputting "I have a joli and lovely doll" in the text input box, Figure 2b The middle step S1 determines the language to which each word segment belongs, wherein step S1 can be performed by Figure 2a The implementation of the scheme shown is not described here; then the language translation model corresponding to the language is determined through step S2, and each word segment is translated based on the determined language translation model, and the translation result is output; then the output translation result is displayed in the translation result box. Among them, steps S1 and S2 can be executed by the background program of the user terminal, or by the server communicating with the user terminal.
[0080] Figure 3 1 is a flow chart of an optional method for identifying a segmentation genre according to Embodiment 1 of the present application, the method comprising the following steps:
[0081] Step S302: Obtaining multilingual monolingual corpora D_a and D_b<src,tgt=””>< / src,> ;
[0082] Step S304, obtaining a multi-language mixed corpus D_ab;
[0083] Step S306, preprocessing the corpus to obtain the processed mixed corpus D_a', D_b', D_ab' and the BPE dictionary Dic;
[0084] Step S308, training the monolingual corpus to obtain the monolingual corpus model M_a and the model M_b;
[0085] Step S310, training the mixed corpus to obtain a mixed language model M_ab;
[0086] Step S312, obtaining a mixed text T=(t_1, t_2, ... t_n) that needs to be predicted by a language model;
[0087] Step S314, use M_ab to perform language model scoring P(t_i|t_i-1) on the t-th BPE segment in the text;
[0088] Step S316, determining whether the probability of language a in the language model probability is greater than that of language b, if so, executing step S318, if not, executing step S324;
[0089] Step S318, determine whether the previous word is language a, if so, execute the following step S320, if not, execute the following step S322;
[0090] Step S320, using Ma as the language model for scoring;
[0091] Step S322, using the weighted score of M_b and M_ab as the language model score of the word;
[0092] Step S324, determine whether the previous word is language a, if so, execute the following step S326, if not, execute the following step S328;
[0093] Step S326, using the weighted score of Ma_a and Ma_ab as the language model score of the word;
[0094] Step S328, use M_b as the language model for scoring.
[0095] In an embodiment of the present application, a text to be detected is obtained, wherein the text to be detected includes at least two corpora of different languages; multiple segmentations of the text to be detected are input into a hybrid language model for analysis to obtain a first scoring index of at least one segmentation in the text to be detected, wherein the first scoring index is used to indicate the probability that the at least one segmentation belongs to the target language; based on the first scoring index, it is determined whether the probability that the at least one segmentation belongs to the first language is greater than the probability that the at least one segmentation belongs to the second language; the language to which the previous segmentation of at least one segmentation in the text to be detected belongs is determined; the language model of at least one segmentation is determined based on the judgment result and the language to which the previous segmentation belongs; and the second scoring index of at least one segmentation is determined based on the determined language model. By inputting multiple segmentations of the text to be detected into a hybrid language model for analysis, the first scoring index of at least one segmentation in the text to be detected is obtained, and the language model corresponding to each segmentation is determined according to the first scoring index, thereby achieving the technical effect of improving the recognition accuracy of the text based on the selection of the hybrid language model and the language model corresponding to the at least one segmentation, thereby solving the technical problem that the existing translation model has low accuracy in recognizing new words and polysemous words.
[0096] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.
[0098] Example 2
[0099] According to an embodiment of the present application, a method for identifying a word type is also provided, such as Figure 4 As shown, Figure 4 : is a flowchart of a method for identifying a word segmentation type according to Embodiment 2 of the present application, the method comprising the following processing flow:
[0100] Step S402, obtaining a text to be detected, wherein the text to be detected includes corpora in at least two different languages.
[0101] Optionally, the text to be detected is a mixed text T=(t_1, t_2, ... t_n) predicted by a language model: the text T may be composed of n BPE segments. The mixed text is a text including corpora of at least two different languages.
[0102] Step S404: input multiple word segments of the text to be detected into the hybrid language model for analysis to obtain a first scoring indicator for at least one word segment in the text to be detected, wherein the first scoring indicator is used to indicate the probability that the at least one word segment belongs to the target language.
[0103] Optionally, the hybrid language model can be Figure 3 M_ab in step S314, step S314 is: use M_ab to perform language model score P(t_i|t_i-1) on the t-th BPE segment in the text, where the score P(t_i|t_i-1) is the first evaluation indicator.
[0104] Step S406: Determine at least one word segmentation language model based on the first scoring indicator.
[0105] Determining the language model of at least one segmentation based on the first scoring indicator can be achieved in the following manner: judging whether the probability that at least one segmentation belongs to the first language is greater than the probability that at least one segmentation belongs to the second language based on the first scoring indicator; determining the language to which the previous segmentation of at least one segmentation in the text to be detected belongs; and determining the language model of at least one segmentation based on the judgment result and the language to which the previous segmentation belongs.
[0106] For example: Figure 3 In step S316, it is determined whether the probability of language a in the language model probability is greater than that of language b; determining whether the probability of language a in the language model probability is greater than that of language b can be: determining whether the probability of each word belonging to language a is greater than the probability of belonging to language b.
[0107] Before inputting multiple word segments of the text to be detected into the hybrid language model for analysis, the method further needs to perform the following steps S2042 to S2048:
[0108] Step S2042, obtaining monolingual corpora in multiple languages.
[0109] Optionally, obtain monolingual corpora D_a and D_b in multiple languages, for example, obtain Chinese and English corpora respectively. Chinese data such as Baidu Encyclopedia, People's Daily News, etc. are used as D_a data. English data such as data in Wikipedia, Associated Press News, movie subtitles, etc. are used as D_b data.
[0110] For example Figure 3 Step S302 in the above example obtains the multilingual monolingual corpora D_a and D_b.<src,tgt=””>< / src,> .
[0111] Step S2044: determining a target monolingual corpus in which the proportion of designated characters in the monolingual corpus of multiple languages is greater than a preset threshold, wherein the designated character proportion is the proportion of characters in a second language different from the first language in the monolingual corpus, wherein the proportion of the corpus in the first language in the monolingual corpus is greater than the proportion of the characters in the second language in the monolingual corpus.
[0112] Specifically, the acquired monolingual corpus may contain other languages. The first language may be Chinese, and the monolingual corpus corresponding to Chinese may contain other languages. When the proportion of characters of the second language in the monolingual corpus is greater than a preset threshold, the corpus containing the first language and the second language is confirmed to be the target monolingual corpus.
[0113] For example Figure 3 In step S304, a multilingual mixed corpus D_ab is obtained, and the D_ab is the target monolingual corpus.
[0114] For example, the part of the acquired monolingual corpus that contains multilingual input is filtered according to the character set and proportion. For example, when the preset threshold is 20%, if the proportion of the specified English characters in the monolingual corpus corresponding to Chinese is greater than 20%, the monolingual corpus containing the English characters is used as the target monolingual corpus.
[0115] Step S2046, preprocessing is performed based on the target monolingual corpus and the monolingual corpora of multiple languages to obtain a target mixed corpus; optionally, preprocessing is performed based on the target monolingual corpus and the monolingual corpora of multiple languages to obtain a target mixed corpus can be achieved in the following ways: performing word segmentation processing on the monolingual corpora of multiple languages to obtain multiple word segments; processing the multiple word segments using a byte pair encoding BPE algorithm to obtain high-frequency sub-words; determining a common vocabulary of multiple languages based on the high-frequency sub-words, and using the common vocabulary as the target mixed corpus.
[0116] like Figure 3 In step S306, the corpus is preprocessed to obtain the processed target mixed corpus D_a', D_b', D_ab' and the BPE dictionary Dic, wherein D_a', D_b', D_ab' are the corpus obtained after preprocessing D_a, D_b, and D_ab respectively.
[0117] Step S2048: Using the monolingual corpus in the target mixed corpus to train the monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language; using the mixed corpus in the target mixed corpus to train to obtain the mixed language model, wherein the mixed corpus includes the corpus of the first language and the corpus of the second language.
[0118] Specifically, the first monolingual language model, the second monolingual language model, and the hybrid language model can all be N-Gram language models or recurrent neural network based language models (Neural Network Based Language Model, abbreviated as RNNLM).
[0119] Among them, N-Gram is a language model commonly used in large vocabulary continuous speech recognition. For Chinese, it is called the Chinese Language Model (CLM). The Chinese language model uses the collocation information between adjacent words in the context to achieve automatic conversion to Chinese characters. The Chinese language model uses the collocation information between adjacent words in the context. When it is necessary to convert continuous pinyin, strokes, or numbers representing letters or strokes without spaces into Chinese character strings (i.e. sentences), it can calculate the sentence with the highest probability, thereby achieving automatic conversion to Chinese characters without manual selection by the user, avoiding the problem of repeated codes of many Chinese characters corresponding to the same pinyin (or stroke string, or number string). The model is based on the assumption that the appearance of the Nth word is only related to the previous N-1 words, but not to any other words. The probability of the whole sentence is the product of the probability of each word. These probabilities can be obtained by directly counting the number of times N words appear at the same time from the corpus.
[0120] Among them, a recurrent neural network (RNN) is a type of recursive neural network that takes sequence data as input, recurses in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain. RNNLM is a machine learning model for language recognition based on RNN.
[0121] Optionally, a monolingual corpus in the target mixed corpus is used to train a monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language. Figure 3 In step S308, the monolingual corpus is trained to obtain a monolingual corpus model M_a and a model M_b, wherein the model M_a may be a first monolingual language model, and the model M_b may be a second monolingual language model.
[0122] Optionally, the above D_a' is used to train the model M_a, and D_b' is used to train the model M_b.
[0123] Optionally, a mixed corpus in the target mixed corpus is used for training to obtain a mixed language model such as Figure 3 In step S310, the mixed language model M_ab is obtained by training the mixed corpus.
[0124] Optionally, D_ab' is used to train the model M_ab.
[0125] In some optional embodiments of the present application, since the recognition rule of the language model is from front to back, the above-mentioned designated participle is the participle located in front of the current participle (that is, the above-mentioned at least one participle). At this time, the above-mentioned designated participle includes but is not limited to: the previous participle of the above-mentioned at least one participle, or the Nth participle in front of the above-mentioned at least one participle (N is greater than 1). For the latter, N can take any value of 2, 3, 4, etc.
[0126] In some embodiments of the present application, in order to ensure the accuracy of the recognition result, the previous segmentation of the at least one segmentation may be used as the designated segmentation. In this case, the language model of the segmentation may be determined in the following manner:
[0127] When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language, and the previous segmented word of the at least one segmented word belongs to the first language, determining to use the first monolingual language model to calculate the second scoring indicator of the at least one segmented word;
[0128] When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language, and the previous segmented word of the at least one segmented word belongs to the second language, determining to use the second monolingual language model and the hybrid language model to calculate the second scoring indicator for the at least one segmented word;
[0129] When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is less than the probability that the at least one segmented word belongs to the second language, and the previous segmented word belongs to the first language, determining to use the first monolingual language model and the hybrid language model to calculate the second scoring indicator for the at least one segmented word;
[0130] When the judgment result indicates that the probability of at least one segmented word belonging to the first language is less than the probability of belonging to the second language, and the previous segmented word belongs to the second language, it is determined to use the second monolingual language model to calculate the second scoring indicator of the at least one segmented word.
[0131] Optionally, determining to use the second monolingual language model and the mixed language model to calculate the second scoring index for at least one word segmentation can be achieved in the following manner: respectively using the second monolingual language model and the mixed language model to calculate an initial scoring index for at least one word segmentation; and performing a weighted operation on the initial scoring indexes calculated using the second monolingual language model and the mixed language model to obtain a second scoring index.
[0132] Alternatively, if Figure 3 As shown in steps S316, S318, and S320, the first language can be language a, and the second language can be language b. When the probability of language a in the language model probability is greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If so, M_a is used as the language model score.
[0133] Alternatively, if Figure 3 As shown in steps S316, S318 and S322, the first language may be language a and the second language may be language b. When the probability of language a in the language model probability is greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If not, the weighted score of M_b and M_ab is used as the language model score for the word.
[0134] Alternatively, if Figure 3 As shown in steps S316, S324 and S326, the first language may be language a and the second language may be language b. When the probability of language a in the language model probability is not greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If so, the weighted score of Ma_a and Ma_ab is used as the language model score for the word.
[0135] Alternatively, if Figure 3 As shown in steps S316, S324 and S328, the first language may be language a and the second language may be language b. When the probability of language a in the language model probability is not greater than that of language b, it is determined whether the previous word of the current word belongs to language a. If not, M_b is used as the language model score.
[0136] Optionally, before inputting multiple word segments of the text to be detected into the hybrid language model for analysis, the method further needs to perform the following steps: using the BPE algorithm to perform word segmentation processing on the text to be detected to obtain subwords, and using the subwords as word segments of the input hybrid language model.
[0137] Step S408: Recognize at least one word segment based on the determined language model.
[0138] Example 3
[0139] According to an embodiment of the present application, a device for identifying a segmented word type for implementing the above-mentioned segmented word type identification method is also provided. Figure 5 As shown, the device includes: an acquisition module 52, an analysis module 54, a judgment module 56, a first determination module 58, a second determination module 510, and a third determination module 512; wherein:
[0140] An acquisition module 52 is used to acquire a text to be detected, wherein the text to be detected includes corpora in at least two different languages;
[0141] An analysis module 54 is used to input multiple segmented words of the text to be detected into the hybrid language model for analysis to obtain a first scoring indicator of at least one segmented word in the text to be detected, wherein the first scoring indicator is used to indicate the probability that the at least one segmented word belongs to the target language;
[0142] A judgment module 56, configured to judge, based on the first scoring indicator, whether the probability that at least one segmented word belongs to the first language is greater than the probability that at least one segmented word belongs to the second language, wherein the second language includes any language other than the first language;
[0143] A first determination module 58 is used to determine the language of a specified word in the text to be detected, wherein the specified word includes a word that is different from at least one word in the text to be detected;
[0144] A second determination module 510 is used to determine a language model of at least one word segmentation according to the judgment result and the language to which the previous word segmentation belongs;
[0145] The third determination module 512 is used to determine a second scoring indicator for at least one word segment based on the determined language model, wherein the second scoring indicator is used to indicate the probability that the at least one word segment belongs to the target language; and determine the language to which the at least one word segment belongs based on the second scoring indicator.
[0146] Before inputting multiple word segments of the text to be detected into the mixed language model for analysis, the device is also used to obtain monolingual corpora in multiple languages; determine a target monolingual corpus in which the proportion of specified characters in the monolingual corpora in multiple languages is greater than a preset threshold, wherein the specified character proportion is the proportion of characters in a second language different from the first language in the monolingual corpus, wherein the proportion of the corpus in the first language in the monolingual corpus is greater than the proportion of characters in the second language in the monolingual corpus; perform preprocessing based on the target monolingual corpus and the monolingual corpora in multiple languages to obtain a target mixed corpus; use the monolingual corpora in the target mixed corpus to train the monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language; use the mixed corpus in the target mixed corpus for training to obtain a mixed language model, wherein the mixed corpus includes the corpus in the first language and the corpus in the second language.
[0147] The device is also used to perform word segmentation processing on monolingual corpora of multiple languages to obtain multiple word segments; process the multiple word segments using a byte pair encoding BPE algorithm to obtain high-frequency subwords; determine a common vocabulary of multiple languages based on the high-frequency subwords, and use the common vocabulary as the target mixed corpus.
[0148] The second determination module 510 is used to determine to use the first monolingual language model to calculate the second scoring index of the at least one segmentation when the judgment result indicates that the probability of the at least one segmentation belonging to the first language is greater than the probability of belonging to the second language, and the previous segmentation of the at least one segmentation belongs to the first language; determine to use the second monolingual language model and the hybrid language model to calculate the second scoring index of the at least one segmentation when the judgment result indicates that the probability of the at least one segmentation belonging to the first language is greater than the probability of belonging to the second language, and the previous segmentation of the at least one segmentation belongs to the second language; determine to use the first monolingual language model and the hybrid language model to calculate the second scoring index of the at least one segmentation when the judgment result indicates that the probability of the at least one segmentation belonging to the first language is less than the probability of belonging to the second language, and the previous segmentation belongs to the first language; determine to use the second monolingual language model to calculate the second scoring index of the at least one segmentation when the judgment result indicates that the probability of the at least one segmentation belonging to the first language is less than the probability of belonging to the second language, and the previous segmentation belongs to the second language.
[0149] The second determination module 510 is used to calculate an initial scoring index of at least one word segmentation using the second monolingual language model and the hybrid language model respectively; and perform a weighted operation on the initial scoring indexes calculated using the second monolingual language model and the hybrid language model to obtain a second scoring index.
[0150] Before inputting multiple word segments of the text to be detected into the hybrid language model for analysis, the device is also used to perform word segmentation processing on the text to be detected using the BPE algorithm to obtain subwords, and use the subwords as word segments input into the hybrid language model.
[0151] It should be noted that the above acquisition module 52, analysis module 54, judgment module 56, first determination module 58, second determination module 510, and third determination module 512 are respectively the same as the examples and application scenarios implemented by step S202, step S204, step S206, step S208, step S210, and step S212 in embodiment 1, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the apparatus can be run in the computing device 10 provided in embodiment 1.
[0152] Example 4
[0153] The embodiment of the present application may provide a computer device, which may be any computer device in a computer device group. Optionally, in this embodiment, the computer device may also be replaced by a terminal device such as a mobile terminal.
[0154] Optionally, in this embodiment, the computer device may be located in at least one network device among a plurality of network devices of a computer network.
[0155] In this embodiment, the above-mentioned computer device can execute the program code of the following steps in the method for identifying the language of a segmentation of an application: obtaining a text to be detected, wherein the text to be detected includes at least two corpora of different languages; inputting multiple segmentations of the text to be detected into a hybrid language model for analysis to obtain a first scoring indicator of at least one segmentation in the text to be detected, wherein the first scoring indicator is used to indicate the probability that the at least one segmentation belongs to the target language; judging based on the first scoring indicator whether the probability that the at least one segmentation belongs to the first language is greater than the probability that the at least one segmentation belongs to the second language; determining the language to which the previous segmentation of at least one segmentation in the text to be detected belongs; determining the language model of at least one segmentation based on the judgment result and the language to which the previous segmentation belongs; and determining a second scoring indicator of at least one segmentation based on the determined language model.
[0156] Optionally, Figure 6 is a structural block diagram of a computer device according to an embodiment of the present application. Figure 6 As shown, the computer device 600 may include: one or more (only one is shown in the figure) processors 602 and a memory 604 .
[0157] Among them, the memory 604 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for identifying the word segmentation type in the embodiment of the present application. The processor 602 executes various functional applications and data processing by running the software programs and modules stored in the memory 604, that is, the above-mentioned method for identifying the word segmentation type is realized. The memory 604 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 604 may further include a memory remotely arranged relative to the processor 602, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0158] The processor 602 may call the information and application programs stored in the memory 604 through the transmission module to perform the following steps:
[0159] A text to be detected is obtained, wherein the text to be detected includes corpora in at least two different languages; a plurality of segmented words of the text to be detected are input into a hybrid language model for analysis to obtain a first scoring index of at least one segmented word in the text to be detected, wherein the first scoring index is used to indicate the probability that the at least one segmented word belongs to the target language; based on the first scoring index, it is determined whether the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language; the language to which the previous segmented word of at least one segmented word in the text to be detected belongs is determined; a language model of the at least one segmented word is determined based on the determination result and the language to which the previous segmented word belongs; and a second scoring index of the at least one segmented word is determined based on the determined language model.
[0160] Optionally, the processor 602 may further execute program codes of the following steps: acquiring monolingual corpora of multiple languages; determining a target monolingual corpus in which a proportion of specified characters in the monolingual corpora of multiple languages is greater than a preset threshold, wherein the specified character proportion is a proportion of characters in a second language different from the first language in the monolingual corpus, wherein the proportion of the corpus in the first language in the monolingual corpus is greater than the proportion of characters in the second language in the monolingual corpus; performing preprocessing based on the target monolingual corpus and the monolingual corpora of multiple languages to obtain a target mixed corpus; training a monolingual language model using the monolingual corpora in the target mixed corpus to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language; training using the mixed corpus in the target mixed corpus to obtain a mixed language model, wherein the mixed corpus includes the corpus in the first language and the corpus in the second language.
[0161] Optionally, the processor 602 may also execute the following program code steps: performing word segmentation processing on monolingual corpora in multiple languages to obtain multiple word segments; processing the multiple word segments using a byte pair encoding BPE algorithm to obtain high-frequency sub-words; determining a common vocabulary of multiple languages based on the high-frequency sub-words, and using the common vocabulary as the target mixed corpus.
[0162] Optionally, the processor 602 may further execute the program code of the following steps: when the judgment result indicates that the probability of at least one segmentation word belonging to the first language is greater than the probability of belonging to the second language, and the previous segmentation word of the at least one segmentation word belongs to the first language, determine to use the first monolingual language model to calculate the second scoring index of the at least one segmentation word; when the judgment result indicates that the probability of at least one segmentation word belonging to the first language is greater than the probability of belonging to the second language, and the previous segmentation word of the at least one segmentation word belongs to the second language, determine to use the second monolingual language model and the mixed language model to calculate the second scoring index of the at least one segmentation word; when the judgment result indicates that the probability of at least one segmentation word belonging to the first language is less than the probability of belonging to the second language, and the previous segmentation word belongs to the first language, determine to use the first monolingual language model and the mixed language model to calculate the second scoring index of the at least one segmentation word; when the judgment result indicates that the probability of at least one segmentation word belonging to the first language is less than the probability of belonging to the second language, and the previous segmentation word belongs to the second language, determine to use the second monolingual language model to calculate the second scoring index of the at least one segmentation word.
[0163] Optionally, the processor 602 may further execute program code of the following steps: respectively calculating an initial scoring index of at least one word segmentation using the second monolingual language model and the hybrid language model; and performing a weighted operation on the initial scoring indexes calculated using the second monolingual language model and the hybrid language model to obtain a second scoring index.
[0164] Optionally, the processor 602 may also execute the program code of the following steps: performing word segmentation processing on the text to be detected by using the BPE algorithm to obtain subwords, and using the subwords as word segmentations of an input hybrid language model.
[0165] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the computer device 600 may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 6 The structure of the electronic device is not limited. For example, the computer device 600 may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations are shown.
[0166] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0167] The embodiment of the present application further provides a processor, which is used to run the following program code:
[0168] A text to be detected is obtained, wherein the text to be detected includes corpora in at least two different languages; a plurality of segmented words of the text to be detected are input into a hybrid language model for analysis to obtain a first scoring index of at least one segmented word in the text to be detected, wherein the first scoring index is used to indicate the probability that the at least one segmented word belongs to the target language; based on the first scoring index, it is determined whether the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language; the language to which the previous segmented word of at least one segmented word in the text to be detected belongs is determined; a language model of the at least one segmented word is determined based on the determination result and the language to which the previous segmented word belongs; and a second scoring index of the at least one segmented word is determined based on the determined language model.
[0169] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for identifying the segmentation genre provided in the first embodiment.
[0170] Optionally, in this embodiment, the above storage medium may be located in any one of the computing devices in the computing device group in the computer network, or in any one of the mobile terminals in the mobile terminal group.
[0171] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:
[0172] A text to be detected is obtained, wherein the text to be detected includes corpora in at least two different languages; a plurality of segmented words of the text to be detected are input into a hybrid language model for analysis to obtain a first scoring index of at least one segmented word in the text to be detected, wherein the first scoring index is used to indicate the probability that the at least one segmented word belongs to the target language; based on the first scoring index, it is determined whether the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language; the language to which the previous segmented word of at least one segmented word in the text to be detected belongs is determined; a language model of the at least one segmented word is determined based on the determination result and the language to which the previous segmented word belongs; and a second scoring index of the at least one segmented word is determined based on the determined language model.
[0173] Example 5
[0174] This embodiment provides a method for determining the genre of a segmentation word, such as Figure 7 As shown, the method includes:
[0175] Step S702, displaying a first control for triggering uploading of a text to be detected in the display interface, wherein the text to be detected includes corpora in at least two different languages;
[0176] Step S704, outputting a first scoring index of at least one word in the text to be detected obtained by inputting multiple word segments of the text to be detected into the hybrid language model for analysis, wherein the first scoring index is used to calculate the probability that the at least one word belongs to the target language;
[0177] Step S706, outputting at least one of the following: a probability that at least one segmented word belongs to a different language determined based on the first scoring indicator; a language to which a specified segmented word in the to-be-detected text belongs, wherein the specified segmented word includes a segmented word in the to-be-detected text that is different from the at least one segmented word; wherein the probability and the language to which the specified segmented word belongs are used to determine a language model corresponding to the at least one segmented word;
[0178] Step S708, outputting a second scoring indicator of at least one word segment determined based on the language model, and outputting a language to which at least one word segment determined based on the second scoring indicator belongs, wherein the second scoring indicator is used to indicate a probability that the at least one word segment belongs to the target language.
[0179] In some embodiments of the present application, the output recognition results can be used for subsequent applications. For example, the recognition results can be manually intervened to correct the recognition results with errors; or, the recognition results can be used to annotate data or to train a model as training data; or, the output recognition results can be used to call a corresponding language translation model for translation or to correct errors in the translation application.
[0180] For example, the implementation process of correcting the recognition result can be expressed as the following implementation process: displaying a second control in the display interface; when it is detected that the second control is triggered, displaying an editing interface for inputting a language type; receiving the language type input by the user through the editing interface, and using the language type input by the user to update the language to which at least one word segment belongs.
[0181] For another example, when the recognition result is used to correct the translation errors of the translation application, the implementation process may be as follows: the target translation application calls the above language model to recognize the text to be translated and obtains a first recognition result; the target translation application determines the corresponding language translation model using the language indicated by the first recognition result; and translates the text to be translated using the determined language translation model; or
[0182] The target translation application uses a local machine learning model to identify the language of the text to be translated and obtains a second recognition result; the target translation application calls the above language model to identify the text to be translated and obtains the first recognition result to obtain the above output recognition result; the target translation application determines whether the languages indicated by the first recognition result and the second recognition result are the same; if they are the same, the target translation application calls the corresponding language translation model to translate the text to be translated; if they are not the same, a prompt message is displayed to the user, which is used to prompt the user that an error has occurred in the current language recognition; and guidance information for the next operation is displayed to the user, for example, prompting the user to select the required language to determine the corresponding language translation model.
[0183] The present application embodiment also provides another method for identifying the language of a word segment, which is used to display some parameters involved in the language identification process to the user, such as Figure 8 As shown, the identification method includes:
[0184] Step S802, displaying a third control for uploading the text to be detected in the display interface of the user terminal device, wherein the text to be detected includes corpora in at least two different languages;
[0185] Step S804, displaying multiple segmented words of the text to be detected output by the background device in the display interface;
[0186] Step S806, displaying in the display interface a first scoring indicator obtained after analyzing at least one word segmentation according to the hybrid language model, wherein the first scoring indicator is used to calculate the probability that the at least one word segmentation belongs to the target language;
[0187] Step S808, displaying the probabilities of different languages determined based on the first scoring indicator and the language to which the specified segmentation in the to-be-detected text belongs in the display interface, wherein the specified segmentation includes a segmentation in the to-be-detected text that is different from at least one segmentation; wherein the probabilities and the language to which the specified segmentation belongs are used to determine a language model corresponding to the at least one segmentation;
[0188] Step S810, displaying a second scoring indicator of at least one word segment determined based on the language model and a language to which at least one word segment belongs determined based on the second scoring indicator in a display interface, wherein the second scoring indicator is used to indicate a probability that the at least one word segment belongs to the target language.
[0189] It should be noted that the preferred implementation methods in the embodiments of the present application can refer to the relevant descriptions in Examples 1 to 4 and will not be repeated here.
[0190] Example 6
[0191] The present application also provides a solution for training a language model. Specifically, Fig. 9 As shown, a model training method provided in an embodiment of the present application includes:
[0192] Step S902, obtaining monolingual corpora in multiple languages;
[0193] Step S904, determining a target monolingual corpus in which the proportion of designated characters in the monolingual corpus of multiple languages is greater than a preset threshold, wherein the designated character proportion is the proportion of characters in a second language different from the first language in the monolingual corpus, wherein the proportion of the corpus in the first language in the monolingual corpus is greater than the proportion of the characters in the second language in the monolingual corpus;
[0194] Step S906, preprocessing is performed based on the target monolingual corpus and the monolingual corpora of multiple languages to obtain a target mixed corpus;
[0195] Step S908, using the monolingual corpus in the target mixed corpus to train a monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language;
[0196] Step S910: training with a mixed corpus in the target mixed corpus to obtain a mixed language model, wherein the mixed corpus includes a corpus in a first language and a corpus in a second language.
[0197] It should be noted that the execution order of steps S908 and S910 is optional, that is, step S908 may be executed first, and then step S910; or, step S910 may be executed first, and then step S908; or, steps S908 and S910 may be executed simultaneously.
[0198] In some embodiments of the present application, the above-mentioned preprocessing process can be implemented in the following ways, but is not limited to them: performing word segmentation processing on monolingual corpora in multiple languages to obtain multiple word segments; processing the multiple word segments using a byte pair encoding algorithm to obtain high-frequency sub-words; determining a common vocabulary of multiple languages based on the high-frequency sub-words, and using the common vocabulary as the target mixed corpus.
[0199] It should be noted that the preferred implementation methods in the embodiments of the present application can refer to the relevant descriptions in Examples 1 to 5 and will not be repeated here.
[0200] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0201] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0202] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0206] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for identifying a word type, characterized in that: include: Acquire a text to be detected, wherein the text to be detected includes corpora in at least two different languages; Inputting multiple segmented words of the text to be detected into a hybrid language model for analysis to obtain a first scoring index of at least one segmented word in the text to be detected, wherein the first scoring index is used to calculate the probability that the at least one segmented word belongs to the target language; Determining, based on the first scoring indicator, whether the probability that the at least one word segment belongs to the first language is greater than the probability that the at least one word segment belongs to the second language, wherein the second language includes any language other than the first language; Determining the language to which a specified participle in the to-be-detected text belongs, wherein the specified participle includes a participle in the to-be-detected text that is different from the at least one participle; Determining a language model of the at least one word segmentation according to the judgment result and the language to which the specified word segmentation belongs; Determining a second scoring indicator for the at least one word segment based on the determined language model, wherein the second scoring indicator is used to indicate a probability that the at least one word segment belongs to the target language; and determining the language to which the at least one word segment belongs based on the second scoring indicator; The mixed language model is obtained by training a preset mixed language model with a target mixed corpus, wherein the target mixed corpus includes predictions of multiple languages.
2. The method according to claim 1, characterized in that: The designated participle includes: a previous participle of the at least one participle in the text to be detected.
3. The method according to claim 1, characterized in that Before inputting the multiple word segments of the to-be-detected text into the hybrid language model for analysis, the method further includes: Obtain monolingual corpora in multiple languages; Acquire the target mixed corpus, wherein the target mixed corpus includes the corpus in the first language and the corpus in the second language; The monolingual corpus in the target mixed corpus is used to train a monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language.
4. The method according to claim 3, characterized in that Acquiring the target mixed corpus includes: Determine a target monolingual corpus in which a proportion of specified characters in the monolingual corpora of the multiple languages is greater than a preset threshold, wherein the specified character proportion is a proportion of characters in a second language different from the first language in the monolingual corpora of the multiple languages, wherein a proportion of the characters in the first language in the monolingual corpus is greater than a proportion of the characters in the second language in the monolingual corpus; A multilingual part of the corpus is extracted from the target monolingual corpus, and the target mixed corpus is determined based on the multilingual part of the corpus, wherein the multilingual part of the corpus includes corpora of at least multiple languages.
5. The method according to claim 4, characterized in that Determining the target mixed corpus based on the corpus of the multilingual part includes: The multilingual corpus is segmented to obtain a plurality of segmented words; the plurality of segmented words are processed using a byte pair encoding algorithm to obtain high-frequency sub-words; a common vocabulary of the plurality of languages is determined based on the high-frequency sub-words, and the common vocabulary is used as the target mixed corpus.
6. The method according to claim 3, characterized in that: Determining the language model of the at least one word segmentation according to the judgment result and the language to which the designated word segmentation belongs includes: In a case where the designated participle includes a previous participle of the at least one participle in the to-be-detected text, when the judgment result indicates that the probability that the at least one participle belongs to the first language is greater than the probability that the at least one participle belongs to the second language, and the previous participle of the at least one participle belongs to the first language, determining to use the first monolingual language model to calculate a second scoring indicator for the at least one participle; When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is greater than the probability that the at least one segmented word belongs to the second language, and a previous segmented word of the at least one segmented word belongs to the second language, determining to use the second monolingual language model and the hybrid language model to calculate a second scoring indicator for the at least one segmented word; When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is less than the probability that the at least one segmented word belongs to the second language, and the previous segmented word belongs to the first language, determining to use the first monolingual language model and the hybrid language model to calculate a second scoring indicator for the at least one segmented word; When the judgment result indicates that the probability that the at least one segmented word belongs to the first language is less than the probability that the at least one segmented word belongs to the second language, and the previous segmented word belongs to the second language, it is determined to use the second monolingual language model to calculate the second scoring indicator for the at least one segmented word.
7. The method according to claim 6, characterized in that Determining to use the second monolingual language model and the hybrid language model to calculate a second scoring indicator for the at least one word segmentation includes: The second monolingual language model and the hybrid language model are respectively used to calculate an initial scoring index of the at least one word segment; and the initial scoring indexes calculated using the second monolingual language model and the hybrid language model are weighted to obtain the second scoring index.
8. The method according to claim 1, characterized in that Before inputting the multiple word segments of the to-be-detected text into the hybrid language model for analysis, the method further includes: The text to be detected is segmented using a byte pair encoding algorithm to obtain subwords, and the subwords are used as the segmentations input into the hybrid language model.
9. A method for determining the genre of a word segmentation, characterized in that: include: Displaying a first control for triggering uploading of a text to be detected in a display interface, wherein the text to be detected includes corpora in at least two different languages; Outputting a first scoring index of at least one word segment in the text to be detected obtained after inputting the multiple word segments of the text to be detected into the hybrid language model for analysis, wherein the first scoring index is used to calculate the probability that the at least one word segment belongs to the target language; Output at least one of the following: a probability that the at least one segmented word belongs to a different language determined based on the first scoring index; a language to which a specified segmented word in the to-be-detected text belongs, wherein the specified segmented word includes a segmented word in the to-be-detected text that is different from the at least one segmented word; wherein the probability and the language to which the specified segmented word belongs are used to determine a language model corresponding to the at least one segmented word; Outputting a second scoring indicator for the at least one word segment determined based on the language model, and outputting a language to which the at least one word segment belongs determined based on the second scoring indicator, wherein the second scoring indicator is used to indicate a probability that the at least one word segment belongs to a target language; The mixed language model is obtained by training a preset mixed language model with a target mixed corpus, wherein the target mixed corpus includes predictions of multiple languages.
10. The method according to claim 9, characterized in that The method further comprises: Displaying a second control in the display interface; When detecting that the second control is triggered, displaying an editing interface for inputting a language type; The language type input by the user is received through the editing interface, and the language type input by the user is used to update the language to which the at least one word segment belongs.
11. A method for identifying a word type, characterized in that: include: Displaying a third control for uploading a text to be detected in a display interface of a user terminal device, wherein the text to be detected includes corpora in at least two different languages; Displaying multiple word segments of the text to be detected output by the background device in the display interface; Displaying, in the display interface, a first scoring indicator obtained after analyzing at least one word segmentation according to the hybrid language model, wherein the first scoring indicator is used to calculate the probability that the at least one word segmentation belongs to the target language; The probability of the different languages determined based on the first scoring indicator and the language to which the specified participle in the to-be-detected text belongs are displayed in the display interface, wherein the specified participle includes a participle in the to-be-detected text that is different from the at least one participle; wherein the probability and the language to which the specified participle belongs are used to determine a language model corresponding to the at least one participle; Displaying, in the display interface, a second scoring indicator of the at least one word segment determined based on the language model, and the language to which the at least one word segment belongs determined based on the second scoring indicator, wherein the second scoring indicator is used to indicate a probability that the at least one word segment belongs to the target language; The mixed language model is obtained by training a preset mixed language model with a target mixed corpus, wherein the target mixed corpus includes predictions of multiple languages.
12. A method for identifying a word type, characterized in that: include: Acquire a text to be detected, wherein the text to be detected includes corpora in at least two different languages; Obtaining the probability of each language appearing in the text to be detected; Determine a language model corresponding to at least one word segment in the text to be detected based on the probability; The at least one word segment is evaluated based on the language model to obtain a scoring indicator, wherein the scoring indicator is used to indicate a probability that the at least one word segment belongs to the target language.
13. A method for identifying a word type, characterized in that: include: Acquire a text to be detected, wherein the text to be detected includes corpora in at least two different languages; Inputting multiple segmented words of the text to be detected into a hybrid language model for analysis to obtain a first scoring indicator for at least one segmented word in the text to be detected, wherein the first scoring indicator is used to indicate the probability that the at least one segmented word belongs to the target language; Determine a language model corresponding to the at least one word segmentation based on the first scoring indicator; Identifying the language to which the at least one word segment belongs based on the determined language model; The mixed language model is obtained by training a preset mixed language model with a target mixed corpus, wherein the target mixed corpus includes predictions of multiple languages.
14. A model training method, characterized in that: include: Obtain monolingual corpora in multiple languages; Determine a target monolingual corpus in which a proportion of specified characters in the monolingual corpora of the multiple languages is greater than a preset threshold, wherein the specified character proportion is a proportion of characters in a second language different from the first language in the monolingual corpus, and wherein a proportion of the corpus in the first language in the monolingual corpus is greater than a proportion of the characters in the second language in the monolingual corpus; Preprocessing is performed based on the target monolingual corpus and the monolingual corpora of the multiple languages to obtain a target mixed corpus; Using the monolingual corpus in the target mixed corpus to train a monolingual language model to obtain a first monolingual language model corresponding to the first language and a second monolingual language model corresponding to the second language; using the mixed corpus in the target mixed corpus to train a mixed language model, wherein the mixed corpus includes the corpus in the first language and the corpus in the second language; Preprocessing is performed based on the target monolingual corpus and the monolingual corpora of the multiple languages to obtain a target mixed corpus, comprising: performing word segmentation processing on the monolingual corpora of the multiple languages to obtain a plurality of word segments; processing the plurality of word segments using a byte pair encoding algorithm to obtain high-frequency sub-words; determining a common vocabulary of the multiple languages based on the high-frequency sub-words, and using the common vocabulary as the target mixed corpus.
15. A device for identifying word types, characterized in that: include: An acquisition module, used for acquiring a text to be detected, wherein the text to be detected includes corpora in at least two different languages; An analysis module is used to input multiple word segments of the text to be detected into a hybrid language model for analysis to obtain a first scoring index of at least one word segment in the text to be detected, wherein the first scoring index is used to calculate the probability that the at least one word segment belongs to the target language; the hybrid language model is obtained by training a preset hybrid language model through a target mixed corpus, wherein the target mixed corpus includes predictions of multiple languages; a judgment module, configured to judge, based on the first scoring indicator, whether the probability that the at least one word segment belongs to the first language is greater than the probability that the at least one word segment belongs to the second language, wherein the second language includes any language other than the first language; A first determination module is used to determine the language to which a specified word in the text to be detected belongs, wherein the specified word includes a word in the text to be detected that is different from the at least one word; A second determination module, configured to determine a language model of the at least one word segmentation according to the judgment result and the language to which the previous word segmentation belongs; A third determination module is used to determine a second scoring indicator for the at least one word segmentation based on the determined language model, wherein the second scoring indicator is used to indicate the probability that the at least one word segmentation belongs to the target language; and determine the language to which the at least one word segmentation belongs based on the second scoring indicator.
16. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method for identifying the segmentation type according to any one of claims 1 to 8.
