Method, apparatus, electronic device and medium for generating language model
By updating the candidate lexicon table to generate the target lexicon table for the target language model, the performance degradation problem of large-scale language models when processing texts of different languages is solved, achieving efficient processing of the target language while maintaining strong processing capabilities for the source language.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING YOUZHUJU NETWORK TECH CO LTD
- Filing Date
- 2023-09-18
- Publication Date
- 2026-06-12
Smart Images
Figure CN117236467B_ABST
Abstract
Description
Technical Field
[0001] This disclosure generally relates to the field of artificial intelligence, and more specifically to methods, apparatus, electronic devices, and media for generating language models. Background Technology
[0002] Large-scale language models (LLMs) are language models composed of neural networks with many parameters (typically billions or more). They learn the latent semantics and grammatical structure of input text, enabling them to predict the probability distribution of the next word or paragraph. Training LLMs requires vast amounts of text data. Common sources of training data include large-scale text corpora on the internet, books, news articles, encyclopedias, etc. Through large-scale unsupervised learning on this data, language models can capture the patterns of language, the semantic relationships between words, and the connections between sentences.
[0003] Large-scale language models have wide applications in natural language processing, including machine translation, text generation, question answering systems, and text classification. In recent years, with the continuous development of deep learning technology and the improvement of computing resources, large-scale language models have achieved significant breakthroughs and progress in text generation, dialogue systems, and natural language understanding. These models have achieved remarkable performance on various natural language processing tasks and have been widely applied in practical applications. Summary of the Invention
[0004] Embodiments of this disclosure provide a method, apparatus, electronic device, and medium for generating language models. When extending the lexicon of an existing language model to train another language model on top of it, it is possible to determine whether candidate lexicons meet requirements before adding them to the existing lexicon. To improve the performance of the existing language model for another language, it can be determined whether candidate lexicons include characters of the target language. If candidate lexicons do not include characters of the target language, the candidate lexicon is updated. Then, a new lexicon can be generated based on the existing language model's lexicon and the updated candidate lexicon. In this way, the performance of the language model for the target language can be improved while reducing the negative impact of the language model on the performance of the source language.
[0005] In a first aspect of the embodiments of this disclosure, a method for generating a language model is provided. The method includes obtaining a source lexicon table and a candidate lexicon table for a source language model, wherein the lexicons in the source lexicon table include characters of the source language. The method further includes determining, for each candidate lexicon in the candidate lexicon table, whether the candidate lexicon includes characters of a target language. The method further includes updating the candidate lexicon table in response to determining that the candidate lexicon does not include characters of the target language. Furthermore, the method includes generating a target lexicon table for a target language model based on the source lexicon table and the updated candidate lexicon table.
[0006] In a second aspect of the embodiments of this disclosure, an apparatus for generating a language model is provided. The apparatus includes a lexical acquisition module configured to acquire a source lexical table and a candidate lexical table of a source language model, wherein the lexical units in the source lexical table include characters of the source language. The apparatus further includes a candidate determination module configured to determine, for each candidate lexical unit in the candidate lexical table, whether a candidate lexical unit includes a character of a target language. The apparatus further includes a candidate update module configured to update the candidate lexical table in response to determining that a candidate lexical unit does not include a character of the target language. The apparatus further includes a target generation module configured to generate a target lexical table of a target language model based on the source lexical table and the updated candidate lexical table.
[0007] In a third aspect of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for generating a language model. The method includes obtaining a source lexicon table and a candidate lexicon table of a source language model, wherein the lexicon tables contain characters of the source language. The method further includes determining, for each candidate lexicon in the candidate lexicon table, whether the candidate lexicon contains characters of a target language. The method further includes updating the candidate lexicon table in response to determining that the candidate lexicon does not contain characters of the target language. Furthermore, the method includes generating a target lexicon table of a target language model based on the source lexicon table and the updated candidate lexicon table.
[0008] In a fourth aspect of embodiments of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements a method for generating a language model. The method includes obtaining a source lexicon table and a candidate lexicon table for a source language model, wherein the lexicon tables contain characters of the source language. The method further includes determining, for each candidate lexicon in the candidate lexicon table, whether the candidate lexicon contains characters of a target language. The method further includes updating the candidate lexicon table in response to determining that the candidate lexicon does not contain characters of the target language. Furthermore, the method includes generating a target lexicon table for a target language model based on the source lexicon table and the updated candidate lexicon table.
[0009] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description
[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0011] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;
[0012] Figure 2 A flowchart of a method for generating a language model according to some embodiments of the present disclosure is shown;
[0013] Figure 3 A schematic diagram illustrating an example of an objective to be achieved using a generated target lexicon table according to some embodiments of the present disclosure;
[0014] Figure 4 A schematic diagram illustrating the process of generating a target lexical table according to some embodiments of the present disclosure is shown;
[0015] Figure 5 A schematic diagram illustrating the process of initializing new lexical units according to some embodiments of the present disclosure is shown;
[0016] Figure 6 A schematic diagram illustrating an example of training data configuration according to some embodiments of the present disclosure is shown;
[0017] Figure 7 Examples of pre-trained benchmark tasks and examples of human-AI dialogue utilizing a target language model are shown according to some embodiments of this disclosure.
[0018] Figure 8 A block diagram of an apparatus for generating a language model according to some embodiments of the present disclosure is shown; and
[0019] Figure 9 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation
[0020] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.
[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.
[0023] Large-scale language models (also referred to as language models in this paper) possess powerful language understanding and generation capabilities, making them widely used in the field of natural language processing. For example, when language models are applied to question-answering systems, users can ask questions, and the language model can understand the semantics of the question text, classify and reason about the text, and generate answers to the questions. However, language models may face some limitations and challenges when processing texts in different languages.
[0024] The quantity and quality of training data are crucial factors affecting the performance of language models. Different languages have unique grammar, vocabulary, and structural rules. By training a language model using training data for a particular language, the model can learn the characteristics and patterns of that language during training. If training data for another language is lacking, the language model may not be sensitive enough to the rules and structure of that language, leading to performance degradation or errors when processing that language.
[0025] In addition, text preprocessing techniques such as tokenizers significantly impact the performance of language models when processing specific languages. A tokenizer divides raw text into discrete units, also known as tokens, according to predetermined rules. In many languages, there are no explicit delimiters between words, thus requiring tokenizers to segment continuous character sequences into meaningful words. For complex languages or languages with rich vocabularies, tokenizers can segment text at the sub-word level, for example, breaking down long words into common prefixes, suffixes, and roots. Tokenizers can perform this segmentation process using a lexicon (also called a vocabulary). A lexicon is a predefined set of tokens containing all words or sub-words that a language model can recognize and process. For a given language, if the lexicon includes a sufficient number of tokens for that language, the tokenizer can achieve high accuracy when segmenting text in that language. If words or subwords not found in the lexicon appear in the text, the language model may not be able to accurately understand and process these words, resulting in the model's inability to process certain key information, inaccurate understanding of the context, and further inaccurate output.
[0026] Several open-source large-scale language models exist, trained autoregressively on massive text datasets and achieving good results. However, most of their training data comes from English scenarios, resulting in significantly weaker Chinese capabilities compared to their superior English performance. For example, LLaMA, one such model, has a tokenizer with 32,000 tokens, but only less than 1,000 of these contain Chinese characters. Given that there are approximately 3,500 commonly used Chinese characters, LLaMA's tokenizer is likely to segment a single Chinese character into multiple tokens, leading to lower efficiency in processing Chinese text.
[0027] Therefore, according to various embodiments of this disclosure, a scheme for generating a language model is provided. This scheme obtains a source lexicon table and a candidate lexicon table for a source language model. The lexicon tables contain characters from the source language. Then, for each candidate lexicon in the candidate lexicon table, it is determined whether the candidate lexicon contains characters from the target language. If the candidate lexicon does not contain characters from the target language, the candidate lexicon table is updated. Then, a target lexicon table for the target language model is generated based on the source lexicon table and the updated candidate lexicon table. In this way, the performance of the language model for the target language can be improved while reducing the negative impact on the performance of the source language.
[0028] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown. For example... Figure 1As shown, the environment 100 includes a source language model 102. The source language model 102 includes a source tokenizer 104. The source tokenizer 104 includes a source token table 106. The source token table 106 includes tokens 108-1, 108-2, …, 108-N (collectively or individually referred to as tokens 108). The source language model 102 is a language model trained for the source language (also referred to as the first language) scenario, and most of the data in its training data is text in the source language, with only a small portion of text in the target language (also referred to as the second language). In addition, as shown in the figure 110, the number of tokens in the source token table 106 that include target language characters is significantly less than the number of tokens that do not include target language characters. For example, in the LLaMA model described above, the source language is English and the target language can be Chinese. The number of tokens that include Chinese characters (e.g., “坚持”) is significantly less than the number of tokens that do not include Chinese characters (e.g., “He”). It should be noted that the source token table 106 can include tokens in multiple languages, special characters (e.g., commas, periods, etc.), or combinations thereof (e.g., a token includes characters in the source language and special characters, or a token includes characters in the source language and characters in the target language).
[0029] As Figure 1 shown, the environment 100 further includes a candidate token table 112. The candidate token table 112 includes tokens 114-1, 114-2, …, 114-N (collectively or individually referred to as tokens 114). The candidate token table 112 can also include tokens in multiple languages, special characters, or combinations thereof. The candidate token table 112 can be used to expand the source token table 106 to enhance the target language processing ability of the tokenizer. Therefore, the candidate token table 112 usually has some tokens that include target language characters. The tokens 114 in the candidate token table 112 can come from the token tables of existing other tokenizers, professional vocabulary in specific fields involved in the application scenario, frequently occurring words or subwords extracted from external text databases, and so on. As Figure 1 shown, the environment 100 further includes an updated candidate token table 116. The updated candidate token table 116 includes tokens 118-1, 118-2, …, 118-N (collectively or individually referred to as tokens 118). The tokens 118 are the tokens finally used to expand the source token table 106. The update operation can include, for example, filtering (e.g., removing tokens 114 that do not meet the conditions), adding (e.g., adding new tokens related to the tokens 114), modifying (e.g., deleting some characters in the tokens 114, or converting the tokens 114 from one language to another).
[0030] As Figure 1As shown, environment 100 also includes a target language model 120, which includes a target word segmenter 122. The target word segmenter 122 includes a target lexical table 124, which includes a word set 126 and a word set 128. The target language model 120 can be a language model generated by further pre-training based on the source language model 102, and the target language model 120 can have better processing capabilities for the target language compared to the source language model 102. The word set 126 in the target lexical table 124 corresponds to the source lexical table 106, that is, the word set 126 includes word 108 from the source lexical table 106. The word set 128 in the target lexical table 124 corresponds to the updated candidate lexical table 116, that is, the word set 128 includes word 118 from the updated candidate lexical table 116.
[0031] In some embodiments, the updated candidate lexical table 116 can be merged with the source lexical table 106 to generate the target lexical table 124. In other embodiments, the candidate lexical table 112 can be merged with the source lexical table 106, and then a portion of the candidate lexical table 112 can be updated to generate the target lexical table 124. As shown in Figure 130, the proportion of lexical units including the target language in the target lexical table 124 is increased compared to Figure 110. It should be noted that Figure 130 is only intended to show that the proportion of lexical units including the target language is increased compared to Figure 110, but is not intended to limit the specific proportion of the number of lexical units including the target language and the number of lexical units excluding the target language. In other words, in the target lexical table 124, the number of lexical units including the target language can be less than, equal to, or greater than the number of lexical units excluding the target language.
[0032] It should be understood that, for ease of understanding, this paper uses English as the source language example and Chinese as the target language example; however, it is not intended to limit the specific types of source and target languages. The solutions provided in this disclosure can be applied to any two languages, for example, the source language is Chinese and the target language is English, or the source language is English and the target language is Spanish, etc. It should also be understood that, although this paper uses the LLaMA model as an example of a source language model, it is not intended to limit the source language model to a specific model; the source language model can be any other language model.
[0033] Figure 2 A flowchart of a method 200 for generating a language model according to some embodiments of the present disclosure is shown. Figure 2 As shown in box 202, method 200 obtains the source lexicon table and candidate lexicon table of the source language model. The lexicons in the source lexicon table include characters from the source language. For example, in... Figure 1In the environment 100 shown, the method 200 can obtain the source token table 106 of the source language model 102. The source language model 102 can be, for example, a language model with powerful English processing capabilities but relatively weak Chinese processing capabilities. And the source token table 106 can have, for example, more tokens including English characters and fewer tokens including Chinese characters. The method 200 can also obtain the candidate token table 112. The candidate token table 112 can have, for example, tokens including Chinese characters and tokens not including Chinese characters. Tokens including Chinese characters can be tokens that only include Chinese characters (e.g., "坚持"), or tokens composed of Chinese characters and other characters (e.g., "U盘").
[0034] In block 204, for a candidate token in the candidate token table, the method 200 can determine whether the candidate token includes characters of the target language. For example, in the environment 100 as shown Figure 1 In the environment 100 shown, for the candidate token 114 in the candidate token table 112, the method 200 can determine whether the candidate token 114 includes characters of the target language. For example, when the target language is Chinese, the method 200 can determine that the tokens "坚持" and "U盘" include Chinese characters, and the token "burger" does not include Chinese characters.
[0035] In block 206, in response to determining that the candidate token does not include characters of the target language, the method 200 updates the candidate token table. For example, in the above example, when the method 200 determines that "burger" does not include Chinese characters, it can update the candidate token table 112 to generate the candidate token table 116. For example, the method can remove "burger" from the candidate token table 112, or after removal, convert "burger" to the Chinese character "汉堡" and add it to the candidate token table 112.
[0036] In block 208, based on the source token table and the updated candidate token table, the method 200 generates the target token table of the target language model. For example, in the environment 100 as shown Figure 1 In the environment 100 shown, the method 200 can generate the target token table 124 based on the source token table 106 and the updated candidate token table 116. The token set 126 in the target token table 124 includes all the tokens 118 in the source token table 106, and the token set 128 includes all the tokens 118 in the updated candidate token table 116. In some embodiments, the updated candidate token table 116 can be merged with the source token table 106 to generate the target token table 124. In other embodiments, the candidate token table 112 can be merged with the source token table 106, and then while keeping the tokens 108 in the source token table 106 unchanged, the part of the candidate token table 112 is updated to generate the target token table 124.
[0037] In this way, Method 200 adds tokens including characters of the target language to the target token table, improving the coverage of the target tokenizer for target language texts, thereby enhancing the performance and processing ability of the target language model for the target language. Additionally, since the tokens in the source token table remain unchanged while maintaining the quality of tokens not including characters of the target language by updating the candidate token table, the impact of expanding the token table on the ability of the target tokenizer to process the source language can be reduced, enabling the target tokenizer to maintain the powerful processing ability of the source tokenizer for the source language.
[0038] In some embodiments, when determining that a candidate token does not include characters of the target language and updating the candidate token table, the candidate token can be removed from the candidate token table. In some embodiments, the target token table can be generated by determining the union of the source token table and the updated candidate token table.
[0039] Figure 3 FIG. 300 is a schematic diagram showing an example of a target to be achieved using the generated target token table according to some embodiments of the present disclosure, and in order to achieve Figure 3 the target shown. Figure 4 FIG. 400 is a schematic diagram showing a process of generating a target token table according to some embodiments of the present disclosure. In Figure 3 the example 300 shown, the source language is English and the target language is Chinese. As Figure 3 shown, the content of the original text 302 is "他坚持不懈。He persisted indefatigably.", which is a parallel text composed of an English text ("He persisted indefatigably") and a Chinese text ("他坚持不懈") representing the same meaning. In example 300, the source tokenizer 304 (e.g., Figure 1 the source tokenizer 104 in Figure 3 ) is used to tokenize the original text 302, and the tokenization result 306 can be obtained. In the tokenization result 306, each token is separated by a vertical line. As Figure 1The source token table 106 in [[ ]] does not include the Chinese characters "坚" and "懈". Therefore, "坚" is tokenized into the tokens "0xE5", "0x9D", and "0x9A" in the three source token tables, and "懈" is also tokenized into the tokens "0xE6", "0x87", and "0x88" in the three source token tables.
[0040] Such tokenization methods may reduce the accuracy and efficiency of the language model in understanding and processing the original text 302. Therefore, it is desirable to enable the target tokenizer 308 to tokenize the original text 302 to generate a tokenization result 310 by expanding the target token table of the target tokenizer 308 (e.g., Figure 1 the target token table 124 in [[ ]]). In the tokenization result 310, the Chinese part of the original text 302 is segmented into three tokens, namely "他", "坚持", and "不懈", while the English part of the original text 302 remains the same as the tokenization result 306. In this way, when training the target language model (e.g., Figure 1 the target language model 120 in [[ ]]) based on the tokenization result 310, it can improve the processing ability of the target language model for "他坚持不懈" while maintaining the processing ability of the target language model for "He persisted indefatigably". An example of reducing the processing ability of the target language model due to expanding the token table can be the tokenization result 312. In the tokenization result 312, the tokenization method of the English word "persisted" by the source tokenizer 304 is changed, that is, it is segmented into "per", "sis", and "ted", which will lead to a decline in the processing ability of the target language model for English.
[0041] Based on Figure 3 the example 300 in [[ ]], Figure 4 FIG. shows a schematic diagram of a process 400 for generating a target token table according to some embodiments of the present disclosure. As Figure 4 shown, the source token table 402 (e.g., Figure 1 the source token table 106 in [[ ]]) contains the token "他" including Chinese characters and the tokens "He" and "0xE5" not including Chinese characters. To improve the ability of the source tokenizer to process Chinese, the source token table 402 can be expanded using the candidate token table 404. However, the candidate token table 404 (e.g., Figure 1 the candidate token table 112 in [[ ]]) includes the tokens "per" and "sis". If these two new tokens are included in the generated target token table, it may cause the English word "persisted" to be tokenized into "per", "sis", and "ted". Therefore, as Figure 4As shown, for each token in candidate token table 404, process 400 can determine whether it includes at least one Chinese character. Process 400 can retain those tokens that include Chinese characters and remove the tokens that do not include Chinese characters from candidate token table 404, thereby generating an updated candidate token table 406. In Figure 4 In the example shown, tokens "坚持", "不懈", "O型", and "U盘" all include Chinese characters, so they are retained, while tokens "per" and "sis" are removed from candidate token table 404 because they do not include Chinese characters. After obtaining the updated candidate token table 406, process 400 can merge source token table 402 and the updated candidate token table 406 (i.e., determine their union), thereby generating target token table 408. In this way, when expanding source token table 402, only tokens that include Chinese characters are newly added, so as to improve the processing ability and performance of the target language model for Chinese while reducing the impact of expanding the token table on the processing ability of the target tokenizer for the source language, enabling the target tokenizer to maintain the powerful processing ability of the source tokenizer for the source language.
[0042] When training the target language model, it is necessary to generate corresponding token embeddings for the tokens in the target token table for subsequent algorithms, and optimize these token embeddings through iteration to minimize the loss of the target language model. The target language model can be continuously pre-trained based on the source language model, so as to be able to improve the processing ability for the target language with the least effort while retaining the processing ability of the source language model for the source language. For example, during the training process, the target language model can generate input embeddings and output embeddings corresponding to the input tokens and output tokens. Then, the target language model can use the input embeddings and output embeddings to predict the probability that the next token of the input token is the output token. The target language model can determine the predicted token as the prediction result based on the prediction probabilities of multiple output tokens, and then optimize the target language model by comparing the predicted token with the actual next token of the input token in the training sample.
[0043] Since the target token table used by the target language model is expanded based on the source token table, for the tokens already in the source token table, available token embeddings can be generated using the source language model. However, for new tokens, available token embeddings cannot be generated using the source language model, so it is necessary to train the target language model to optimize the token embeddings generated for these new tokens. In the traditional solution, the embeddings of these new tokens can be randomly initialized, but this method may lead to a slow convergence rate during the training process and a low performance of the model trained on a limited dataset.
[0044] Therefore, in some embodiments, the target lexical units in the updated candidate lexical unit table can be segmented using a source language model to generate a source lexical unit set. Then, based on the source lexical unit set, a source input embedding set can be generated using the source language model. In these embodiments, the target input embedding can be generated based on the source input embedding set, and then the target language model can be generated based on the target input embedding. In some embodiments, a source output embedding set can also be generated based on the source lexical unit set using the source language model. Then, the target output embedding is generated based on the source output embedding set, and the target language model is trained based on the target input embedding and the target output embedding.
[0045] In some embodiments, to generate a target input embedding, the value at a corresponding position of the target input embedding can be determined based on the value at a specific position of each source input embedding in the source input embedding set. In some embodiments, the value at a corresponding position of the target input embedding can be determined based on the value at a specific position of each source input embedding in the source input embedding set. In some embodiments, a weighted average of the values at a specific position of each source input embedding in the source input embedding set can be determined, and the value at a corresponding position of the target lexical embedding can be determined based on this weighted average.
[0046] Figure 5 A schematic diagram of a process 500 for initializing new lexical units according to some embodiments of the present disclosure is shown. For example... Figure 5 As shown, the target input embedding matrix 502 includes target input embeddings corresponding to each word in the target lexicon table. For example, target input embedding 504 is the embedding corresponding to the word "persist" in the target lexicon table. The target output embedding matrix 532 includes output embeddings corresponding to each word in the target lexicon table. For example, target output embedding 534 is the embedding corresponding to the word "unyielding" in the target lexicon table. The words "persist" and "unyielding" are new words in the updated candidate lexicon set, therefore, target input embedding 504 and target output embedding 534 need to be initialized.
[0047] When initializing the target input embedding 504, process 500 can utilize the source language model (e.g., Figure 1 The source word segmenter (e.g., in the source language model 102) of the source language model 102 Figure 1The source tokenizer 104) in it is used to tokenize the token "persist", so as to obtain the token set "0xE5", "0x9D", "0x9A" and "持". Then, process 500 can use the source language model to generate a set of source input embeddings 512 corresponding to the token set. The set of source input embeddings 512 includes a source input embedding 514 corresponding to the token "0xE5", a source input embedding 516 corresponding to the token "0x9D", a source input embedding 518 corresponding to the token "0x9A", and a source input embedding 520 corresponding to the token "持". Then, process 500 can determine the weighted average of the values of the specific positions of each source input embedding in the set of source input embeddings 512, and determine the value of the corresponding position of the target input embedding 504 based on this weighted average. For example, process 500 can determine the weighted average of the values at positions 514, 516, 518, and 520, and then determine this weighted average as the initial value of position 506 of the target input embedding 504. By initializing each position of the target input embedding 504, the initialization of the target input embedding 504 can be completed.
[0048] Similarly, when initializing the target output embedding 534, process 500 can use the source tokenizer of the source language model to tokenize the token "unremitting", so as to obtain the token set "0xE6", "0x87" and "0x88". Then, process 500 can use the source language model to generate a corresponding set of source output embeddings 542. The set of source output embeddings 542 includes a source output embedding 544 corresponding to the token "0xE6", a source output embedding 546 corresponding to the token "0x87", and a source output embedding 546 corresponding to the token "0x88". Then process 500 can determine the weighted average of the values at positions 544, 546, and 548, and then determine this weighted average as the initial value of position 536 of the target output embedding 534. By initializing each position of the target output embedding 504, the initialization of the target output embedding 504 can be completed.
[0049] Process 500 can find the target input embeddings in the target input embedding matrix 502 that correspond to the new lexical units in the target lexical table, and perform the above initialization process for each of these target input embeddings. Similarly, process 500 can find the target output embeddings in the target output embedding matrix 502 that correspond to the new lexical units in the target lexical table, and perform the above initialization process for each of these target output embeddings. Then, based on the initialized target input embedding matrix 502 and target output embedding matrix 532, process 500 can use the target language model 560 to generate the predicted probability that each lexical unit is the next lexical unit of the specified input lexical unit. Then, the target language model 560 can iteratively optimize the input and output embeddings of the new lexical unit by comparing the predicted next lexical unit with the actual next lexical unit.
[0050] In this way, the embedding of new lexical units in the lexical table of the target language model 560 can be initialized using the trained source language model, thereby improving the convergence speed of the target language model 560 during training. Furthermore, the improved convergence speed allows the target language model 560 to achieve better training results with limited training data, thus improving the model's accuracy.
[0051] To further improve the performance of the target language model, the composition of the training dataset can be adjusted. In some embodiments, a training text dataset can be generated, which consists of the following parts: a dataset including multiple source language texts (also called a first text dataset), a dataset including multiple target language texts (also called a second text dataset), and a dataset including multiple parallel language texts (also called a third text dataset). The mixed language texts include source language texts and target language texts with the same meaning. A target language model is generated based on the target lexicon and the training text dataset. In some embodiments, the ratio of the number of lexicons in the source text dataset to the number of lexicons in the target text dataset is equal to a predetermined ratio. In some embodiments, the ratio of the number of lexicons in the parallel text dataset to the number of lexicons in the target text dataset is also equal to this predetermined ratio. In some embodiments, the ratio of the number of lexicons in the source text dataset, the number of lexicons in the parallel text dataset, and the number of lexicons in the target text dataset is 1:1:18. Constructing the training dataset in this way can improve the target language model's ability to process the target language, accelerate the model's convergence speed, and retain the source language model's ability to process the source language.
[0052] Figure 6 A schematic diagram of an example 600 composed of training data according to some embodiments of the present disclosure is shown. Figure 6As shown in Example 600, the source language is English, and the target language is Chinese. To improve the target language model's ability to handle Chinese, a Chinese text dataset can be included in the training dataset. To maintain the source language model's ability to handle English during training, an English text dataset can be included in the training dataset. Parallel Chinese and English texts (i.e., parallel language texts) help the target language model converge quickly in the early stages of training; therefore, a parallel Chinese and English dataset can be included in the training dataset.
[0053] The proportion of each dataset in the training dataset can significantly affect the performance of the target language model. Since the texts in the parallel text dataset are typically shorter, while longer texts are more effective for training, the ratio of the number of tokens in the English-Chinese parallel dataset to the number of tokens in the Chinese dataset can be made equal to a predetermined ratio when constructing the training dataset. For example, in Example 600, the predetermined ratio is 1:18. Furthermore, since the training objective of the target language model is to enhance its Chinese processing capabilities, the ratio of the number of tokens in the English dataset to the number of tokens in the Chinese dataset can also be made equal to this predetermined ratio when constructing the training dataset. For example, in Example 600, the ratio of the number of tokens in the English dataset to the number of tokens in the Chinese dataset is also 1:18.
[0054] Furthermore, the source of the training data also affects the training performance of the model. For example, in Figure 6 In Example 600 shown, the sources and proportions of the Chinese dataset are as follows: content distribution platforms (25% of the total training dataset), novel platforms (5%), encyclopedia platforms (10%), social platforms (5%), Q&A platforms (5%), knowledge creation platforms (15%), existing public datasets (5%), and other Chinese platforms (20%). The English dataset is sourced from existing public datasets (5%), and the bilingual (Chinese and English) dataset is sourced from websites with bilingual (Chinese and English) text (5%).
[0055] By constructing a sophisticated training dataset, we can improve the target language model's ability to process the target language, accelerate the model's convergence speed, and retain the source language model's ability to process the source language.
[0056] In some embodiments, to evaluate the performance of a pre-trained target language model, the target language model can be computed on the input text X = x1, x2, ..., x using the following equation (1) for a pre-trained benchmark task used to evaluate model performance. N The level of confusion PPL on:
[0057]
[0058] Here, θ represents the parameters of the target language model, and a lower perplexity indicates better model performance.
[0059] Figure 7 Example 700 of a pre-trained benchmark task according to some embodiments of this disclosure and example 710 of human-AI dialogue utilizing a target language model are shown. Figure 7 As shown, Example 700 includes an example of a Chinese pre-trained benchmark task, which can be implemented by providing a query, context, options, and the correct option to the language model. Figure 7 The underscores in the text are used to evaluate the language model's performance on Chinese tasks. Example 700 also includes an example of a pre-trained benchmark task for English, which can be evaluated by providing the language model with a query, options, and the correct option (…). Figure 7 The underscores (indicated in Chinese) are used to evaluate the language model's ability to handle English tasks. In Example 710, a question can be asked in Chinese, and the target language model provides a Chinese answer based on that question.
[0060] Figure 8 A block diagram of an apparatus 800 for generating a language model according to some embodiments of the present disclosure is shown. Figure 8 As shown, the device 800 includes a lexical acquisition module 802, configured to acquire a source lexical table and a candidate lexical table of a source language model, wherein the lexical elements in the source lexical table include characters of the source language. The device 800 also includes a candidate determination module 804, configured to determine whether candidate lexical elements in the candidate lexical table include characters of the target language. The device 800 further includes a candidate update module 806, configured to update the candidate lexical table in response to determining that candidate lexical elements do not include characters of the target language. The device 800 also includes a target generation module 808, configured to generate a target lexical table of the target language model based on the source lexical table and the updated candidate lexical table.
[0061] It is understood that by utilizing the apparatus 800 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, by adding lexical units containing characters of the target language to the target lexical table, the coverage of the target word segmenter for target language text is improved, thereby enhancing the performance and processing capabilities of the target language model for the target language. Furthermore, since the quality of lexical units excluding target language characters is maintained by updating the candidate lexical table while keeping the lexical units in the source lexical table unchanged, the impact of expanding the lexical table on the target word segmenter's ability to process the source language is reduced, allowing the target word segmenter to maintain the source word segmenter's strong processing capabilities for the source language.
[0062] Figure 9 A block diagram of an electronic device 900 according to certain embodiments of the present disclosure is shown. Device 900 may be the device or apparatus described in the embodiments of the present disclosure. Figure 9As shown, device 900 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. Various programs and data required for the operation of device 900 can also be stored in RAM 903. CPU / GPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904. Although not shown in... Figure 9 As shown, device 900 may also include a coprocessor.
[0063] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0064] The various methods or processes described above can be executed by CPU / GPU 901. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU / GPU 901, one or more steps or actions in the methods or processes described above can be performed.
[0065] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.
[0066] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0067] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0068] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.
[0069] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0070] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0071] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0072] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
[0073] The following are some example implementations of this disclosure.
[0074] Example 1. A method for generating language models, comprising:
[0075] Obtain the source lexicon table and candidate lexicon table of the source language model, wherein the lexicons in the source lexicon table include characters of the first language;
[0076] For each candidate lexical in the candidate lexical table, determine whether the candidate lexical includes characters from a second language;
[0077] In response to determining that the candidate lexical does not include characters of the second language, the candidate lexical table is updated; and
[0078] Based on the source lexicon and the updated candidate lexicon, a target lexicon is generated for the target language model.
[0079] Example 2. According to the method described in Example 1, updating the candidate lexicon table includes:
[0080] Remove the candidate lexicon from the candidate lexicon table.
[0081] Example 3. The method described in Examples 1-2, wherein generating the target lexicon table includes:
[0082] The target lexicon is generated by determining the union of the source lexicon and the updated candidate lexicon.
[0083] Example 4. The method described in Examples 1-3 further includes:
[0084] By using the source language model to segment the target words in the updated candidate word list, a source word set is generated.
[0085] Based on the source lexical set, the source language model is used to generate a source input embedding set;
[0086] Based on the source input embedding set, generate the target input embedding; and
[0087] The target language model is generated based on the target input embedding.
[0088] Example 5. The method according to Examples 1-4, wherein generating the target input embedding includes:
[0089] The value at the corresponding position of the target input embedding is determined based on the value at a specific position of each source input embedding in the source input embedding set.
[0090] Example 6. The method according to Examples 1-5, wherein generating the value at the specific location of the target input embedding includes:
[0091] Determine the weighted average of the values at the specific positions of each source input embedding in the source input embedding set; and
[0092] Based on the weighted average value, the value of the corresponding position of the target word embedding is determined.
[0093] Example 7. The method described in Examples 1-6 further includes:
[0094] Based on the source lexical set, the source language model is used to generate a source output embedding set;
[0095] Based on the source output embedding set, generate the target output embedding; and
[0096] The target language model is generated based on the target input embedding and the target output embedding.
[0097] Example 8. The method described in Examples 1-7 further includes:
[0098] Generate a text dataset, which includes a first text dataset, a second text dataset, and a third text dataset. The first text dataset includes multiple first language texts, the second text dataset includes multiple second language texts, and the third text dataset includes multiple parallel language texts, which include first language texts and second language texts with the same meaning.
[0099] The target language model is generated based on the target lexicon and the text dataset.
[0100] Example 9. According to the method described in Examples 1-8, wherein the ratio of the number of lexical units in the first text dataset to the number of lexical units in the second text dataset is equal to a predetermined ratio, and the ratio of the number of lexical units in the third text dataset to the number of lexical units in the second text dataset is equal to the predetermined ratio.
[0101] Example 10. According to the method described in Examples 1-9, the ratio of the number of lexical units in the first text dataset, the number of lexical units in the third text dataset, and the number of lexical units in the second text dataset is 1:1:18.
[0102] Example 11. An apparatus for generating a language model, comprising:
[0103] The lexical acquisition module is configured to acquire the source lexical table and the candidate lexical table of the source language model, wherein the lexical units in the source lexical table include characters of the first language;
[0104] The candidate determination module is configured to determine whether the candidate lexicons in the candidate lexicon table include characters of a second language.
[0105] The candidate update module is configured to update the candidate lexicon table in response to determining that the candidate lexicon does not include characters of the second language; and
[0106] The target generation module is configured to generate a target lexicon table for the target language model based on the source lexicon table and the updated candidate lexicon table.
[0107] Example 12. The apparatus according to Example 11, wherein updating the candidate lexicon table includes:
[0108] The lexical removal module is configured to remove the candidate lexical from the candidate lexical table.
[0109] Example 13. The apparatus according to Examples 11-12, wherein generating the target lexicon table includes:
[0110] The union determination module is configured to generate the target lexicon table by determining the union of the source lexicon table and the updated candidate lexicon table.
[0111] Example 14. The apparatus according to Examples 11-13 further includes:
[0112] The target word segmentation module is configured to segment the target words in the updated candidate word table using the source language model to generate a source word set.
[0113] The source embedding generation module is configured to generate a source input embedding set based on the source lexical set and using the source language model;
[0114] The target embedding generation module is configured to generate a target input embedding based on the source input embedding set; and
[0115] The target embedding module is configured to generate the target language model based on the target input embedding.
[0116] Example 15. The apparatus according to Examples 11-14, wherein generating the target input embedding comprises:
[0117] The target embedding value determination module is configured to determine the value at the corresponding position of the target input embedding based on the value at a specific position of each source input embedding in the source input embedding set.
[0118] Example 16. The apparatus according to Examples 11-15, wherein generating the value at the specific location embedded in the target input includes:
[0119] A weighted average determination module is configured to determine a weighted average of the values at a specific position for each source input embedding in the source input embedding set; and
[0120] The weighted average module is configured to determine the value of the corresponding position of the target word embedding based on the weighted average value.
[0121] Example 17. The apparatus according to Examples 11-16 further includes:
[0122] The source output generation module is configured to generate a source output embedding set based on the source lexical set and using the source language model;
[0123] The target output generation module is configured to generate a target output embedding based on the source output embedding set; and
[0124] The input / output module is configured to generate the target language model based on the target input embedding and the target output embedding.
[0125] Example 18. The apparatus according to Examples 11-17 further includes:
[0126] The dataset generation module is configured to generate a text dataset, which includes a first text dataset, a second text dataset, and a third text dataset. The first text dataset includes multiple first language texts, the second text dataset includes multiple second language texts, and the third text dataset includes multiple parallel language texts, which include first language texts and second language texts with the same meaning.
[0127] The dataset usage module is configured to generate the target language model based on the target lexicon and the text dataset.
[0128] Example 19. The apparatus according to Examples 11-18, wherein the ratio of the number of lexical units in the first text dataset to the number of lexical units in the second text dataset is equal to a predetermined ratio, and the ratio of the number of lexical units in the third text dataset to the number of lexical units in the second text dataset is equal to the predetermined ratio.
[0129] Example 20. The apparatus according to Examples 11-19, wherein the ratio of the number of lexical units in the first text dataset, the number of lexical units in the third text dataset, and the number of lexical units in the second text dataset is 1:1:18.
[0130] Example 21. An electronic device comprising:
[0131] Processor; and
[0132] A memory coupled to the processor, the memory having instructions stored therein, the instructions which, when executed by the processor, cause the electronic device to perform actions, the actions including:
[0133] Obtain the source lexicon table and candidate lexicon table of the source language model, wherein the lexicons in the source lexicon table include characters of the first language;
[0134] For each candidate lexical in the candidate lexical table, determine whether the candidate lexical includes characters from a second language;
[0135] In response to determining that the candidate lexical does not include characters of the second language, the candidate lexical table is updated; and
[0136] Based on the source lexicon and the updated candidate lexicon, a target lexicon is generated for the target language model.
[0137] Example 22. The device according to Example 21, wherein updating the candidate lexicon table includes:
[0138] Remove the candidate lexicon from the candidate lexicon table.
[0139] Example 23. The device according to Examples 21-22, wherein generating the target lexicon table includes:
[0140] The target lexicon is generated by determining the union of the source lexicon and the updated candidate lexicon.
[0141] Example 24. The device according to Examples 21-23 further includes:
[0142] By using the source language model to segment the target words in the updated candidate word list, a source word set is generated.
[0143] Based on the source lexical set, the source language model is used to generate a source input embedding set;
[0144] Based on the source input embedding set, generate the target input embedding; and
[0145] The target language model is generated based on the target input embedding.
[0146] Example 25. The device according to Examples 21-24, wherein generating the target input embedding includes:
[0147] The value at the corresponding position of the target input embedding is determined based on the value at a specific position of each source input embedding in the source input embedding set.
[0148] Example 26. The device according to Examples 21-25, wherein generating the value at the specific location of the target input embedding includes:
[0149] Determine the weighted average of the values at the specific positions of each source input embedding in the source input embedding set; and
[0150] Based on the weighted average value, the value of the corresponding position of the target word embedding is determined.
[0151] Example 27. The device according to Examples 21-26 further includes:
[0152] Based on the source lexical set, the source language model is used to generate a source output embedding set;
[0153] Based on the source output embedding set, generate the target output embedding; and
[0154] The target language model is generated based on the target input embedding and the target output embedding.
[0155] Example 28. The device according to Examples 21-27 further includes:
[0156] Generate a text dataset, which includes a first text dataset, a second text dataset, and a third text dataset. The first text dataset includes multiple first language texts, the second text dataset includes multiple second language texts, and the third text dataset includes multiple parallel language texts, which include first language texts and second language texts with the same meaning.
[0157] The target language model is generated based on the target lexicon and the text dataset.
[0158] Example 29. The device according to Examples 21-28, wherein the ratio of the number of lexical units in the first text dataset to the number of lexical units in the second text dataset is equal to a predetermined ratio, and the ratio of the number of lexical units in the third text dataset to the number of lexical units in the second text dataset is equal to the predetermined ratio.
[0159] Example 30. The device according to Examples 21-29, wherein the ratio of the number of lexical units in the first text dataset, the number of lexical units in the third text dataset, and the number of lexical units in the second text dataset is 1:1:18.
[0160] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
Claims
1. A method for generating language models, comprising: Obtain the source lexicon table and the candidate lexicon table of the source language model, wherein the lexicon in the source lexicon table includes characters of the first language, and at least a portion of the lexicon in the candidate lexicon table includes characters of the second language; For each candidate lexical in the candidate lexical table, determine whether the candidate lexical includes characters of the second language; In response to determining that the candidate lexicons do not include characters of the second language, the candidate lexicon table is updated; as well as Based on the source lexicon and the updated candidate lexicon, a target lexicon is generated for the target language model.
2. The method according to claim 1, wherein updating the candidate lexicon table comprises: Remove the candidate lexicon from the candidate lexicon table.
3. The method according to claim 1, wherein generating the target lexical table comprises: The target lexicon is generated by determining the union of the source lexicon and the updated candidate lexicon.
4. The method according to claim 1, further comprising: By using the source language model to segment the target words in the updated candidate word list, a source word set is generated. Based on the source lexical set, the source language model is used to generate a source input embedding set; Based on the source input embedding set, generate the target input embedding; as well as The target language model is generated based on the target input embedding.
5. The method of claim 4, wherein generating the target input embedding comprises: The value at the corresponding position of the target input embedding is determined based on the value at a specific position of each source input embedding in the source input embedding set.
6. The method of claim 5, wherein generating the value at the specific location of the target input embedding comprises: Determine the weighted average of the values at the specific positions of each source input embedding in the source input embedding set; as well as Based on the weighted average value, the value of the corresponding position of the target word embedding is determined.
7. The method according to claim 4, further comprising: Based on the source lexical set, the source language model is used to generate a source output embedding set, wherein the source output embedding in the source output embedding set is the embedding corresponding to the next lexical of the source input lexical; Based on the source output embedding set, a target output embedding is generated, wherein the target output embedding is the embedding corresponding to the next word of the target word; as well as The target language model is generated based on the target input embedding and the target output embedding.
8. The method according to claim 1, further comprising: Generate a text dataset comprising a first text dataset, a second text dataset, and a third text dataset. The first text dataset includes multiple texts in a first language, the second text dataset includes multiple texts in a second language, and the third text dataset includes multiple texts in parallel languages, where each parallel text includes both first-language and second-language texts with the same semantic meaning. The target language model is generated based on the target lexicon and the text dataset.
9. The method according to claim 8, wherein the ratio of the number of lexical units in the first text dataset to the number of lexical units in the second text dataset is equal to a predetermined ratio, and the ratio of the number of lexical units in the third text dataset to the number of lexical units in the second text dataset is equal to the predetermined ratio.
10. The method according to claim 9, wherein the ratio of the number of lexical units in the first text dataset, the number of lexical units in the third text dataset, and the number of lexical units in the second text dataset is 1:1:
18.
11. An apparatus for generating a language model, comprising: The lexical acquisition module is configured to acquire a source lexical table and a candidate lexical table of the source language model, wherein the lexical units in the source lexical table include characters of the first language, and at least a portion of the lexical units in the candidate lexical table include characters of the second language. The candidate determination module is configured to determine whether the candidate lexicons in the candidate lexicon table include characters of the second language. The candidate update module is configured to update the candidate lexicon table in response to determining that the candidate lexicon does not include characters of the second language; as well as The target generation module is configured to generate a target lexicon table for the target language model based on the source lexicon table and the updated candidate lexicon table.
12. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 10.
13. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 10.