Language model construction method and electronic device, computer readable storage medium

Abstract

The present disclosure provides a language model construction method and device, an electronic device and a computer readable storage medium. The language model construction method comprises: obtaining a to-be-processed text; determining the relevance degrees of the to-be-processed text and initial language models of at least one scene; if the relevance degree of the to-be-processed text and a first initial language model meets a preset condition, fusing the language model corresponding to the to-be-processed text and the first initial language model, and replacing the first initial language model with a fused model; and if the relevance degrees of the to-be-processed text and the initial language models of the at least one scene do not meet the preset condition, taking the language model corresponding to the to-be-processed text as an initial language model of a second scene. The present disclosure can reduce the training cost and obtain the language models of multiple scenes in a continuous expansion and optimization manner.

Description

Language model construction methods and electronic devices, computer-readable storage media Technical Field

[0001] This disclosure relates to the field of artificial intelligence (AI) technology, and in particular to a language model construction method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Currently, language models are typically applied to a specific scenario, such as sports news. Therefore, before training a language model, it is usually necessary to collect a large amount of corpus data from relevant scenarios. This not only results in high training costs but also leads to language models that are only suitable for specific scenarios and have poor generalization capabilities. Summary of the Invention

[0003] This disclosure provides a language model construction method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] Firstly, this disclosure provides a language model construction method, which includes:

[0005] Get the text to be processed;

[0006] Determine the degree of relevance between the text to be processed and the initial language model of at least one pre-trained scenario, wherein the initial language model of at least one scenario is trained based on the base text;

[0007] If the correlation between the text to be processed and the first initial language model meets a preset condition, then the language model corresponding to the text to be processed is fused with the first initial language model to obtain a fused model, wherein the first initial language model is one of the initial language models of the at least one scenario; the fused model replaces the first initial language model as a new initial language model for the first scenario corresponding to the first initial language model.

[0008] If the relevance between the text to be processed and the initial language model of the at least one scenario does not meet the preset conditions, then the language model corresponding to the text to be processed is used as the initial language model of the second scenario, where the second scenario refers to the scenario associated with the text to be processed, and the second scenario is different from the at least one scenario.

[0009] Secondly, this disclosure provides a language model building apparatus, which includes:

[0010] The acquisition module is used to acquire the text to be processed;

[0011] A determination module is used to determine the degree of relevance between the text to be processed and the initial language model of at least one pre-trained scenario, wherein the initial language model of at least one scenario is trained based on the base text.

[0012] The fusion module is used to fuse the language model corresponding to the text to be processed with the first initial language model if the correlation between the text to be processed and the first initial language model meets a preset condition, to obtain a fusion model, wherein the first initial language model is one of the initial language models of the at least one scenario; and to replace the first initial language model with the fusion model as a new initial language model for the first scenario corresponding to the first initial language model.

[0013] An update module is used to use the language model corresponding to the text to be processed as the initial language model of the second scenario if the relevance between the text to be processed and the initial language model of the at least one scenario does not meet the preset conditions. The second scenario refers to the scenario associated with the text to be processed, and the second scenario is different from the at least one scenario.

[0014] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the language model construction method described above.

[0015] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described language model construction method.

[0016] Fifthly, this disclosure provides a computer program or computer program product, the computer program product including a computer program stored in a computer-readable storage medium, the computer program implementing the above-described language model construction method when executed by a processor.

[0017] In the embodiments provided in this disclosure, at least one initial language model corresponding to each scene is first trained based on existing basic text. Then, after obtaining the text to be processed, the relevance of the text to be processed to the initial language model of each scene is determined. If the relevance of the text to be processed to the first initial language model meets a preset condition, the language model corresponding to the text to be processed is fused with the first initial language model to obtain a fused model. The fused model replaces the first initial language model as a new initial language model for the first scene corresponding to the first initial language model, and the text to be processed is added to the corpus of the corresponding scene. The first initial language model is one of the initial language models of the at least one scene. If the relevance of the text to be processed to the initial language models of the at least one scene does not meet the preset condition, the language model corresponding to the text to be processed is used as the initial language model of a second scene. The second scene refers to the scene associated with the text to be processed, and the second scene is different from the at least one scene. As can be seen, this embodiment no longer collects corpora for training specific scenarios. Instead, it pre-trains at least one basic language model corresponding to a scenario based on existing corpora, serving as the initial language model. That is, the existing corpora include text corpora from various scenarios, allowing for the training of the corresponding initial language model. After obtaining the text to be processed, if the text to be processed has a high degree of correlation with one of the at least one initial language models, the language model corresponding to the text to be processed is fused with the highly correlated initial language model to optimize the language model for the scenario corresponding to the highly correlated initial language model. If the text to be processed is unrelated to any of the at least one scenario, the scenario represented by the text to be processed is treated as a new scenario independent of the at least one scenario, and the language model of the text to be processed is used as the initial language model for the new scenario. Since it is not necessary to collect a large amount of corpora in advance for each scenario, the computational load is relatively small when training the initial language model for each scenario. Furthermore, after receiving text to be processed that matches an existing scenario, the corresponding language model for that scenario is optimized through model fusion. Compared to the existing method of retraining the language model based on the updated text, the implementation method disclosed herein can reduce training costs and obtain language models for multiple scenarios through continuous expansion and optimization.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 is a flowchart of a language model construction method provided in an embodiment of this disclosure;

[0021] Figure 2 is a block diagram of a language model construction scenario provided in an embodiment of this disclosure;

[0022] Figure 3 is a block diagram of a language model usage scenario provided by an embodiment of this disclosure;

[0023] Figure 4 is a block diagram of a language model construction device provided in an embodiment of this disclosure;

[0024] Figure 5 is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0027] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0029] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0030] This disclosure relates to the field of Natural Language Processing (NLP), specifically to language model training techniques. Language models can be used to process language and text, and for example, can perform at least one of the following processing operations: language recognition, machine translation, question answering, information retrieval, text extraction, etc.

[0031] Language models are typically trained or used to identify patterns in linguistic texts. However, the terminology and frequency of terms in the corpus vary across different implementation scenarios. For example, in a financial context, the corpus contains frequently occurring terms like "bond" and "fund," while in a sports context, the corpus may not even contain such terms. Therefore, conventional methods for training language models include collecting a large amount of text corpus from the target scenario and using this corpus to train a language model applicable to that scenario. However, this approach is costly in terms of collecting large amounts of text corpus, and training a language model based on such corpus is computationally intensive and complex. Furthermore, the resulting language model may only be applicable to the target scenario and exhibit poor generalization.

[0032] In view of this, embodiments of this disclosure provide a language model construction method. This method trains at least one initial language model corresponding to each scenario based on basic text, where the basic text is the existing text. After obtaining the text to be processed, the matching degree between the text to be processed and each scenario in at least one scenario is determined. If it matches any of the at least one scenario, the language model corresponding to the text to be processed is fused with the initial language model corresponding to the corresponding scenario, and the fused language model is used to replace the initial language model corresponding to the corresponding scenario to optimize the language model corresponding to that scenario. If it does not match any of the at least one scenario, the scenario associated with the text to be processed is taken as a new scenario independent of the at least one scenario, and the language model of the text to be processed is taken as the initial language model of the new scenario. In this way, not only is it possible to reduce the training cost by eliminating the need to collect a large amount of text corpus, but the computational load of the training process of each scenario language model is also relatively small. Furthermore, it is possible to continuously expand the language model to add new scenarios and optimize the language model of existing scenarios.

[0033] The language model construction method according to embodiments of this disclosure can be executed by an electronic device, which may be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. The electronic device can be used as a terminal device or a server.

[0034] Figure 1 is a flowchart of a language model construction method provided in an embodiment of this disclosure. Referring to Figure 1, the method includes:

[0035] In step S11, the text to be processed is obtained.

[0036] The text to be processed refers to new text that is different from the existing basic text.

[0037] It should be noted that before step S11, the electronic device can train at least one initial language model corresponding to each scenario based on the base text. The base text described in this embodiment can be obtained through various publicly available and legal channels. For example, the base text may include technical documents, teaching materials, news articles, etc., obtained from relevant websites. The base text can be the text corpus of at least one scenario, such as text from a financial scenario.

[0038] It should be noted that, in this embodiment of the disclosure, the electronic device may pre-maintain a network to be trained. Furthermore, after acquiring the basic text, the electronic device may classify the basic text according to the scenario. Then, for each scenario after classification, the text of the corresponding scenario is preprocessed and segmented, etc., and then the network to be trained is trained based on the segmentation results, and the trained language model is used as the initial language model corresponding to the corresponding scenario.

[0039] For example, the basic text may include text from financial scenarios, text from sports scenarios, and text from entertainment scenarios. The electronic device can divide the basic text into text from financial scenarios, text from sports scenarios, and text from entertainment scenarios according to the scenario. Then, for the text from financial scenarios, preprocessing, word segmentation, and other operations are performed to train an initial language model for the financial scenario; for the text from sports scenarios, preprocessing, word segmentation, and other operations are performed to train an initial language model for the sports scenario; and for the text from entertainment scenarios, preprocessing, word segmentation, and other operations are performed to train an initial language model for the entertainment scenario.

[0040] In step S12, the degree of relevance of the text to be processed to the initial language model of at least one pre-trained scene is determined.

[0041] The degree of correlation between the text to be processed and the initial language model of scenario i can characterize the degree of association between the text to be processed and scenario i, so as to determine whether the text to be processed belongs to the text describing scenario i, where scenario i is any of the above-mentioned at least one scenario.

[0042] In some implementations, after acquiring the text to be processed, the electronic device can segment the text into multiple word segments, each segment corresponding to a phrase length. The phrase length of segment j indicates the number of words constituting segment j, and segment j is any one of the multiple word segments. Then, based on the multiple word segments, a relevance parameter between each initial language model and the text to be processed is determined. Each relevance parameter characterizes the degree of relevance between the corresponding initial language model and the text to be processed.

[0043] For example, the third initial speech model is any one of the initial language models of the at least one scenario; wherein, the specific implementation of determining the relevance parameter between the third initial language model and the text to be processed based on the plurality of word segments is as follows: calling the third initial language model to calculate the relevance parameter between the third initial language model and the text to be processed based on the plurality of word segments, wherein the relevance parameter is used to characterize the degree of relevance between the text to be processed and the scenario corresponding to the third initial language model.

[0044] For example, an electronic device can segment the text to be processed into multiple word segments based on the n-gram mechanism. The n-gram mechanism refers to a sliding window with a step size of n starting from the first character of the text to be processed, sequentially forming words from the first to the nth characters to obtain a word segment, then forming words from the second to the (n-1)th characters to obtain a word segment, and then sliding backwards in this manner until the last character. Here, the step size n is the length of the aforementioned word segment.

[0045] In some implementations, electronic devices can flexibly set the phrase length n according to the actual implementation scenario, and can set multiple n-grams. For example, in one implementation scenario, the phrase length n can be set to any value from 2 to 8.

[0046] It should be noted that, based on the foregoing description, the word combinations, word frequencies, and distributions vary across texts describing different scenarios. Therefore, using a language model corresponding to the first scenario to predict the credibility of text in the second scenario generally yields poor credibility parameters, while using a language model corresponding to the first scenario to predict the credibility of text in the first scenario typically yields higher credibility parameters. The credibility parameters of a text characterize the credibility of the text composed of these word segments, and are usually presented as probabilities. For example, a higher probability indicates higher credibility, and a lower probability indicates lower credibility.

[0047] Based on this, in step S12, taking the example of calling the third initial language model to calculate the relevance parameter between the third initial language model and the text to be processed based on the multiple word segments, the electronic device can call the third initial language model to calculate the credibility parameter of each word segment among the multiple word segments. The credibility parameter is used to characterize the credibility of the corresponding word segment in the scenario corresponding to the third initial language model. Then, the relevance parameter between the third initial language model and the text to be processed is calculated based on the credibility parameter of each word segment.

[0048] For example, the n-gram mechanism can be used to segment the text S to be processed into word segments W1, W2, ... up to word segment W. k That is, S = W1, W2, ... W k Corresponding to the third initial language model, the relevance parameter P(S) between the third initial language model and the text S to be processed can be implemented as follows:

[0049] P(S) = P(W1, W2, ..., W...) k )=p(W1)P(W2|W1)…P(W k |W1, W2, ..., W k-1 )

[0050] Among them, P(W i () refers to the word segment w calculated using the third initial language model. i The credibility parameter of the constituent words. Taking 3-gram as an example:

[0051] P(W i |W i-1 W i-2 )≈count(W i-2 W i-1 W i ) / count(W i-2 W i-1 )

[0052] Among them, count(Wi-2 W i-1 W i ) refers to the word W i-2 W i-1 W i Word frequency in all words composed of each word segment.

[0053] In some implementations, the relevance parameter between the initial language model and the text to be processed includes the perplexity (ppl) parameter between the initial language model and the text to be processed. Taking the text to be processed S as an example, the ppl parameter between the third initial language model and the text to be processed S can be implemented as follows:

[0054]

[0055] It is evident that the larger the relevance parameter P(S) between the third initial language model and the text S to be processed, the higher the probability of obtaining the text S using the third initial language model. Therefore, a smaller ppl parameter between the third initial language model and the text S indicates that the third initial language model is less "confusing" with the text S, and thus, a higher degree of relevance between the third initial language model and the text S. Conversely, a smaller relevance parameter P(S) indicates a lower probability of obtaining the text S using the third initial language model. Therefore, a larger ppl parameter between the third initial language model and the text S indicates that the third initial language model is more "confusing" with the text S, and thus, a lower degree of relevance between the third initial language model and the text S.

[0056] In step S13, if the correlation between the text to be processed and the first initial language model meets the preset conditions, the language model corresponding to the text to be processed is fused with the first initial language model to obtain a fused model. The fused model replaces the first initial language model and serves as the new initial language model for the first scenario corresponding to the first initial language model.

[0057] The first initial language model is one of the initial language models for the at least one scenario. Taking the relevance level as a ppl parameter as an example, the relevance level meeting a preset condition can be, for example, the ppl parameter being less than a preset threshold. The relevance level between the text to be processed and the first initial language model meets the preset condition, that is, the ppl parameter between the text to be processed and the first initial language model is less than a preset threshold.

[0058] In some implementations, after acquiring the text to be processed, the electronic device can also train a language model corresponding to the text to be processed. For example, after segmenting the text to be processed into multiple word segments in step S12, the electronic device can train a network to be trained based on these multiple word segments, and use the trained model as the language model corresponding to the text to be processed.

[0059] According to the foregoing description, if the correlation between the text to be processed and the first initial language model meets the preset conditions, it means that the text to be processed is the text of the scene corresponding to the first initial language model. Then, the electronic device can optimize the first initial language model based on the text to be processed and the language model corresponding to the text to be processed.

[0060] In some implementations, the electronic device can determine the interpolation coefficients based on the correlation between the language model corresponding to the text to be processed and the first initial language model, and then interpolate the language model of the text to be processed and the first initial language model according to the interpolation coefficients to obtain the fusion model.

[0061] For example, when the relevance parameter between the first initial language model and the text to be processed is the ppl parameter, the difference between the preset threshold and the ppl parameter corresponding to the first initial language model can be calculated, and the ratio of the difference to the preset threshold can be used as the interpolation coefficient, that is, interpolation coefficient = (preset threshold - ppl) / preset threshold.

[0062] Furthermore, the electronic device can use the first initial language model as the interpolation model, and interpolate the language model corresponding to the text to be processed into the interpolation model to obtain the interpolated model. For example, corresponding to each phrase length, the electronic device obtains the probability distribution of the language model of the text to be processed for each word segment of the corresponding phrase length as the first probability distribution. Then, corresponding to each phrase length, it obtains the probability distribution of the first initial language model for word segments of the corresponding phrase length in the corresponding scene text as the second probability distribution. The first initial language model corresponds to the classification operation of the basic text from the scene text, as detailed in step S11, which will not be repeated here. Furthermore, corresponding to each phrase length, the electronic device uses the aforementioned interpolation coefficients as algorithm weights to fuse the corresponding first probability distribution and the corresponding second probability distribution using a weighted algorithm. The fused data is used to replace the model parameters of the first initial language model, and the language model with the replaced model parameters is used as the fused model. Therefore, the fused model can be used to replace the first initial language model.

[0063] In other implementations, the text to be processed can be added to the corpus of the scene corresponding to the first initial language model.

[0064] As can be seen, by adopting this implementation method, for each scenario's language model, through small-batch input of corpus, the language model of the relevant scenario is continuously optimized by interpolation. This not only eliminates the need to repeatedly train the model based on historical corpus, thus reducing training costs, but also continuously expands the amount of text corpus.

[0065] In step S14, if the correlation between the text to be processed and the initial language model of at least one scenario does not meet the preset conditions, then the language model corresponding to the text to be processed is used as the initial language model of the second scenario.

[0066] The second scenario refers to the scenario associated with the file to be processed, and the second scenario is different from the at least one scenario.

[0067] If the correlation between the text to be processed and the initial language model of at least one scenario does not meet the preset conditions, it can be understood that the ppl parameters of the initial language model of the text to be processed and the initial language model of at least one scenario are both greater than the preset threshold. That is, the scenario associated with the text to be processed does not match at least one scenario. In this case, the electronic device can take the scenario associated with the text to be processed as a new scenario other than at least one scenario, take the language model of the text to be processed as the initial language model of the new scenario, and take the text to be processed as the corpus of the new scenario.

[0068] By adopting this implementation method, the number of language models adapted to the new scenarios can be continuously expanded by inputting text corpora of new scenarios, thereby enabling the training of language models adapted to multiple scenarios at a low cost.

[0069] The language model construction method provided in this disclosure will be described below with reference to examples.

[0070] Referring to Figure 2, which is a block diagram of a language model construction scenario provided by an embodiment of this disclosure, in this example, the base text is existing corpus and the text to be processed is new text.

[0071] In the preprocessing stage, the electronic device classifies the existing corpus according to the scenario, for example, to obtain sub-texts of three scenarios: A, B, and C. Then, for each sub-text of scenario A, B, and C, preprocessing, word segmentation, and other operations are performed. After that, based on the word segmentation of each scenario sub-text, language models corresponding to each scenario are trained: language model A, language model B, and language model C. The language models A, B, and C are used as the framework model.

[0072] In the language model construction phase, new text is received, and then the electronic device performs preprocessing and word segmentation on the new text. Next, on one hand, a language model corresponding to the new text is trained based on the word segmentation: a new language model. On the other hand, the segmented new text segments are input into language models A, B, and C respectively, so that each language model calculates a ppl value (ppl). For example, language model A calculates ppl.A, language model B calculates ppl.B, and language model C calculates ppl.C. Then, the ppl.A, ppl.B, and ppl.C values ​​are compared with preset ppl thresholds. If all three values ​​are greater than the ppl threshold, it indicates that the new text does not match any of the three scenarios (A, B, and C). In this case, the new text is added as corpus for scenario D to a scenario that this disclosure can adapt to, and the new language model is added as language model D to the framework model.

[0073] In another implementation, for example, if the value of ppl.A is less than the ppl threshold, it means that the new text matches scenario A, that is, the new text is text under scenario A. Then, the interpolation coefficient is calculated as (ppl threshold - ppl.A) / ppl threshold. This interpolation coefficient is used as the weight for interpolation calculation. The language model A is interpolated according to the new language model to obtain the interpolated language model A'. Then, the language model A' is used to replace the language model A as the language model of scenario A.

[0074] It is understood that the above is only an illustrative description using language model A as an example, and does not constitute a limitation on the embodiments of this disclosure. If the ppl of language model B or language model C is less than the ppl threshold, then the new language model is interpolated with the corresponding language model to optimize the language model for the corresponding scenario.

[0075] As can be seen, the language model construction method provided in this embodiment no longer collects training corpora for specific scenarios, but pre-trains at least one basic language model corresponding to a scenario based on existing corpora (basic text) as the initial language model. After obtaining the text to be processed, if the correlation between the text to be processed and the first initial language model in at least one initial language model meets a preset condition, it indicates that the text to be processed matches the scenario corresponding to the first initial language model. By fusing the language model corresponding to the text to be processed and the first initial language model, the language model corresponding to that scenario is optimized. If the text to be processed does not match any of the at least one scenario, the scenario corresponding to the text to be processed is taken as a new scenario independent of the at least one scenario, and the language model of the text to be processed is taken as the initial language model of the new scenario. In this way, not only can the training cost be reduced, but the computational load during the training process of each model is also relatively small. Furthermore, by inputting small batches of corpora, if the initial language model used as the basis includes the language model corresponding to the input corpora, then the language model for the relevant scenarios can be continuously optimized by interpolation, without having to repeatedly train the model based on historical corpora; if the initial language model used as the basis does not include the language model corresponding to the input corpora, then the number of language models for the adapted scenarios can be expanded, thereby training a language model adapted to multiple scenarios.

[0076] It should be noted that, in this embodiment of the disclosure, the electronic device can deploy at least one language model involved in the above method embodiment in a multi-scene language model training system, and can call the language model in the multi-scene language model training system according to the scene during the process of continuously optimizing the language models of each scene in the multi-scene language model training system.

[0077] For example, the language model may be a language recognition model, a machine translation model, a text recognition model, an information retrieval model, a text extraction model, etc.

[0078] For example, how does the obtained language model perform text processing? This includes acquiring the text to be processed, inputting the text into the model for XX operations, and obtaining what from the text.

[0079] For example, speech recognition models can be used for speech recognition tasks, which can be applied to chatbots. Chatbots can interact with users to meet their needs. Therefore, chatbots can be categorized into chatbots, business-oriented chatbots, and so on. To communicate with users, a chatbot first needs to accurately understand the user's intended speech. Typically, electronic devices can convert the user's speech into text, then perform text recognition to obtain the user's intent or keywords, and further generate a response (speech or text) based on the user's intent, which is then output to the user through the chatbot.

[0080] As shown in Figure 3, Figure 3 is a block diagram of a language model application scenario provided by an embodiment of this disclosure. After receiving text corresponding to any scenario in a multi-scenario language model training system, the multi-scenario language model training system can calculate the ppl value of the existing language model and the corresponding text, and then select the language model with the smallest ppl value as the model to be called.

[0081] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0082] In addition, this disclosure also provides a language model building apparatus, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the language model building methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0083] Figure 4 is a block diagram of a language model construction device provided in an embodiment of this disclosure.

[0084] Referring to Figure 4, this embodiment of the present disclosure provides a language model construction apparatus, which includes: an acquisition module 41, a determination module 42, a fusion module 43, and an update module 44. Each module, during operation, can implement some or all of the functions described in the above method implementation.

[0085] For details on the specific implementation method, please refer to the methods illustrated in Figures 1 and 2 above, which will not be repeated here.

[0086] It is understandable that the above division of modules is only a logical functional division. In actual implementation, each of the above modules can be integrated into the hardware implementation. For example, the function of the acquisition module 41 in the above implementation can be integrated into the I / O interface implementation, and the functions of the determination module 42, the fusion module 43 and the update module 44 can be integrated into the processor implementation.

[0087] Referring to FIG5, FIG5 provides an electronic device according to an embodiment of the present disclosure. The electronic device includes: at least one processor 501; at least one memory 502; and one or more I / O interfaces 503 connected between the processor 501 and the memory 502. The memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501 to enable the at least one processor 501 to execute the above-described language model construction method.

[0088] This disclosure also provides a computer-readable storage medium, which may be a volatile or non-volatile computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by the processor 501:

[0089] Obtain the text to be processed; determine the relevance of the text to be processed to the initial language models of at least one pre-trained scenario, wherein the initial language models of the at least one scenario are trained based on the basic text; if the relevance of the text to be processed to the first initial language model meets a preset condition, then the language model corresponding to the text to be processed is fused with the first initial language model to obtain a fused model, wherein the first initial language model is one of the initial language models of the at least one scenario; replace the first initial language model with the fused model as a new initial language model for the first scenario corresponding to the first initial language model; if the relevance of the text to be processed to the initial language models of the at least one scenario does not meet the preset condition, then the language model corresponding to the text to be processed is used as the initial language model for the second scenario, wherein the second scenario refers to the scenario associated with the text to be processed, and the second scenario is different from the at least one scenario.

[0090] In some embodiments, the processor 501 is further configured to segment the text to be processed into multiple word segments, each word segment corresponding to a word group length, the word group length of each word segment being used to indicate the number of words constituting the corresponding word segment; and to determine a relevance parameter between each initial language model and the text to be processed based on the multiple word segments, each relevance parameter being used to characterize the degree of relevance between the corresponding initial language model and the text to be processed; wherein, the specific implementation of determining the relevance parameter between the third initial language model and the text to be processed based on the multiple word segments includes: calling the third initial language model to calculate the relevance parameter between the third initial language model and the text to be processed based on the multiple word segments, the third initial language model being any one of the initial language models of the at least one scenario; the relevance parameter being used to characterize the degree of relevance between the text to be processed and the scenario corresponding to the third initial language model.

[0091] In some embodiments, the processor 501 is further configured to call the third initial language model to calculate the credibility parameter of each of the plurality of word segments, the credibility parameter being used to characterize the credibility of the corresponding word segment as a word segment in the scenario corresponding to the third initial language model; and to calculate the relevance parameter between the third initial language model and the text to be processed based on the credibility parameter of each word segment.

[0092] In some embodiments, the relevance parameter between the initial language model and the text to be processed includes: the perplexity parameter between the initial language model and the text to be processed, wherein the relevance between the text to be processed and the first initial language model meets a preset condition, including: the perplexity parameter between the text to be processed and the first initial language model is less than a preset threshold.

[0093] In some embodiments, the processor 501 is further configured to determine interpolation coefficients based on the correlation between the language model corresponding to the text to be processed and the first initial language model; and to interpolate the language model of the text to be processed and the first initial language model based on the interpolation coefficients to obtain the fusion model.

[0094] In some embodiments, the processor 501 is further configured to calculate the difference between a preset threshold and the perplexity parameter corresponding to the first initial language model; and to use the ratio of the difference to the preset threshold as the interpolation coefficient.

[0095] In some embodiments, the length of the phrase corresponding to each word segment is an integer greater than or equal to 2. The processor 501 is further configured to, for each phrase length, obtain the probability distribution of the language model of the text to be processed for each word segment of the corresponding phrase length as a first probability distribution; for each phrase length, obtain the probability distribution of the first initial language model for word segments of the corresponding phrase length in the corresponding scene text as a second probability distribution, wherein the scene text corresponding to the first initial language model comes from the base text; for each phrase length, use the interpolation coefficient as algorithm weights to fuse the corresponding first probability distribution and the corresponding second probability distribution through a weighted algorithm; use the fused data to replace the model parameters of the first initial language model, and use the language model after replacing the model parameters as the fused model.

[0096] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described language model construction method.

[0097] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0098] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0099] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0100] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0101] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0102] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0104] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A language model construction method, characterized in that, include: Get the text to be processed; The relevance of the text to be processed to at least one pre-trained initial language model for at least one scenario is determined. The initial language models for at least one scenario are trained based on the basic text. If the relevance of the text to be processed to the first initial language model meets a preset condition, the language model corresponding to the text to be processed is fused with the first initial language model to obtain a fused model. The first initial language model is one of the initial language models for the at least one scenario. The fused model replaces the first initial language model as a new initial language model for the first scenario corresponding to the first initial language model. If the relevance of the text to be processed to the initial language models for the at least one scenario does not meet the preset condition, the language model corresponding to the text to be processed is used as the initial language model for a second scenario. The second scenario refers to the scenario associated with the text to be processed, and the second scenario is different from the at least one scenario.

2. The language model construction method according to claim 1, characterized in that, The step of determining the relevance of the text to be processed to the initial language models of at least one pre-trained scenario includes: segmenting the text to be processed into multiple word segments, each word segment corresponding to a phrase length, the phrase length of each word segment indicating the number of words constituting the corresponding word segment; determining the relevance parameter between each initial language model and the text to be processed based on the multiple word segments, each relevance parameter characterizing the relevance between the corresponding initial language model and the text to be processed; wherein, the specific implementation of determining the relevance parameter between the third initial language model and the text to be processed based on the multiple word segments includes: calling the third initial language model to calculate the relevance parameter between the third initial language model and the text to be processed based on the multiple word segments, the third initial language model being any one of the initial language models of the at least one scenario; the relevance parameter characterizing the relevance between the text to be processed and the scenario corresponding to the third initial language model.

3. The language model construction method according to claim 2, characterized in that, The step of calling the third initial language model to calculate the relevance parameters between the third initial language model and the text to be processed based on the multiple word segments includes: calling the third initial language model to calculate the credibility parameter of each word segment among the multiple word segments, wherein the credibility parameter is used to characterize the credibility of the corresponding word segment as a word segment in the corresponding scenario of the third initial language model; and calculating the relevance parameters between the third initial language model and the text to be processed based on the credibility parameter of each word segment.

4. The language model construction method according to claim 3, characterized in that, The relevance parameter between the initial language model and the text to be processed includes: the perplexity parameter between the initial language model and the text to be processed. The degree of relevance between the text to be processed and the first initial language model meets a preset condition, including: the perplexity parameter between the text to be processed and the first initial language model is less than a preset threshold.

5. The language model construction method according to claim 1, characterized in that, The step of fusing the language model corresponding to the text to be processed with the first initial language model to obtain a fused model includes: determining interpolation coefficients based on the correlation between the language model corresponding to the text to be processed and the first initial language model; and interpolating the language model of the text to be processed with the first initial language model based on the interpolation coefficients to obtain the fused model.

6. The language model construction method according to claim 4 or 5, characterized in that, The step of determining the interpolation coefficient based on the correlation between the language model corresponding to the text to be processed and the first initial language model includes: calculating the difference between a preset threshold and the perplexity parameter corresponding to the first initial language model; the perplexity parameter corresponding to the first initial language model is the perplexity parameter between the text to be processed and the first initial language model; and using the ratio of the difference to the preset threshold as the interpolation coefficient.

7. The language model construction method according to claim 5, characterized in that, The phrase length corresponding to each word segment is an integer greater than or equal to 2. The step of interpolating the language model of the text to be processed with the first initial language model according to the interpolation coefficient to obtain a fusion model includes: for each phrase length, obtaining the probability distribution of the language model of the text to be processed for each word segment of the corresponding phrase length as a first probability distribution; for each phrase length, obtaining the probability distribution of the first initial language model for word segments of the corresponding phrase length in the corresponding scene text as a second probability distribution, wherein the scene text corresponding to the first initial language model comes from the base text; for each phrase length, using the interpolation coefficient as algorithm weights, fusing the corresponding first probability distribution and the corresponding second probability distribution through a weighted algorithm; replacing the model parameters of the first initial language model with the fused data, and using the language model after replacing the model parameters as the fusion model.

8. A language model construction device, characterized in that, include: The acquisition module is used to acquire the text to be processed; A determination module is used to determine the degree of relevance between the text to be processed and the initial language model of at least one pre-trained scenario, wherein the initial language model of at least one scenario is trained based on the base text. The fusion module is configured to, if the relevance between the text to be processed and the first initial language model meets a preset condition, fuse the language model corresponding to the text to be processed with the first initial language model to obtain a fusion model, wherein the first initial language model is one of the initial language models of the at least one scenario; and replace the first initial language model with the fusion model as a new initial language model for the first scenario corresponding to the first initial language model; the update module is configured to, if the relevance between the text to be processed and the initial language models of the at least one scenario does not meet the preset condition, use the language model corresponding to the text to be processed as the initial language model for a second scenario, wherein the second scenario refers to the scenario associated with the text to be processed, and the second scenario is different from the at least one scenario.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the language model construction method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the language model construction method as described in any one of claims 1-7.

Images