A language model training method, device, electronic device and storage medium

Abstract

The present application provides a language model training method, apparatus, electronic device, and storage medium, comprising: obtaining initial text data related to a predetermined natural language processing task according to the predetermined natural language processing task; obtaining a text loading template corresponding to the predetermined natural language processing task for loading training text samples; loading the initial text data according to the text loading template to obtain training text samples for training the language model; wherein the training text samples include training text with identification information and training text without identification information; and iteratively updating the initial language model using the training text samples to generate a target language model. The present application improves model accuracy by simultaneously training the language model using training text with identification information and training text without identification information.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method, device, electronic device and storage medium for training a language model. Background Art

[0002] Pre-trained language models are the basic research work of Natural Language Processing, and are widely used in various task scenarios such as text classification, semantic similarity, and entity recognition. After Google released the open source pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) in 2019, research and application in this field have become increasingly popular. The standard paradigm used for models currently applied to various natural language tasks is pre-training + fine-tuning (Pretrain+Finetune), that is, first pre-training a language model on a large amount of unlabeled corpus, and then adding some modules such as fully connected layers to the model, and putting it on the labeled data of the task for Finetune (fine-tuning). However, this training method will lead to a gap between the model in the pre-training stage and the model in the fine-tuning stage of the downstream task, resulting in a lower model accuracy of the final language model, and this training method also requires a lot of manual labeling, which makes the training cost higher. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a language model training method, device, electronic device and storage medium, which can improve the model accuracy by using training text with identification information and training text without identification information to simultaneously train the language model.

[0004] The present invention provides a method for training a language model, which includes:

[0005] According to a preset natural language processing task, obtaining initial text data related to the preset natural language processing task;

[0006] Obtaining a text loading template for loading a training text sample corresponding to the preset natural language processing task;

[0007] Loading initial text data according to the text loading template to obtain training text samples for training a language model; wherein the training text samples include training text with identification information and training text without identification information;

[0008] The initial language model is iteratively updated using the training text samples to generate a target language model.

[0009] Optionally, when the preset natural language processing task is a news topic classification task, the training method includes:

[0010] Obtain initial news text data;

[0011] Obtaining a text loading template for the news topic classification task;

[0012] Loading initial news text data according to the text loading template of the news topic classification task to obtain training text samples for training the news topic classification model; wherein the training text samples include training text with identification information and training text without identification information;

[0013] The initial news topic classification model is iteratively updated using the training text samples to generate a target news topic classification model.

[0014] Optionally, after obtaining the initial text data, the training method further includes:

[0015] The initial text data is preprocessed to remove special characters, spaces, garbled characters and to cut the initial text data into a predetermined length to obtain preprocessed initial text data, wherein the preprocessed initial text data is the initial text data loaded by the text loading template; the initial text data includes text data with identification information and text data without identification information.

[0016] Optionally, before obtaining a text loading template for loading a training text sample corresponding to the preset natural language processing task, the training method further includes:

[0017] Obtain multiple text loading templates pre-designed by the user and the natural language processing tasks corresponding to each text loading template;

[0018] Each acquired text loading template is bound to the corresponding natural language processing task and stored to construct a text loading template library; the text loading template includes a text loading position and a text answer position.

[0019] Optionally, loading initial text data according to the text loading template to obtain a training text sample for training a language model includes:

[0020] Loading the text in the text data with identification information into the text loading position in the text loading template, and loading the identification information corresponding to the text into the text answer position in the text loading template to generate a training text with identification information;

[0021] The text data without identification information is used as the training text without identification information.

[0022] Optionally, the natural language processing task includes a sentiment classification task, a news topic classification task, an intent recognition task, a named entity recognition task, and a semantic matching task. When the natural language processing task to be processed is an intent recognition task, after generating the target language model, the training method further includes:

[0023] Obtain the trained intent recognition model, intent recognition text loading template, and the text data to be processed for intent recognition;

[0024] Loading the to-be-processed text data into the intention recognition text loading template, and determining the to-be-predicted text data whose text answer position is empty;

[0025] The text data to be predicted is output to the intention recognition model, a predicted answer of a text answer position in the text data to be predicted is determined, and the predicted answer is determined as an intention recognition result of the text data to be processed.

[0026] The present application also provides a language model training device, the training device comprising:

[0027] A first acquisition module is used to acquire initial text data related to a preset natural language processing task according to the preset natural language processing task;

[0028] A second acquisition module is used to acquire a text loading template for loading a training text sample corresponding to the preset natural language processing task;

[0029] A loading module, configured to load initial text data according to the text loading template to obtain training text samples for training a language model; wherein the training text samples include training text with identification information and training text without identification information;

[0030] The generation module is used to iteratively update the initial language model using the training text samples to generate a target language model.

[0031] Optionally, when the preset natural language processing task is a news topic classification task, the training device is used to:

[0032] Obtain initial news text data;

[0033] Obtaining a text loading template for the news topic classification task;

[0034] Loading initial news text data according to the text loading template of the news topic classification task to obtain training text samples for training the news topic classification model; wherein the training text samples include training text with identification information and training text without identification information;

[0035] The initial news topic classification model is iteratively updated using the training text samples to generate a target news topic classification language model.

[0036] Optionally, the training device further includes a preprocessing module, which is configured to:

[0037] The initial text data is preprocessed to remove special characters, spaces, garbled characters and to cut the initial text data into a predetermined length to obtain preprocessed initial text data, wherein the preprocessed initial text data is the initial text data loaded by the text loading template; the initial text data includes text data with identification information and text data without identification information.

[0038] Optionally, the training device further includes a template library construction module, and the template library construction module is used to:

[0039] Obtain multiple text loading templates pre-designed by the user and the natural language processing tasks corresponding to each text loading template;

[0040] Each acquired text loading template is bound to the corresponding natural language processing task and stored to construct a text loading template library; the text loading template includes a text loading position and a text answer position.

[0041] Optionally, when the loading module is used to load initial text data according to the text loading template to obtain a training text sample for training a language model, the loading module is used to:

[0042] Loading the text in the text data with identification information into the text loading position in the text loading template, and loading the identification information corresponding to the text into the text answer position in the text loading template to generate a training text with identification information;

[0043] The text data without identification information is used as the training text without identification information.

[0044] Optionally, the training device further includes an application module. When the natural language processing task to be processed is an intent recognition task, the application module is configured to:

[0045] Obtain the trained intent recognition model, intent recognition text loading template, and the text data to be processed for intent recognition;

[0046] Loading the to-be-processed text data into the intention recognition text loading template, and determining the to-be-predicted text data whose text answer position is empty;

[0047] The text data to be predicted is output to the intention recognition model, a predicted answer of a text answer position in the text data to be predicted is determined, and the predicted answer is determined as an intention recognition result of the text data to be processed.

[0048] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the training method described above are performed.

[0049] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the training method described above are executed.

[0050] The embodiments of the present application provide a language model training method, device, electronic device and storage medium, including: obtaining initial text data related to a preset natural language processing task according to the preset natural language processing task; obtaining a text loading template for loading training text samples corresponding to the preset natural language processing task; loading the initial text data according to the text loading template to obtain training text samples for training the language model; wherein the training text samples include training text with identification information and training text without identification information; and using the training text samples to iteratively update the initial language model to generate a target language model.

[0051] This application generates training text with identification information by using a text loading template, and then uses the training text with identification information and the training text without identification information to train the language model at the same time. In this way, the pre-training and fine-tuning stage tasks can be combined into one training task, so that a better language model can be trained on less manually labeled data, thereby eliminating the need for a large amount of manual labeling costs and reducing model training time.

[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0054] Figure 1A flowchart of a language model training method provided in an embodiment of the present application;

[0055] Figure 2 This is a structural diagram of a language model training device provided in an embodiment of the present application;

[0056] Figure 3 A second structural diagram of a language model training device provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0059] The standard model used for various natural language tasks is pretraining + fine-tuning. This involves pretraining a language model on a large amount of unlabeled data, then adding modules such as fully connected layers to the model and fine-tuning it on labeled data for the task. However, this training approach can lead to a discrepancy between the pretraining model and the downstream fine-tuning model, resulting in lower accuracy for the resulting language model. Furthermore, this training approach requires extensive manual annotation, which increases training costs.

[0060] Based on this, an embodiment of the present application provides a language model training method, which achieves the end-to-end training goal and can improve the model accuracy.

[0061] See also Figure 1 , Figure 1 This is a flow chart of a method for training a language model provided in an embodiment of the present application. Figure 1 As shown in , the language model training method provided in the embodiment of the present application includes:

[0062] S101. According to a preset natural language processing task, initial text data related to the preset natural language processing task is obtained.

[0063] It should be noted that natural language processing tasks can include lexical analysis tasks, sentence analysis tasks, semantic analysis tasks, information extraction tasks and top-level tasks.

[0064] The lexical analysis task is to analyze natural language at the lexical level, which is the basic work of natural language processing (NLP). It can specifically include word segmentation, new word discovery, morphological analysis, part-of-speech tagging, spelling correction, etc.

[0065] The sentence analysis tasks are sentence-level analysis of natural language, including syntactic analysis and other sentence-level analysis tasks, specifically including: chunk analysis, super tagging, component syntactic analysis, dependency syntactic analysis, language identification, etc.

[0066] The semantic analysis task is to analyze and understand the given text to form a formal representation or distributed representation that can express semantics, including: word sense disambiguation, semantic role labeling, abstract semantic representation analysis, first-order predicate logic calculus, frame semantic analysis, vectorized representation of vocabulary / sentences / paragraphs, etc.

[0067] The information extraction tasks are to extract structured information from unstructured text, including: named entity recognition, entity disambiguation, term extraction, coreference resolution, relationship extraction, event extraction, sentiment analysis, intent recognition, slot filling, etc.

[0068] The top-level task is a system-level task that provides natural language processing product services directly to ordinary users. It will use natural language processing technologies at multiple levels, including: machine translation, text summarization, question-answering systems, dialogue systems, etc.

[0069] The natural language processing tasks in this step may include all of the above tasks, or may include some of the above tasks. For example, the natural language processing tasks include sentiment classification tasks, news topic classification tasks, intent recognition tasks, named entity recognition tasks, and semantic matching tasks.

[0070] The initial text data obtained here is determined by the specific, pre-defined natural language processing task. For example, to develop a knowledge question-answering system for the insurance sector, a crawler could be used to collect text from encyclopedias of insurance terminology, insurance clauses, and other related text. During the crawling process, as much unsupervised text data relevant to the task as possible is collected, along with some supervised text data. Alternatively, some unsupervised text data can be extracted from the collected unsupervised text data and manually labeled to generate supervised text data.

[0071] In an example of the present application, after obtaining the initial text data, the training method further includes: preprocessing the initial text data, removing special characters, spaces, garbled characters, and cropping the initial text data into a predetermined length to obtain preprocessed initial text data, wherein the preprocessed initial text data is the initial text data loaded by the text loading template; the initial text data includes text data with identification information and text data without identification information.

[0072] Preprocessing here includes one or more of the following: regularization and text length trimming. Regularization cleans up the initial text data, removing meaningless special characters, spaces, and garbled characters. Text length trimming trims overly long text to a length that meets the model training requirements.

[0073] S102: Obtain a text loading template corresponding to the preset natural language processing task for loading training text samples.

[0074] In one example of the present application, before obtaining a text loading template for loading training text samples corresponding to the preset natural language processing task, the training method further includes: obtaining multiple text loading templates pre-designed by the user and the natural language processing task corresponding to each text loading template; binding and storing each obtained text loading template and the corresponding natural language processing task to construct a text loading template library; the text loading template includes a text loading position and a text answer position.

[0075] In this step, the user pre-designs a text loading template based on the specific natural language processing task being processed. These templates are prompt templates. Once designed, each text loading template is bound to its corresponding natural language processing task. A text loading template library is constructed based on these pre-designed templates. Each text loading template includes a text loading location and a text answer location.

[0076] It should be noted that the text loading template in this step is generated based on prompt. This template is usually a piece of natural language and contains two or more empty positions. These empty positions include at least one position for loading text and one position for generating a text answer.

[0077] For example, see Table 1, which shows an example of a text loading template. As shown in Table 1, different natural language processing tasks correspond to different text loading templates.

[0078] Table 1: Text loading template example

[0079] Natural language tasks Template Examples Sentiment Analysis

text

answer

text

answer

text

answer

text

answer

text

text

answer

[0080] S103 , loading initial text data according to the text loading template to obtain training text samples for training a language model; wherein the training text samples include training texts with identification information and training texts without identification information.

[0081] In one example of the present application, the initial text data is loaded according to the text loading template to obtain a training text sample for training a language model, including: loading the text in the text data with identification information into the text loading position in the text loading template, loading the identification information corresponding to the text into the text answer position in the text loading template to generate training text with identification information; and using the text data without identification information as the training text without identification information.

[0082] This step involves preprocessing the initial text data to obtain initial text data that meets the requirements, and then generating samples for model training. Specifically, the steps include: using the obtained text loading template to load text data with identification information in the initial text data, loading the text data into the text loading position in the text loading template, loading the identification information corresponding to the text data into the text answer position of the text loading template, and using the natural language after loading as a training sample for the language model. This sample is training text with identification information. For text data without identification information in the initial text data, because this text data lacks identification information and cannot fill the text answer position in the text loading model, this text data is directly used as a training sample for the model. This sample is training text without identification information.

[0083] Here, through the above steps, a plurality of training texts with identification information and a plurality of training texts without identification information can be obtained.

[0084] For example, please refer to Table 1. Assume that the natural language task to be processed is news topic classification, and the obtained text loading template is "[text], this report is about [answer]." The initial text data obtained includes text data with identification information such as "The Lakers defeated the Rockets by a large margin in the semifinals, sports," text data without identification information such as "XX will host the 20AA annual sports meeting," and text data without identification information such as "The Lakers defeated the Rockets by a large margin in the semifinals." When constructing training text samples using the text loading template, training text with identification information such as "The Lakers defeated the Rockets by a large margin in the semifinals, this report is about sports," training text without identification information such as "XX will host the 20AA annual sports meeting," and training text without identification information such as "The Lakers defeated the Rockets by a large margin in the semifinals."

[0085] S104: Iteratively update the initial language model using the training text samples to generate a target language model.

[0086] In an example of the present application, after obtaining multiple training text samples, the training method further includes: dividing the training text samples into two groups of training text samples according to a preset sample segmentation ratio, one group being a training set and the other group being a verification set; wherein, the training set is used for training the initial language model, and the verification set is used for verifying the model effect during the training process.

[0087] It should be noted that the initial language model can be a BERT model. During the model training process, methods such as gradient backpropagation, gradient descent, and adaptive time estimation can be used for training. When the loss function converges or meets the preset requirements, the training is stopped and the target language model is generated.

[0088] In addition, after training, save the Prompt template as a json format file and save the trained model parameters as a binary format file for prediction in the online system.

[0089] In another example of the present application, when the pre-set natural language processing task is a news topic classification task, the training method includes: obtaining initial news text data; obtaining a text loading template for the news topic classification task; loading the initial news text data according to the text loading template for the news topic classification task to obtain training text samples for training a news topic classification model; wherein the training text samples include training text with identification information and training text without identification information; and using the training text samples to iteratively update the initial news topic classification model to generate a target news topic classification model.

[0090] This step is the specific process of training and generating a news topic classification model when the preset natural language processing task is the news topic classification task.

[0091] In another example of the present application, the natural language processing tasks include sentiment classification tasks, news topic classification tasks, intent recognition tasks, named entity recognition tasks, and semantic matching tasks. When the natural language processing task to be processed is an intent recognition task, after generating the target language model, the training method also includes: obtaining a trained intent recognition model, an intent recognition text loading template, and text data to be processed that requires intent recognition; loading the text data to be processed into the intent recognition text loading template, and determining the text data to be predicted whose text answer position is empty; outputting the text data to be predicted to the intent recognition model, determining the predicted answer of the text answer position in the text data to be predicted, and determining the predicted answer as the intent recognition result of the text data to be processed.

[0092] For example, in actual application, when the natural language processing task to be solved is intent recognition, the text loading template selected is the text loading template for intent recognition "

text

answer

answer

answer

[0093] A language model training method provided in an embodiment of the present application includes: obtaining initial text data related to a preset natural language processing task according to the preset natural language processing task; obtaining a text loading template corresponding to the preset natural language processing task for loading training text samples; loading the initial text data according to the text loading template to obtain training text samples for training the language model; wherein the training text samples include training text with identification information and training text without identification information; and iteratively updating the initial language model using the training text samples to generate a target language model.

[0094] This application generates training text with identification information by using a text loading template, and then uses the training text with identification information and the training text without identification information to train the language model at the same time. In this way, the pre-training and fine-tuning stage tasks can be combined into one training task, so that a better language model can be trained on less manually labeled data, thereby eliminating the need for a large amount of manual labeling costs and reducing model training time.

[0095] See also Figure 2 、 Figure 3 , Figure 2 This is one of the structural diagrams of a language model training device provided in an embodiment of the present application. Figure 3 This is a second structural diagram of a language model training device provided in an embodiment of the present application. Figure 2 As shown in FIG, the training device 200 includes:

[0096] A first acquisition module 210 is configured to acquire initial text data related to a preset natural language processing task according to the preset natural language processing task;

[0097] A second acquisition module 220 is configured to acquire a text loading template corresponding to the preset natural language processing task and used to load a training text sample;

[0098] The loading module 230 is used to load the initial text data according to the text loading template to obtain a training text sample for training the language model; wherein the training text sample includes a training text with identification information and a training text without identification information;

[0099] The generation module 240 is configured to iteratively update the initial language model using the training text samples to generate a target language model.

[0100] Optionally, when the preset natural language processing task is a news topic classification task, the training device 200 is used to:

[0101] Obtain initial news text data;

[0102] Obtaining a text loading template for the news topic classification task;

[0103] Loading initial news text data according to the text loading template of the news topic classification task to obtain training text samples for training the news topic classification model; wherein the training text samples include training text with identification information and training text without identification information;

[0104] The initial news topic classification model is iteratively updated using the training text samples to generate a target news topic classification language model.

[0105] Optional, such as Figure 3 As shown, the training device 200 further includes a pre-processing module 250, which is used to:

[0106] The initial text data is preprocessed to remove special characters, spaces, garbled characters and to cut the initial text data into a predetermined length to obtain preprocessed initial text data, wherein the preprocessed initial text data is the initial text data loaded by the text loading template; the initial text data includes text data with identification information and text data without identification information.

[0107] Optionally, the training device 200 further includes a template library construction module 260, and the template library construction module 260 is used to:

[0108] Obtain multiple text loading templates pre-designed by the user and the natural language processing tasks corresponding to each text loading template;

[0109] Each acquired text loading template is bound to the corresponding natural language processing task and stored to construct a text loading template library; the text loading template includes a text loading position and a text answer position.

[0110] Optionally, when the loading module 230 is used to load initial text data according to the text loading template to obtain a training text sample for training a language model, the loading module 230 is used to:

[0111] Loading the text in the text data with identification information into the text loading position in the text loading template, and loading the identification information corresponding to the text into the text answer position in the text loading template to generate a training text with identification information;

[0112] The text data without identification information is used as the training text without identification information.

[0113] Optionally, the training device 200 further includes an application module 270. When the natural language processing task to be processed is an intent recognition task, the application module 270 is configured to:

[0114] Obtain the trained intent recognition model, intent recognition text loading template, and the text data to be processed for intent recognition;

[0115] Loading the to-be-processed text data into the intention recognition text loading template, and determining the to-be-predicted text data whose text answer position is empty;

[0116] The text data to be predicted is output to the intention recognition model, a predicted answer of a text answer position in the text data to be predicted is determined, and the predicted answer is determined as an intention recognition result of the text data to be processed.

[0117] An embodiment of the present application provides a language model training device, comprising: obtaining initial text data related to a preset natural language processing task according to the preset natural language processing task; obtaining a text loading template corresponding to the preset natural language processing task for loading training text samples; loading the initial text data according to the text loading template to obtain training text samples for training the language model; wherein the training text samples include training text with identification information and training text without identification information; and iteratively updating the initial language model using the training text samples to generate a target language model.

[0118] This application generates training text with identification information by using a text loading template, and then uses the training text with identification information and the training text without identification information to train the language model at the same time. In this way, the pre-training and fine-tuning stage tasks can be combined into one training task, so that a better language model can be trained on less manually labeled data, thereby eliminating the need for a large amount of manual labeling costs and reducing model training time.

[0119] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in FIG, the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0120] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The specific implementation of the steps of the training method in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0121] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps of the training method in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0122] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0126] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0127] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for training a language model, characterized in that: The training method comprises: According to a preset natural language processing task, initial text data related to the preset natural language processing task is obtained; the initial text data includes text data with identification information and text data without identification information; Obtaining a text loading template for loading a training text sample corresponding to the preset natural language processing task; the text loading template includes at least one text loading position for loading text and a text answer position for generating a text answer; Loading initial text data according to the text loading template to obtain training text samples for training a language model; wherein the training text samples include training text with identification information and training text without identification information; Iteratively updating the initial language model using the training text samples to generate a target language model; When the natural language processing task to be processed is an intent recognition task, after generating the target language model, the training method further includes: Obtaining a trained intent recognition model, an intent recognition text loading template, and text data to be processed for intent recognition; wherein the intent recognition text loading template includes a text loading position and a text answer position; Loading the to-be-processed text data into the text loading position in the intention recognition text loading template, and determining the to-be-predicted text data whose text answer position is empty; Outputting the text data to be predicted to the intention recognition model, determining a predicted answer for a text answer position in the text data to be predicted, and determining the predicted answer as an intention recognition result of the text data to be processed; Initial text data is loaded according to the text loading template to obtain a training text sample for training a language model, including: for text data with identification information in the initial text data, the text data is loaded into the text loading position in the text loading template, the identification information corresponding to the text data is loaded into the text answer position of the text loading template, and the natural language after loading is used as a training sample of the language model, and the sample is a training text with identification information; for text data without identification information in the initial text data, the text data is directly used as a training sample of the model, and the training sample is a training text without identification information.

2. The training method according to claim 1, characterized in that When the preset natural language processing task is a news topic classification task, the training method includes: Obtain initial news text data; Obtaining a text loading template for the news topic classification task; Loading initial news text data according to the text loading template of the news topic classification task to obtain training text samples for training the news topic classification model; wherein the training text samples include training text with identification information and training text without identification information; The initial news topic classification model is iteratively updated using the training text samples to generate a target news topic classification model.

3. The training method according to claim 1, characterized in that After obtaining the initial text data, the training method further includes: The initial text data is preprocessed to remove special characters, spaces, garbled characters and to cut the initial text data into a predetermined length to obtain preprocessed initial text data, which is the initial text data loaded by the text loading template.

4. The training method according to claim 3, characterized in that Before obtaining a text loading template for loading a training text sample corresponding to the preset natural language processing task, the training method further includes: Obtain multiple text loading templates pre-designed by the user and the natural language processing tasks corresponding to each text loading template; Each acquired text loading template is bound to the corresponding natural language processing task and stored to build a text loading template library.

5. The training method according to claim 1, wherein: The natural language processing tasks include sentiment classification tasks, news topic classification tasks, intent recognition tasks, named entity recognition tasks, and semantic matching tasks.

6. A language model training device, characterized in that: The training device comprises: A first acquisition module is configured to acquire initial text data related to a preset natural language processing task according to the preset natural language processing task; the initial text data includes text data with identification information and text data without identification information; a second acquisition module, configured to acquire a text loading template for loading a training text sample corresponding to the preset natural language processing task; the text loading template comprising at least one text loading position for loading text and a text answer position for generating a text answer; A loading module, configured to load initial text data according to the text loading template to obtain training text samples for training a language model; wherein the training text samples include training text with identification information and training text without identification information; A generation module, configured to iteratively update the initial language model using the training text samples to generate a target language model; The training device further includes an application module. When the natural language processing task to be processed is an intent recognition task, the application module is configured to: Obtaining a trained intent recognition model, an intent recognition text loading template, and text data to be processed for intent recognition; wherein the intent recognition text loading template includes a text loading position and a text answer position; Loading the to-be-processed text data into the text loading position in the intention recognition text loading template, and determining the to-be-predicted text data whose text answer position is empty; Outputting the text data to be predicted to the intention recognition model, determining a predicted answer for a text answer position in the text data to be predicted, and determining the predicted answer as an intention recognition result of the text data to be processed; Initial text data is loaded according to the text loading template to obtain a training text sample for training a language model, including: for text data with identification information in the initial text data, the text data is loaded into the text loading position in the text loading template, the identification information corresponding to the text data is loaded into the text answer position of the text loading template, and the natural language after loading is used as a training sample of the language model, and the sample is a training text with identification information; for text data without identification information in the initial text data, the text data is directly used as a training sample of the model, and the training sample is a training text without identification information.

7. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executed by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the training method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the training method according to any one of claims 1 to 5 are executed.

Images