A language model pre-training method, device, medium and electronic equipment

Abstract

The specification discloses a pre-training method and device of a language model, a medium and an electronic device. A sample text is obtained first, and the sample text is input into a feature extraction network to extract text features of the sample text, and prompt features are generated through a prompt network. Then, the text features and the prompt features are fused to obtain fused features. Finally, for each preset training task, the encoding end of the language model is pre-trained according to the fused features and a training target of the training task. By adding the prompt network to the encoding end of the pre-trained language model, the text features of the sample text and the prompt features obtained through the prompt network can be fused when the language model is pre-trained, and then the training target is completed according to the training task, so that the pre-trained language model can better adapt to the actual tasks in the application scenarios, thereby improving the task execution capability of the language model and achieving better results.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a language model pre-training method and device, a medium and an electronic device. BACKGROUND

[0002] With the development of science and technology, privacy data is increasingly concerned. With the development of science and technology, artificial intelligence is developing rapidly. Among them, natural language processing (NLP) is widely concerned, and natural language processing researches various theories and methods for effective communication between people and computers through natural language.

[0003] Generally, a general corpus can be used as a sample to pre-train a model to obtain a pre-trained language model, which is subsequently used to process language texts in different scenarios to implement actual businesses in different scenarios, such as performing sentiment analysis on evaluation sentences after user shopping to evaluate the popularity of the item, performing text mining according to user browsing records to recommend related information to the user, and the like. However, due to the large structural differences of language texts in different scenarios, the pre-trained language model is difficult to adapt to actual businesses in different scenarios. Therefore, how to make the pre-trained language model adapt to actual businesses in different scenarios is a problem to be solved.

[0004] Based on this, the present application provides a language model pre-training method. SUMMARY

[0005] The present application provides a language model pre-training method and device, a storage medium and an electronic device to at least partially solve the above problems existing in the prior art.

[0006] The present application adopts the following technical solutions:

[0007] The present application provides a language model pre-training method, which is used to train an encoding end in a language model, the encoding end at least including a feature extraction network and a prompt network, and the method includes:

[0008] obtaining a sample text;

[0009] inputting the sample text into the feature extraction network to extract text features from the sample text through the feature extraction network; and generating prompt features through the prompt network;

[0010] determining fusion features according to the text features and the prompt features;

[0011] For each preset training task, the encoding end is pre-trained according to the fusion features and a training target of the training task.

[0012] Optionally, the prompt feature is generated by the prompt network, and specifically includes:

[0013] The prompt information is obtained and input into the prompt network to extract the prompt feature from the prompt information by the prompt network.

[0014] Optionally, the language model further includes a decoding end;

[0015] The sample text is input into the feature extraction network, and specifically includes:

[0016] The sample text is processed by a mask, and the processed sample text is input into the feature extraction network;

[0017] According to the fusion feature and the training target of the training task, the encoding end is trained, and specifically includes:

[0018] The fusion feature is input into the decoding end to obtain an output text that restores the processed sample text;

[0019] The encoding end is trained with the goal of minimizing the difference between the output text and the sample text.

[0020] Optionally, the sample text is input into the feature extraction network, and specifically includes:

[0021] For each sample text, a derivative text related to the sample text is determined;

[0022] The sample text and the derivative text of the sample text are input into the encoding end of the language model as positive samples;

[0023] Other sample texts except the sample text are input into the encoding end of the language model as negative samples;

[0024] According to the text feature and the prompt feature, a fusion feature is determined, and specifically includes:

[0025] Based on the input positive sample, the fusion feature corresponding to the positive sample is determined according to the text feature extracted from the positive sample and the prompt feature;

[0026] Based on the input negative sample, the fusion feature corresponding to the negative sample is determined according to the text feature extracted from the negative sample and the prompt feature;

[0027] According to the fusion feature and the training target of the training task, the encoding end is trained, and specifically includes:

[0028] The encoding end is trained with a target of maximizing similarity between corresponding fusion features of positive samples and minimizing similarity between the corresponding fusion features of the positive samples and fusion features of negative samples.

[0029] Optionally, the method further comprises:

[0030] Optionally, the method further comprises:

[0031] Optionally, the method further comprises:

[0032] Optionally, the method further comprises:

[0033] Optionally, the method further comprises:

[0034] Optionally, the method further comprises:

[0035] Optionally, the method further comprises:

[0036] Optionally, the method further comprises:

[0037] Optionally, the method further comprises:

[0038] Optionally, the method further comprises:

[0039] The present specification provides a language model pre-training device, the device is used for training an encoding end in a language model, the encoding end at least includes a feature extraction network and a prompt network, comprising:

[0040] An acquisition module is configured to acquire a sample text.

[0041] An extraction module is configured to input the sample text into the feature extraction network to extract text features from the sample text through the feature extraction network, and generate prompt features through the prompt network.

[0042] A fusion module is configured to determine fusion features according to the text features and the prompt features.

[0043] The training module is configured to pre-train the encoding end according to the fusion feature and a training target of each preset training task.

[0044] Optionally, the extraction module is specifically configured to obtain prompt information and input the prompt information into the prompt network to extract prompt features from the prompt information through the feature extraction network.

[0045] Optionally, the language model further comprises a decoding end.

[0046] The extraction module is specifically configured to process the sample text through a mask and input the processed sample text into the feature extraction network.

[0047] The training module is specifically configured to input the fusion feature into the decoding end to obtain output text for restoring the processed sample text, and train the encoding end with the smallest difference between the output text and the sample text as the target.

[0048] Optionally, the extraction module is specifically configured to determine, for each sample text, a derivative text related to the sample text, input the sample text and the derivative text of the sample text as positive samples into the encoding end of the language model, and input other sample texts except the sample text as negative samples into the encoding end of the language model.

[0049] The fusion module is specifically configured to determine, based on the input positive samples, the fusion features corresponding to the positive samples according to the text features extracted from the positive samples and the prompt features, and determine, based on the input negative samples, the fusion features corresponding to the negative samples according to the text features extracted from the negative samples and the prompt features.

[0050] The training module is specifically configured to train the encoding end with the maximum similarity between the fusion features corresponding to the positive samples and the minimum similarity between the fusion features corresponding to the positive samples and the fusion features corresponding to the negative samples as the target.

[0051] Optionally, the extraction module is specifically configured to determine a parallel corpus text of the sample text, and take the parallel corpus text as the derivative text of the sample text, or insert a stop word in the sample text, and take the sample text after inserting the stop word as the derivative text.

[0052] Optionally, the training module is further configured to determine a target business to which the pre-trained language model is to be applied, acquire business sample texts and labels corresponding to the business sample texts according to the target business, and fine-tune the pre-trained language model according to the business sample texts and the labels corresponding to the business sample texts to obtain the trained language model.

[0053] Optionally, the training module is further configured to input the business sample texts into the language model to determine an output result, determine a loss according to the output result and the labels corresponding to the business sample texts, and adjust parameters of the prompt network of the language model according to the loss.

[0054] The specification provides a computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the pre-training method of the language model.

[0055] The specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the pre-training method of the language model when executing the program.

[0056] The above at least one technical solution adopted by the specification can achieve the following beneficial effects:

[0057] In the pre-training method of the language model provided by the specification, sample texts can be acquired first, the sample texts are input into a feature extraction network to extract text features of the sample texts, and prompt features are generated through a prompt network. Then the text features and the prompt features are fused to obtain fused features. Finally, for each preset training task, the encoding end of the language model is pre-trained according to the fused features and a training target of the training task.

[0058] As can be seen from the above method, by adding a prompt network to the encoding end of the pre-trained language model, the text features of the sample texts and the prompt features obtained through the prompt network can be fused when the language model is pre-trained, and then the training target is completed according to the training task. The pre-trained language model can better adapt to the actual tasks in the application scenario to improve the ability of the language model to perform actual tasks, so that the language model achieves better results when performing actual tasks. BRIEF DESCRIPTION OF DRAWINGS

[0059] The drawings described herein are used to provide further understanding of the specification, and form a part of the specification. The illustrative embodiments of the specification and their descriptions serve to explain the specification, and do not constitute an improper limitation on the specification. In the drawings:

[0060] Figure 1A flowchart of a pre-training method of a language model provided in the specification;

[0061] Figure 2 A pre-training diagram of a language model provided in the specification;

[0062] Figure 3 A pre-training diagram of a language model provided in the specification;

[0063] Figure 4 A pre-training device diagram of a language model provided in the specification;

[0064] Figure 5 A schematic diagram of an electronic device corresponding to Figure 1 provided in the specification. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the specification clearer, the technical solutions of the present application will be described in detail below with reference to specific embodiments of the specification and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0066] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0067] Figure 1 A flowchart of a pre-training method of a language model provided in the specification, which can specifically include the following steps:

[0068] S100: Obtain sample text.

[0069] Because the language text in actual business in different scenarios has great structural differences, it is difficult for the pre-trained language model to adapt to actual business in different scenarios. Based on this, the present specification provides a pre-training scheme of a language model, so that the pre-trained language model better adapts to actual business in different scenarios and has better ability to execute business. The execution subject executing the technical solutions of the present specification can be any device with computing ability (such as a server), and the present specification will be described with the server as the execution subject.

[0070] When pre-training the language model, the server can first obtain sample text. The sample text can be general corpus, which can be obtained in the corpus library, so that the language model can learn the basic logic in the sample text.

[0071] S102: input the sample text into the feature extraction network to extract text features from the sample text by the feature extraction network; and generate prompt features by the prompt network.

[0072] S104: determine fusion features according to the text features and the prompt features.

[0073] In one or more embodiments of the present specification, in order to enable the pre-trained language model to better adapt to the actual tasks in various application scenarios, a prompt network can be added to the encoding end of the language model, so that the language model can learn to extract prompt features through the prompt network during the pre-training process. That is, the language model can extract text features in the sample text based on the prompt features, so that the encoding end of the language model can learn what text features should be extracted in the sample text under the influence of the prompt features.

[0074] Specifically, the server can input the obtained sample text into the feature extraction network of the encoding end of the language model, so that the feature extraction network can extract text features in the sample text. And the prompt information can be obtained, and the prompt features in the prompt information can be extracted by the prompt network of the encoding end. And the fusion features are obtained by fusing the prompt features and the text features, so that the encoding end can be trained through the fusion features and combined with the training target of the training task in the subsequent steps.

[0075] Wherein, the prompt information can give the language model a "prompt" together with the input sample text, so that the language model learns the context of the text and the relationship between the prompt information and the sample text. The prompt information can be a prompt text derived from the actual task of the language model in different scenarios, or a text with a specified number of tokens, and the present specification does not make any limitation.

[0076] S106: For each preset training task, pre-train the encoding end according to the fusion features and the training target of the training task.

[0077] The server can pre-train the encoding end of the language model for each preset training task, based on the fusion features and the training objective of the training task. In one or more embodiments of this specification, the language model has an encoding end, which includes a feature extraction network and a prompting network. The training task can be a mask learning task, since the mask learning task needs to process the fusion features extracted by the encoding end to obtain the processed output text. Therefore, the language model may also include a decoding end. In step S102 above, the server can first process the sample text using a mask. For example, if the sample text is "The weather is really nice today.", then after masking, that is, after masking or replacing the token in the sample text, the sample text can be "[mask] day [mask] is really nice." The processed sample text "[mask] day [mask] is really nice." is then input into the feature extraction network to extract the text features in the processed sample text. The text features and prompting features are then fused to obtain the fusion features. Then, the fused feature is input into the decoder of the language model to obtain the output text after restoring the masked sample text. Continuing with the previous example, the output text is "The sky is so nice today." Finally, the encoder of the language model is trained with the goal of minimizing the difference between the output text "The sky is so nice today" and the sample text "The weather is so nice today."

[0078] It should be noted that, in one or more embodiments of this specification, the parameter adjustment of the language model is for the encoding end and does not require adjustment of the parameters of the decoding end.

[0079] In one or more embodiments of this specification, during the masking process described above, the token in the sample text can be masked with a certain probability, or the token in the sample text can be replaced with other tokens with a certain probability, etc.

[0080] like Figure 2 The diagram shown illustrates the pre-training of the language model provided in this specification. Figure 2 As can be seen, this language model has an encoder and a decoder. The encoder includes a feature extraction network and a prompting network. Sample text is masked and then input into the feature extraction network, while prompting information is input into the prompting network. The resulting text features and prompting features are fused to obtain a fused feature. This fused feature is then input into the decoder to obtain the output text. The loss can then be determined based on the output text and the sample text, and the parameters of the encoder in the language model can be adjusted accordingly.

[0081] In the feature fusion, a global context network (GCNet), a spatial pyramid pooling network (PSPNet), or the like can be used, and the specific feature fusion method is not limited in the specification. Moreover, how to perform mask processing on the sample text is a relatively mature technology, and thus will not be described herein.

[0082] The sample text is subjected to mask processing, and the prompt feature is extracted by the prompt network. The prompt feature is fused with the text feature to obtain a fused feature. The fused feature is input into a decoding end to obtain an output text. Then, the encoding end of the language model is pre-trained to minimize the difference between the output text and the sample text, so as to obtain a pre-trained language model. The language model can learn the context relationship of the text and the basic logic of the language, and the prompt feature is fused into the text feature, so that the pre-trained language model can better adapt to actual tasks in different scenarios.

[0083] Based on Figure 1 As shown in the language model pre-training method, the server adds a prompt network to the encoding end of the pre-trained language model, so that when the language model is pre-trained, the text feature of the sample text can be extracted, and the text feature is fused with the prompt feature obtained by the prompt network, and then the training target is completed according to the fused feature and the training task. By setting the prompt network in the encoding end to extract the prompt feature, the language model is pre-trained based on the prompt feature, so that the language model can learn the relationship between the prompt text and the sample text, that is, what kind of text feature should be extracted in the sample text under the influence of the prompt feature. Then, the pre-trained language model can extract the text feature in the sample text based on the prompt feature, so that the pre-trained language model can better adapt to actual tasks in different scenarios, improve the ability of the language model to perform tasks, and achieve better results.

[0084] Further, the pre-training method of the language model uses a mask learning task, which requires an output text corresponding to the fused feature extracted by the encoding end. Therefore, the decoding end is set in the language model to process the fused feature to obtain the output text. However, in the absence of the decoding end, in order to enable the language model to be pre-trained, the training task can be a contrast learning task.

[0085] Specifically, in one or more embodiments of the specification, when the sample text is input into the feature extraction network in the above step S102, a derivative text related to each sample text can also be determined.

[0086] The sample text and its derived text are semantically similar; that is, the semantic similarity between the sample text and its derived text is greater than a preset threshold. Therefore, when determining the derived text related to the sample text, we can identify its parallel corpus texts and use those parallel corpus texts as derived texts. Alternatively, we can insert stop words into the sample text and use the sample text with the inserted stop words as derived text. Other methods include synonym replacement, meaningless words in repeated sentences, etc., as long as semantically similar derived texts can be identified. This specification does not limit the specific methods used to determine derived texts.

[0087] Then, the sample text and its derived texts are used as positive samples, and all other sample texts are used as negative samples. The positive and negative samples are then input into the encoding end of the language model to extract features, resulting in fused features (the fusion of text features and prompt features for positive samples) and fused features (the fusion of text features and prompt features for negative samples). The language model's encoding end is trained with the goal of maximizing the similarity between the fused features corresponding to each positive sample and minimizing the similarity between the fused features corresponding to the positive and negative samples.

[0088] In one or more embodiments of this specification, when performing a contrastive learning task, although the sample text in the language model is unlabeled, since the derived text related to the sample text is semantically similar to the sample text, the extracted text features for the sample text and its related derived text should also be similar. However, the sample text and its related derived text are semantically dissimilar to other sample texts; therefore, the extracted text features of the sample text and its related derived text are dissimilar to the extracted text features of other sample texts. Therefore, the encoding end of the language model can be trained with the goal of maximizing the similarity between the fused features corresponding to each positive sample (i.e., the derived text and sample text related to the sample text) and minimizing the similarity between the fused features corresponding to the positive samples and the fused features corresponding to the negative samples (i.e., other sample texts).

[0089] like Figure 3 The diagram shown illustrates the pre-training of the language model provided in this specification. Figure 3As can be seen, the sample texts are n, and for sample text 1, the sample text 1 can be subjected to data enhancement processing to determine the derived text of the sample text 1. Then, the sample text 1 and the derived text of the sample text 1 are taken as positive samples, and the sample texts 2-n other than the sample text 1 are taken as negative samples, which are respectively input into the feature extraction network and combined with the prompt features in the prompt information extracted by the prompt network to obtain the fusion features corresponding to the positive samples and the fusion features corresponding to the negative samples. Further, the similarity between the fusion features corresponding to each positive sample can be maximized, and the similarity between the fusion features corresponding to the positive samples and the fusion features corresponding to the negative samples can be minimized, that is, the distance between the positive samples (samples of the same category) is short, and the distance between the positive samples and the negative samples (samples of different categories) is long. The encoding end is trained so that the pre-trained language model has better representation ability to adapt to actual tasks in different scenarios (such as sentence similarity matching, target recognition in a sentence, and sentiment classification of a sentence).

[0090] It should be noted that in the above language model pre-training method, the order of the feature extraction network and the prompt network in the encoding end is not fixed, Figure 2 and Figure 3 only one example, the prompt network can be in front and the feature extraction network can be behind. Moreover, the order of the above preset training tasks (mask learning task and contrast learning task) is not fixed.

[0091] Further, in one or more embodiments of the present specification, for the pre-trained language model, the server can also determine the target business to which it is to be applied, that is, the specific scenario in which it is applied, for example: the target business to which the pre-trained language model is applied is to perform sentiment analysis on the evaluation sentence of the user after shopping to evaluate whether the item is popular. Then, according to the target business, the business sample text and the label corresponding to the business sample text can be obtained. For example, the business sample text can be the user's evaluation sentence "This cup is very beautiful, I like it very much.", and the label corresponding to the business sample text is "positive" (i.e. popular with users).

[0092] Further, according to the business sample text and the label corresponding to the business sample text, the pre-trained language model is fine-tuned to obtain a trained language model, so that the trained language model can perform the target business.

[0093] In the fine-tuning of the pre-trained language model, since the pre-trained language model has learned the representation of the text well, only the prompt network part can be fine-tuned, that is, the parameters of other networks including the feature extraction network are frozen, so as to adjust fewer parameters in the language model, and a large amount of computing power can be saved. Specifically, the business sample text is input into the pre-trained language model to determine the output result. Then, according to the output result and the label corresponding to the business sample text, the loss is determined. That is, the output result corresponding to each business sample text and the label corresponding to each business sample text are input into the loss function, and the loss is calculated according to the loss function. Then, the gradient that minimizes the loss is determined, and the parameters of the prompt network in the pre-trained language model are adjusted according to the gradient descent method, so that the difference between the output result corresponding to each business sample text and the label corresponding to each business sample text is minimized.

[0094] According to the above method, the pre-trained language model is trained using the business sample text and the label corresponding to the business sample text, so that the trained language model can perform the business, and the ability of the language model to perform the target business is improved, and better results can be achieved in the application scenario.

[0095] Of course, when fine-tuning the pre-trained language model, the parameters of the feature extraction network and the prompt network in the pre-trained language model can also be adjusted at the same time. Specifically, when fine-tuning the parameters of the feature extraction network and the prompt network at the same time, the parameters can be adjusted according to a certain weight. Since the feature extraction network has learned the extraction of the text features of the sample text well to a certain extent, and the prompt network is to determine the prompt according to the target business to which the language model is applied, when adjusting the parameters according to the target business to which the language model is applied, the weight of the adjustment of the parameters of the prompt network is greater than the weight of the adjustment of the parameters of the feature extraction network.

[0096] It should be noted that in one or more embodiments of the present specification, whether in the above language model pre-training method or in the above pre-trained language model training method, the present specification does not limit when the (pre) training of the language model is determined to be completed. For example, when the number of (pre) training iterations reaches a preset threshold, it is determined that the (pre) training of the language model is completed, or when the determined loss is less than a preset value, it is determined that the (pre) training of the language model is completed.

[0097] Based on the above-described language model pre-training method, the present specification also correspondingly provides a pre-training device for a language model, as shown in Figure 4 .

[0098] Figure 4A schematic diagram of a pre-training device of a language model is provided for an embodiment of the present specification, the device is used to train an encoding end in a language model, the encoding end at least includes a feature extraction network and a prompt network, and the device includes:

[0099] An acquisition module 400 is configured to acquire sample text.

[0100] An extraction module 402 is configured to input the sample text into the feature extraction network to extract text features from the sample text through the feature extraction network, and generate prompt features through the prompt network.

[0101] A fusion module 404 is configured to determine fusion features according to the text features and the prompt features.

[0102] A training module 406 is configured to pre-train the encoding end according to the fusion features and a training target of each preset training task.

[0103] Optionally, the extraction module 402 is specifically configured to obtain prompt information and input the prompt information into the prompt network to extract prompt features from the prompt information through the prompt network.

[0104] Optionally, the language model further includes a decoding end.

[0105] The extraction module 402 is specifically configured to process the sample text through a mask and input the processed sample text into the feature extraction network.

[0106] The training module 406 is specifically configured to input the fusion features into the decoding end to obtain output text for restoring the processed sample text, and train the encoding end with the objective of minimizing the difference between the output text and the sample text.

[0107] Optionally, the extraction module 402 is specifically configured to determine, for each sample text, derivative text related to the sample text, input the sample text and the derivative text of the sample text as positive samples into the encoding end of the language model, and input other sample texts except the sample text as negative samples into the encoding end of the language model.

[0108] The fusion module 404 is specifically configured to determine, based on the input positive samples, fusion features corresponding to the positive samples according to the text features extracted from the positive samples and the prompt features, and determine, based on the input negative samples, fusion features corresponding to the negative samples according to the text features extracted from the negative samples and the prompt features.

[0109] The training module 406 is specifically configured to train the encoding end with the maximum similarity between the fusion features corresponding to each positive sample and the minimum similarity between the fusion features corresponding to the positive sample and the fusion features corresponding to the negative sample as the target.

[0110] Optionally, the extraction module 402 is specifically configured to determine a parallel corpus text of the sample text, and take the parallel corpus text as the derived text of the sample text; or, insert a stop word in the sample text, and take the sample text after the stop word is inserted as the derived text.

[0111] Optionally, the training module 406 is further configured to determine a target business to which a pre-trained language model is to be applied; acquire a business sample text and a label corresponding to the business sample text according to the target business; and fine-tune the pre-trained language model according to the business sample text and the label corresponding to the business sample text, to obtain a trained language model.

[0112] Optionally, the training module 406 is further configured to input the business sample text into the language model, and determine an output result; determine a loss according to the output result and the label corresponding to the business sample text; and adjust a parameter of a prompt network of the language model according to the loss.

[0113] The embodiments of the present specification further provide a computer-readable storage medium storing a computer program, and the computer program can be used to execute the pre-training method of the language model described in the above content.

[0114] Based on the pre-training method of the language model described in the above content, the embodiments of the present specification further provide Figure 5 The schematic structural diagram of the electronic device is shown in FIG. 1. As shown in FIG. 1, the electronic device includes a plurality of logical units, which are respectively a processor 101, a memory 102, a network interface 103, a non-volatile memory 104, and a power supply 105. Figure 5 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course, can also include other hardware required by a business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the pre-training method of the language model described in the above content.

[0115] Of course, in addition to the software implementation manner, the present specification does not exclude other implementation manners, such as a logic device or a combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or a logic device.

[0116] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0117] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0118] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] For the sake of description, the above apparatuses are described in various units with functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in implementing the present specification.

[0120] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0121] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product according to embodiments of the specification, it can be understood that each block in the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product according to embodiments of the specification, it can be understood that each block in the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks.

[0122] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product according to embodiments of the specification, it can be understood that each block in the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product according to embodiments of the specification, it can be understood that each block in the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks.

[0123] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product according to embodiments of the specification, it can be understood that each block in the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Although the flow diagrams and / or block diagrams can present a method, apparatus or computer program product according to embodiments of the specification, it can be understood that each block in the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks.

[0124] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0125] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) for example. Mass storage can include, in examples, EEPROM, flash memory or other non-volatile memory. The memory is an example of computer readable media.

[0126] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0127] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0128] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0129] The present specification can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0130] The various embodiments described in this specification are described using a numbering of embodiments approach: these are each individually integrated contributions pertaining to different aspects of the description. For each embodiment, the description focuses on the differences from the other embodiments. In particular, the description of the system embodiments is relatively brief, as the system embodiments are largely analogous to the method embodiments. The relevant parts of the description of the method embodiments are therefore referred to.

[0131] The above description is embodied in the form of embodiments only and is not intended to limit the present specification. The present specification can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. A method for pre-training a language model, the language model comprising an encoding end and a decoding end, the encoding end comprising a feature extraction network and a prompt network, the method comprising: obtaining a sample text; inputting the sample text into the feature extraction network to extract text features from the sample text by the feature extraction network; obtaining prompt information, inputting the prompt information into the prompt network to extract prompt features from the prompt information by the prompt network, wherein the prompt information is prompt text from actual tasks in different scenarios, and the feature extraction network extracts text features from the sample text under the influence of the prompt features; determining fusion features according to the text features and the prompt features; and pre-training the encoding end according to the fusion features and a training target of each preset training task. 2.The method of claim 1, wherein the language model further comprises a decoding end; and inputting the sample text into the feature extraction network comprises: processing the sample text by a mask and inputting the processed sample text into the feature extraction network; and wherein pre-training the encoding end according to the fusion features and the training target of each preset training task comprises: inputting the fusion features into the decoding end to obtain output text that restores the processed sample text; and training the encoding end to minimize a difference between the output text and the sample text. 3.The method of claim 1, wherein inputting the sample text into the feature extraction network comprises: determining a derivative text related to each sample text; inputting the sample text and the derivative text of the sample text as positive samples into the encoding end of the language model; inputting other sample texts except the sample text as negative samples into the encoding end of the language model; and wherein determining fusion features according to the text features and the prompt features comprises: determining fusion features corresponding to the positive samples based on the text features extracted from the positive samples and the prompt features; and determining fusion features corresponding to the negative samples based on the text features extracted from the negative samples and the prompt features; and wherein pre-training the encoding end according to the fusion features and the training target of each preset training task comprises: training the encoding end to maximize a similarity between the fusion features corresponding to the positive samples and minimize a similarity between the fusion features corresponding to the positive samples and the fusion features corresponding to the negative samples. 4.The method of claim 3, wherein determining a derivative text related to each sample text comprises: determining a parallel corpus text of the sample text, and taking the parallel corpus text as the derivative text of the sample text; or inserting a stop word into the sample text, and taking the sample text after inserting the stop word as the derivative text. 5.The method of claim 1, further comprising: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ determine a target business to which a pre-trained language model is to be applied; obtain business sample texts and labels corresponding to the business sample texts according to the target business; fine-tune the pre-trained language model according to the business sample texts and the labels corresponding to the business sample texts to obtain a trained language model.

6. The method of claim 5, wherein fine-tuning the pre-trained language model according to the business sample texts and the labels corresponding to the business sample texts specifically comprises: inputting the business sample texts into the language model to determine an output result; determining a loss according to the output result and the labels corresponding to the business sample texts; adjusting parameters of a prompt network of the language model according to the loss.

7. A device for pre-training a language model, the device being configured to train an encoding end of the language model, the encoding end comprising at least a feature extraction network and a prompt network, the device specifically comprising: a obtaining module configured to obtain sample texts; an extraction module configured to input the sample texts into the feature extraction network to extract text features from the sample texts by the feature extraction network, and obtain prompt information, input the prompt information into the prompt network to extract prompt features from the prompt information by the prompt network, the prompt information being prompt texts from actual tasks in different scenarios, wherein the feature extraction network extracts text features from the sample texts under the influence of the prompt features; a fusion module configured to determine fusion features according to the text features and the prompt features; a training module configured to pre-train the encoding end according to the fusion features and a training target of each preset training task.

8. The device of claim 7, wherein the language model further comprises a decoding end; the extraction module is specifically configured to process the sample texts by a mask and input the processed sample texts into the feature extraction network; the training module is specifically configured to input the fusion features into the decoding end to obtain output texts that restore the processed sample texts, and train the encoding end with the goal of minimizing the difference between the output texts and the sample texts.

9. The device of claim 7, wherein the extraction module is specifically configured to determine, for each sample text, a derivative text related to the sample text, input the sample text and the derivative text of the sample text as a positive sample into the encoding end of the language model, and input other sample texts except the sample text as negative samples into the encoding end of the language model; the fusion module is specifically configured to determine, based on the input positive sample, fusion features corresponding to the positive sample according to text features extracted from the positive sample and the prompt features, and determine, based on the input negative sample, fusion features corresponding to the negative sample according to text features extracted from the negative sample and the prompt features. The training module is specifically configured to train the encoding end with the maximum similarity between the fusion features corresponding to each positive sample and the minimum similarity between the fusion features corresponding to the positive sample and the fusion features corresponding to the negative sample as the target. 10.The apparatus of claim 9, wherein the extraction module is specifically configured to determine a parallel corpus text of the sample text, and take the parallel corpus text as the derived text of the sample text; or insert a stop word into the sample text, and take the sample text after the stop word is inserted as the derived text. 11.The apparatus of claim 7, wherein the training module is further configured to determine a target service to which a pre-trained language model is to be applied; obtain a service sample text and a label corresponding to the service sample text according to the target service; and fine-tune the pre-trained language model according to the service sample text and the label corresponding to the service sample text, to obtain a trained language model. 12.The apparatus of claim 11, wherein the training module is further configured to input the service sample text into the language model, and determine an output result; determine a loss according to the output result and the label corresponding to the service sample text; and adjust parameters of a prompt network of the language model according to the loss. 13.A computer readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the method of any one of claims 1-6. 14.An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1-6 when executing the program.

Images