Method, device, readable storage medium and equipment for performing language processing task

Abstract

The specification discloses a language processing task execution method, device, readable storage medium and equipment. According to the type of the language processing task, a target template parameter is determined. According to the target template parameter and a target mapping matrix for mapping different types of language processing tasks to different templates, a target template is determined. The target template and text information are combined and input into a pre-trained target language model to obtain an execution result of the text information. The method can use the target language model to execute multiple language processing tasks, has low cost and high utilization.

Description

Technical Field

[0001] This specification relates to the field of natural language processing technology, and in particular to a method, apparatus, readable storage medium, and device for performing a language processing task. Background Technology

[0002] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. For example, NLP techniques can be used to process various types of corpus data (such as speech data and text data).

[0003] In existing technologies, a common method for performing language processing tasks is based on language models. Specifically, the text information to be processed in natural language processing is first determined. Then, the text information is used as input to a pre-trained language model to obtain the language features of the text information. Finally, the natural language processing task is performed based on the determined language features.

[0004] However, natural language processing tasks include many types. Currently, when training language models, a large number of labeled samples corresponding to each type of language processing task are usually selected to train the language model. This makes the existing technology time-consuming, costly, and inefficient in training language models. Summary of the Invention

[0005] This specification provides a method, apparatus, readable storage medium, and device for performing language processing tasks, in order to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a method for performing a language processing task, including:

[0008] In response to a language processing task, determine the text information corresponding to the language processing task, and determine the target template parameters corresponding to the type of the language processing task.

[0009] Based on the target template parameters and the pre-determined target mapping matrix, the target template for the language processing task is determined, wherein the target mapping matrix is ​​used to map the template parameters corresponding to different types of language processing tasks to different templates;

[0010] The target template and the text information are combined, and the combination result is used as input to a pre-trained target language model to obtain the execution result of the text information.

[0011] Optionally, the target mapping matrix and the target template parameters corresponding to each type are determined in the following manner, wherein:

[0012] Obtain the model parameters of a specified language model that has been pre-trained in self-supervised training based on several unlabeled text information, and use them as the model parameters of the target language model.

[0013] Obtain the undetermined template parameters corresponding to each type, and determine the undetermined template corresponding to each type based on each undetermined template parameter and the undetermined mapping matrix;

[0014] Each undetermined template and each piece of text information are combined to determine each first training sample, and the corresponding annotations for each first training sample are determined.

[0015] The loss is determined based on each first training sample and its annotation. The undetermined mapping matrix and the undetermined template parameters corresponding to each type are adjusted, and the adjustment results are used as the target mapping matrix corresponding to the target language model and the target template parameters corresponding to each type.

[0016] Optionally, the method further includes:

[0017] Receive an adjustment request carrying the type of language processing task, and obtain the target template parameters corresponding to the type according to the type;

[0018] Based on the target template parameters and the target mapping matrix, determine the target template corresponding to the type;

[0019] The target template and each text information are combined to determine each second training sample, and the annotation corresponding to each second training sample is determined.

[0020] Based on the annotations of each of the second training samples, the parameters of the target template are readjusted.

[0021] Optionally, the method further includes:

[0022] The type of each language processing task corresponding to each of the first training samples is taken as the first type;

[0023] Receive an adjustment request carrying the type of language processing task, and determine whether the type belongs to each of the first types;

[0024] If not, obtain the target template parameters corresponding to the type, determine the target template corresponding to the type based on the target template parameters and the target mapping matrix, and combine each text information and the target template to determine each second training sample and its annotation, and adjust the target template parameters corresponding to the type based on the annotation of each second training sample.

[0025] Optionally, the target template for the language processing task is determined based on the target template parameters and a pre-determined target mapping matrix, specifically including:

[0026] Based on the target template parameters and the target mapping matrix, the feature vector corresponding to the language processing task is determined;

[0027] From the pre-determined manual annotations corresponding to each type, determine the manual annotation corresponding to the type;

[0028] Based on the manual annotations and the feature vectors, the target template for the language processing task is determined.

[0029] Optionally, the specified language model is trained in the following manner, wherein:

[0030] Obtain some text information and some masks;

[0031] For each mask, the mask and each piece of text information are combined to determine each third training sample and its annotation corresponding to the mask. The third training sample is the result after the text information is covered by the mask, and the annotation is the text information.

[0032] Based on each third training sample and its annotation, the loss is determined, and the model parameters of the specified language model are adjusted according to the loss.

[0033] Optionally, the target template parameters are readjusted based on the annotations of each of the second training samples, specifically including:

[0034] Each of the second training samples is used as input to the target language model to obtain the execution results corresponding to each of the second training samples output by the target language model.

[0035] An evolutionary strategy is used to adjust the target template parameters, and each second training sample and its execution result are determined iteratively based on the adjusted target template parameters until the execution result of each second training sample is consistent with the annotation.

[0036] This specification provides a language processing device, including:

[0037] The response module is used to respond to a language processing task by determining the template parameters corresponding to the language processing task and the text information corresponding to the language processing task based on the type of the language processing task.

[0038] The determining module is used to determine the template of the language processing task based on the template parameters and a pre-determined mapping matrix, wherein the mapping matrix is ​​used to map the template parameters corresponding to different language processing tasks to different templates;

[0039] The execution module is used to combine the template and the text information, and input the combination result into a pre-trained language model to obtain the execution result of the text information output by the language model.

[0040] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for performing language processing tasks.

[0041] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described language processing task.

[0042] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0043] In the language processing task execution method provided in this specification, the target template parameters are determined according to the type of language processing task. Then, based on the target template parameters and the target mapping matrix used to map the template parameters corresponding to different types of language processing tasks to different templates, the target template is determined. Finally, the target template and text information are combined and input into a pre-trained target language model to obtain the execution result of the text information.

[0044] As can be seen from the above method, the method can use the target language model to perform a variety of language processing tasks, with low cost and high utilization. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0046] Figure 1 A flowchart illustrating the execution method of the language processing tasks provided in this specification;

[0047] Figure 2 This is a flowchart illustrating the fine-tuning method for the pre-trained language model provided in this manual.

[0048] Figure 3 This is a flowchart illustrating the fine-tuning method for the pre-trained language model provided in this manual.

[0049] Figure 4 A schematic diagram of the execution device for the language processing task provided in this specification;

[0050] Figure 5 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0052] In the field of Natural Language Processing (NLP), with the development of technology, people are increasingly using NLP techniques to process various text information and perform corresponding tasks based on the processing results. A common method for performing NLP tasks is based on preprocessing language models. The training of preprocessing language models generally consists of three steps: pretraining, fine-tuning, and usage.

[0053] Specifically, the language model can be pre-trained using a large amount of unsupervised text information.

[0054] Then, based on the type of natural language processing task, a small amount of labeled text information is determined, and the model parameters are fine-tuned based on the determined labeled text information so that the model parameters can complete the corresponding natural language processing task. The natural language processing task can be a classification task, sequence labeling task, sentence relationship judgment task, or machine reading comprehension task, etc. The specific task type can be set as needed, and this specification does not impose any restrictions on it.

[0055] Finally, after the model is trained, the corresponding natural language processing task can be performed based on the trained language model.

[0056] In the process of pre-training a language model, a method of masking part of the text information is often used to determine the training samples.

[0057] Specifically, we can first determine several unlabeled text messages and several masks. The position and length of each mask can be set as needed.

[0058] Then, each text information and each mask are combined to determine each training sample. Each training sample is the text information after a portion of the text information has been masked using a mask, and the annotation for this training sample is the original text information.

[0059] Finally, the language model can be trained based on the identified training samples and their annotations.

[0060] It should be noted that since the above training process learns from the training samples, the learned content is the language of the training samples, rather than learning for the corresponding natural language processing task. A fine-tuning process is required afterward. Therefore, the above process is called the pre-training process, rather than the process of training an inaccurate language model.

[0061] After pre-training, the language model is generally considered to be able to learn language information from text. However, the learned language information is quite complex, and directly performing natural language processing tasks based on the learned language information may result in low accuracy. Therefore, fine-tuning the model is necessary.

[0062] During the fine-tuning of the model, two adjustment methods are typically provided to adjust the model parameters.

[0063] 1. Adjust based on network parameters

[0064] Specifically, labeled text information can be obtained and used as input to a pre-trained language model. The language model outputs the execution result of the natural language task corresponding to the text information. Based on the execution result and its labeling, the model parameters in the language model are adjusted.

[0065] 2. Adjustments based on template

[0066] Specifically, labeled text information can be obtained, and templates can be designed. The templates and text information are used as inputs to a pre-trained language model to obtain the execution results of the natural language task output by the language model. Based on the execution results and labels of each text information, the model parameters and templates can be adjusted.

[0067] However, in order to learn a wider range of language information, language models are typically trained using a large number of unsupervised samples during pre-training. This results in a complex language model structure with numerous parameters. Consequently, the fine-tuning stage requires adjusting many parameters, thus placing significant demands on computational resources during this phase.

[0068] Furthermore, currently, when training language models, due to the differences between natural language processing tasks, a typical approach is to assign one language model to one natural language processing task. However, the large number of model parameters makes transfer learning inefficient, which in turn leads to high costs in training language models.

[0069] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0070] Figure 1 The flowchart illustrating the execution method of the language processing task provided in this specification includes the following steps:

[0071] S100: In response to a language processing task, determine the text information corresponding to the language processing task, and determine the target template parameters corresponding to the type according to the type of the language processing task.

[0072] In one or more embodiments provided in this specification, the method for executing the language processing task provided in this specification can be specifically applied to a pre-trained language model in the template-based fine-tuning stage.

[0073] Unlike current methods that require training a specific language model for each type of language processing task and then using that model to perform natural language processing tasks, which results in high training costs, this approach offers a more efficient solution.

[0074] This specification provides a novel method for performing language processing tasks. By using a language model obtained by fine-tuning a template based on various types of natural language processing tasks during the fine-tuning phase, when a language processing task needs to be performed, the template parameters corresponding to the type of language processing task can be determined, and the language processing task can be performed based on the template parameters. Only one language model is needed to perform multiple language processing tasks, which improves execution efficiency and reduces the cost of model training.

[0075] Based on this, the type of natural language processing task can be determined.

[0076] In the field of natural language processing, the execution method for this language processing task is typically executed by a server. Similarly, the execution method for this language processing task in this specification can also be executed by a server. Furthermore, the server executing the execution method for this language processing task and the server training the language model can be the same server or different servers.

[0077] Specifically, in response to a language processing task, the server can determine the text information corresponding to that task. This language processing task can be initiated automatically by the server upon receiving an execution request carrying text information, or it can be initiated automatically based on preset conditions. For example, it can be initiated automatically when text information is detected being uploaded to the server. How to generate the language processing task and how to determine the text information to be executed can be configured as needed; this manual does not impose any restrictions on this.

[0078] Furthermore, since this specification requires generating templates based on template parameters corresponding to the type of language processing task, the server also needs to determine the target template parameters corresponding to this type from the pre-stored target template parameters corresponding to each type. These target template parameters can be in various forms, such as matrices or vectors, and the specific form of these template parameters can be set as needed; this specification does not impose any restrictions on this.

[0079] S102: Determine the target template for the language processing task based on the target template parameters and the pre-determined target mapping matrix, wherein the target mapping matrix is ​​used to map the template parameters corresponding to different types of language processing tasks to different templates.

[0080] In one or more embodiments provided in this specification, as described above, a single language model is used to execute multiple language processing tasks. The reason this specification can use a single model to handle multiple types of language processing tasks is that the language model used in this specification can execute language processing tasks based on templates corresponding to each type. The templates corresponding to each type can be determined based on template parameters corresponding to multiple types and a unique mapping matrix.

[0081] Based on this, the target template corresponding to this type can be determined based on the target template parameters determined in step S100 and the target mapping matrix determined by cosine.

[0082] Specifically, the server can multiply the target template parameters and the mapping matrix accordingly, and use the resulting matrix as the target template. This target mapping matrix is ​​used to map the template parameters corresponding to different types of language processing tasks to different templates.

[0083] Furthermore, since the execution result may be stored or displayed to the user after the execution result is determined, the template may also include manually labeled text information.

[0084] Specifically, the target template may include two parts: manual annotation and feature vector.

[0085] Therefore, the server can first determine the feature vector corresponding to the language processing task based on the target template parameters and the target mapping matrix.

[0086] Then, the server can determine the corresponding manual annotation for each type from the pre-determined manual annotations for each type.

[0087] Finally, the server can determine the target template for the language processing task based on the manual annotation and the feature vector.

[0088] In determining the target template, the server may convert manual annotations into annotation features, fuse the annotation features and feature vectors, and use the fused result as the target template. The specific method by which the server determines the target template based on manual annotations and feature vectors can be set as needed, and this manual does not impose any restrictions on this.

[0089] S104: Combine the target template and the text information, and use the combination result as input to the pre-trained target language model to obtain the execution result of the text information.

[0090] In one or more embodiments provided in this specification, when executing a target language model based on a template, the template typically contains a mask. Taking "The emotion of this sentence is [mask]" as an example, the [mask] is the mask part, which is also the content that the language model needs to fill in. Therefore, it is common practice to combine the template and text information, input the combination result into the language model, and let the language model fill in the mask part in the template, and use the filled content as the execution result.

[0091] Based on this, the server can combine templates and text information, and determine the execution result based on the combination result.

[0092] Specifically, the server can first determine the text features corresponding to the text information.

[0093] Then, the server can combine the text features and the features corresponding to the template to determine the combination result.

[0094] Finally, the combined result is used as input to a pre-trained target language model to obtain the execution result of the text information, namely, the mask part in the template.

[0095] It should be noted that the execution method for the language processing task provided in this manual can be applied to various scenarios, such as needing to continue making recommendations to users based on the execution results and conducting human-computer interaction with users. The specific use of the execution results can be set as needed, and this manual does not impose any restrictions on this.

[0096] based on Figure 1 This method for executing language processing tasks involves determining target template parameters based on the type of the language processing task. Then, based on these target template parameters and a target mapping matrix used to map template parameters corresponding to different types of language processing tasks to different templates, a target template is determined. Finally, the target template and text information are combined and input into a pre-trained target language model to obtain the execution result of the text information. This method can use the target language model to perform various language processing tasks, offering low cost and high utilization.

[0097] Following the same approach, this specification also provides a method for fine-tuning a pre-trained language model, such as... Figure 2 As shown.

[0098] Figure 2 The fine-tuning method for the pre-trained language model provided in this specification includes:

[0099] S200: Obtain the model parameters of a specified language model that has been pre-trained in self-supervised training based on several unlabeled text information, and use them as the model parameters of the target language model.

[0100] In one or more embodiments provided in this specification, the fine-tuning method for the pre-trained language model is applied in the stage of adjusting the template after the language model has been pre-trained. The fine-tuning stage requires adjusting the template based on the model parameters.

[0101] Based on this, the model parameters of the target language model can be determined first.

[0102] Specifically, the server can use the model parameters of a specified language model obtained through self-supervised training based on several unlabeled text information as the model parameters of the target language model. Alternatively, it can directly obtain the model parameters of an existing language model capable of performing language processing tasks and use them as the model parameters of the target language model. The specific method of determination can be configured as needed, and this manual does not impose any restrictions on it.

[0103] S202: Obtain the undetermined template parameters corresponding to each type, and determine the undetermined template corresponding to each type based on the undetermined template parameters and the undetermined mapping matrix.

[0104] Unlike current methods that require independent template adjustments for each type of language processing task, resulting in excessive computation when determining templates for multiple language processing tasks, this specification provides a fine-tuning method for pre-trained language models. This method allows for fine-tuning of template parameters for each language processing task, as well as mapping matrices that map different template parameters to different templates.

[0105] Based on this, the server can determine the template corresponding to each type based on the template parameters corresponding to each type and the preset mapping matrix.

[0106] Specifically, the server can obtain the corresponding pending template parameters for each type. These pending template parameters can be manually filled matrices, manually initialized matrices, or pending template parameters adjusted during the previous training, which can then be used as the pending template parameters for this training. The specific method for determining the pending template parameters can be set as needed, and this manual does not impose any restrictions on it.

[0107] Then, the server can determine the corresponding undetermined template for each type based on the undetermined template parameters and the undetermined mapping matrix. Similar to the undetermined template parameters, the undetermined mapping matrix can be a manually filled matrix, a manually initialized matrix, or a undetermined mapping matrix adjusted during the previous training. The specific method for determining the undetermined mapping matrix can be set as needed, and this manual does not impose any restrictions on it.

[0108] Furthermore, since different language processing task templates are needed during the fine-tuning phase, pre-annotated text information can be added to the templates to better distinguish between samples.

[0109] Specifically, the template to be determined may include two parts: manual annotation and feature vector.

[0110] Therefore, the server can first determine the features to be determined for each type based on the parameters of each template to be determined and the mapping matrix.

[0111] Then, the server can fuse the manual annotations corresponding to each type with the undetermined features corresponding to that type to determine each undetermined template.

[0112] For details on how to determine the template to be determined, please refer to the content on determining the target template in step S102. This instruction will not repeat it here.

[0113] S204: Combine each pending template and each text information to determine each first training sample, and determine the annotation corresponding to each first training sample.

[0114] S206: Determine the loss based on each first training sample and its annotation, adjust the undetermined mapping matrix and the undetermined template parameters corresponding to each type, and use the adjustment results as the target mapping matrix corresponding to the target language model and the target template parameters corresponding to each type.

[0115] In one or more embodiments provided in this specification, since the language model requires labeled samples to adjust the templates, the server can determine the training samples and their labels based on each pending template and each text information.

[0116] Specifically, the server can combine the pending templates and text information determined in step S202 to determine each first training sample.

[0117] Secondly, the server can determine the annotation for each first training sample. This annotation can be manually generated or determined by inputting corpus information into another language model. The specific method for determining the annotation can be configured as needed, and this specification does not impose any restrictions on it.

[0118] Then, the server can use each first training sample as input to the target language model to obtain the execution results corresponding to each first training sample output by the target language model.

[0119] Finally, the server can determine the loss based on the execution results and annotations corresponding to each first training sample, and adjust the undetermined mapping matrix and the undetermined template parameters corresponding to each type based on the loss. The adjustment results are then used as the target mapping matrix corresponding to the target language model and the target template parameters corresponding to each type, respectively.

[0120] Furthermore, if the mapping matrix is ​​considered as the commonalities among various types of language processing tasks, then the template parameters corresponding to each type can be regarded as the differences among the various types of language processing tasks. Based on the adjustment results of templates corresponding to multiple types, it is easier to learn the commonalities among the types, while the learning of differences may be somewhat weakened. Therefore, after obtaining the target mapping matrix and target template parameters, the server can further refine the target template parameters corresponding to each type.

[0121] Specifically, the server can receive adjustment requests carrying the type of language processing task. Based on this type, the server can obtain the target template parameters corresponding to that type. Specifically, if the type belongs to any of the types used to determine the first training sample, the target template parameters corresponding to that type are determined; otherwise, a manually preset matrix is ​​used as the target template parameters for that type.

[0122] Then, the server can determine the target template corresponding to this type based on the target template parameters and the target mapping matrix.

[0123] Finally, the server can combine the target template and each text information to determine each second training sample, determine the corresponding annotation for each second training sample, and readjust the target template parameters based on the annotations of each second training sample.

[0124] For details on how to adjust the target template parameters, please refer to the section on determining the target template parameters described above. Alternatively, the server can also use other algorithms (such as evolutionary strategies) to adjust the target model parameters.

[0125] Furthermore, the language model can be applied to perform other types of language processing tasks besides those used when determining the first training sample. For these other types of language processing tasks, it is necessary to determine their corresponding target template parameters. Therefore, the server can also determine the target template parameters only for language processing tasks other than the preset types.

[0126] Specifically, the server can take the type of each language processing task corresponding to each first training sample as each first type, and receive adjustment requests carrying the type of language processing task, and determine whether the type belongs to each first type.

[0127] If so, the server can return a prompt message based on the adjustment request, which indicates that the target template parameters have been adjusted.

[0128] If not, the server can obtain the target template parameters corresponding to the type, determine the target template corresponding to the type based on the target template parameters and the target mapping matrix, and combine each text information with the target template to determine each second training sample and its annotation, and adjust the target template parameters corresponding to the type based on the annotation of each second training sample.

[0129] Furthermore, as mentioned earlier, language models typically have a large number of model parameters. Therefore, adjusting the target template parameters based on these model parameters can be computationally intensive. In such cases, the server can employ an evolutionary strategy to adjust the model parameters.

[0130] Specifically, the server can take each second training sample as input and input it into the target language model to obtain the execution results corresponding to each second training sample of the target language output.

[0131] Then, the server can use an evolutionary strategy to adjust the target template parameters, and iteratively determine each second training sample and its execution result based on the adjusted target template parameters, until the execution result of each second training sample is consistent with the annotation.

[0132] Since the use of evolutionary strategies to adjust model parameters is already a relatively mature technology, this manual will not elaborate on it further.

[0133] It should be noted that the server for executing the language processing task method described above and the server for fine-tuning the pre-trained language model described above can be the same server or different servers. The language model and the determined target mapping matrix, target template parameters, etc., for executing the fine-tuning method of the pre-trained language model provided in this specification are applied in steps S100-S104.

[0134] Following the same approach, this specification also provides a method for fine-tuning a pre-trained language model, such as... Figure 3 As shown.

[0135] Figure 3 The diagram illustrates the fine-tuning process for a pre-trained language model. In the left rectangle, the target language model is deployed offline, allowing adjustments to the undetermined mapping matrix and template parameters based on the model parameters. The right rectangle shows the target language model deployed in the cloud, where model parameters are unavailable; in this case, adjustments to the target template parameters must be made based on an evolutionary strategy.

[0136] When determining and adjusting the undetermined mapping matrix and undetermined template parameters, the first training sample can be determined first. In the figure, when determining the first training sample, the natural language processing task types used are sentiment classification and intent classification. For each type, the server can obtain the undetermined template parameters corresponding to that type, where Z1 is the undetermined template parameter corresponding to the sentiment classification type, Z2 is the undetermined template parameter corresponding to the intent classification type, and A is the undetermined mapping matrix.

[0137] For each type, the server can determine the pending template corresponding to that type based on the pending template parameters and the pending mapping matrix. In the figure, the black vector P1 is the pending template corresponding to the sentiment classification type, and the white vector P2 is the pending template corresponding to the intent classification type.

[0138] After determining the templates to be determined, the server can combine each template to be determined with each text to determine the first training sample. Then, based on the first training sample, the parameters of the templates to be determined and the mapping matrix to be determined are adjusted to obtain the target template parameters corresponding to each type and the target mapping matrix corresponding to the target language model.

[0139] Then, the language model can be deployed in the cloud. When it is necessary to refine the target template parameters, determine the template parameters, or perform natural language processing tasks, samples are input into the language model through a preset input interface, and the execution results are output through a preset output interface. Specifically, when refining the target template parameters, the model can refine the template parameters of the type used when determining the first training sample, or it can refine the template parameters not used when determining the first training sample.

[0140] The figure illustrates examples of refining template parameters for sentiment classification and determining template parameters for news classification. Z3 represents the undetermined template parameters corresponding to the news classification type, and the gray vector P3 represents the undetermined template corresponding to the news classification type.

[0141] Specifically, the server can receive an adjustment request carrying the type of language processing task, obtain the target template parameters corresponding to the type, determine the target template based on the target template parameters and the target mapping matrix, combine the target template with each text information to determine each second training sample, and determine the annotation corresponding to each second training sample. Finally, the server readjusts the target template parameters based on the annotations of each second training sample.

[0142] Of course, the text information used when determining the first training sample and the second training sample can be the same text information or different text information.

[0143] It should be noted that all actions involving the acquisition of signals, information, or data in this manual are performed in accordance with the relevant data protection laws and regulations of the country where the device is located, and with the authorization of the owner of the relevant device.

[0144] The above describes one or more embodiments of a language processing task execution method provided in this specification. Based on the same idea, this specification also provides a corresponding language processing task execution device, such as... Figure 4 As shown

[0145] Figure 4 The execution device for the language processing task provided in this specification includes:

[0146] The response module 300 is used to respond to a language processing task, determine the text information corresponding to the language processing task, and determine the target template parameters corresponding to the type of the language processing task.

[0147] The determining module 302 is used to determine the target template of the language processing task based on the target template parameters and the pre-determined target mapping matrix, wherein the target mapping matrix is ​​used to map the template parameters corresponding to different types of language processing tasks to different templates.

[0148] The execution module 304 is used to combine the target template and the text information, and input the combination result into a pre-trained target language model to obtain the execution result of the text information.

[0149] The device further includes:

[0150] The adjustment module 306 is used to determine the target mapping matrix and the target template parameters corresponding to each type in the following manner: obtaining the model parameters of a specified language model pre-trained based on several unlabeled text information in a self-supervised manner, and using them as the model parameters of the target language model; obtaining the undetermined template parameters corresponding to each type; determining the undetermined templates corresponding to each type according to the undetermined template parameters and the undetermined mapping matrix; combining each undetermined template and each text information respectively to determine each first training sample, and determining the annotations corresponding to each first training sample respectively; determining the loss according to each first training sample and its annotation; adjusting the undetermined mapping matrix and the undetermined template parameters corresponding to each type respectively; and using the adjustment results as the target mapping matrix corresponding to the target language model and the target template parameters corresponding to each type respectively.

[0151] Optionally, the adjustment module 306 is configured to receive an adjustment request carrying the type of language processing task, and according to the type, obtain the target template parameters corresponding to the type, determine the target template corresponding to the type according to the target template parameters and the target mapping matrix, combine the target template and each text information to determine each second training sample, determine the annotation corresponding to each second training sample, and readjust the target template parameters according to the annotation of each second training sample.

[0152] Optionally, the adjustment module 306 is configured to take the type of each language processing task corresponding to each first training sample as each first type, receive an adjustment request carrying the type of the language processing task, and determine whether the type belongs to each first type. If not, it obtains the target template parameter corresponding to the type, determines the target template corresponding to the type based on the target template parameter and the target mapping matrix, and determines each second training sample and its annotation based on the combination of each text information and the target template, and adjusts the target template parameter corresponding to the type based on the annotation of each second training sample.

[0153] Optionally, the adjustment module 306 is used to acquire several text information and several masks, and for each mask, combine the mask and each text information to determine each third training sample and its label corresponding to the mask. The third training sample is the result after covering the text information with the mask, and the label is the text information. Based on each third training sample and its label, the loss is determined, and the model parameters of the specified language model are adjusted based on the loss.

[0154] Optionally, the adjustment module 306 is used to input each of the second training samples as input into the target language model, obtain the execution results corresponding to each of the second training samples output by the target language model, adjust the target template parameters using an evolutionary strategy, and iteratively determine each of the second training samples and their execution results based on the adjusted target template parameters until the execution results of each of the second training samples are consistent with the annotation.

[0155] Optionally, the determining module 302 is used to determine the feature vector corresponding to the language processing task based on the target template parameters and the target mapping matrix, determine the manual annotation corresponding to the type from the pre-determined manual annotations corresponding to each type, and determine the target template of the language processing task based on the manual annotations and the feature vector.

[0156] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method for executing language processing tasks.

[0157] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method for executing language processing tasks.

[0158] This instruction manual also provides Figure 5 The diagram shows a schematic structural representation of the electronic device. Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The method for executing the language processing task is described. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0159] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0160] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0161] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0162] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0163] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0168] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0169] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0171] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0173] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0174] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for performing a language processing task, characterized in that, include: In response to a language processing task, determine the text information corresponding to the language processing task, and determine the target template parameters corresponding to the type of the language processing task. Based on the target template parameters and the pre-determined target mapping matrix, the target template for the language processing task is determined, wherein the target mapping matrix is ​​used to map the template parameters corresponding to different types of language processing tasks to different templates; the target template and the text information are combined, and the combination result is used as input to the pre-trained target language model to obtain the execution result of the text information; The target mapping matrix and the target template parameters corresponding to each type are determined in the following manner: Model parameters of a specified language model pre-trained based on several unlabeled text information are obtained and used as the model parameters of the target language model; undetermined template parameters corresponding to each type are obtained, and undetermined templates corresponding to each type are determined according to the undetermined template parameters and the undetermined mapping matrix; each undetermined template and each piece of text information are combined to determine each first training sample, and the annotations corresponding to each first training sample are determined; the loss is determined according to each first training sample and its annotations, and the undetermined mapping matrix and the undetermined template parameters corresponding to each type are adjusted, and the adjustment results are used as the target mapping matrix corresponding to the target language model and the target template parameters corresponding to each type, respectively. The target template for the language processing task is determined based on the target template parameters and the pre-determined target mapping matrix. Specifically, this includes: determining the feature vector corresponding to the language processing task based on the target template parameters and the target mapping matrix; determining the manual annotation corresponding to each type from the pre-determined manual annotations corresponding to each type; and determining the target template for the language processing task based on the manual annotations and the feature vector.

2. The method as described in claim 1, characterized in that, The method further includes: receiving an adjustment request carrying a type of language processing task, and obtaining target template parameters corresponding to the type according to the type; determining a target template corresponding to the type according to the target template parameters and the target mapping matrix; combining the target template and each text information to determine each second training sample, and determining the annotation corresponding to each second training sample respectively; and readjusting the target template parameters according to the annotation of each second training sample.

3. The method as described in claim 1, characterized in that, The method further includes: taking the type of each language processing task corresponding to each first training sample as each first type; receiving an adjustment request carrying the type of the language processing task, and determining whether the type belongs to each first type; if not, obtaining the target template parameter corresponding to the type, determining the target template corresponding to the type according to the target template parameter and the target mapping matrix, and combining each text information and the target template to determine each second training sample and its annotation, and adjusting the target template parameter corresponding to the type according to the annotation of each second training sample.

4. The method as described in claim 1, characterized in that, The specified language model is trained in the following manner: acquiring several text information and several masks; for each mask, combining the mask and each text information to determine each third training sample and its annotation corresponding to the mask, wherein the third training sample is the result of covering the text information with the mask, and the annotation is the text information; determining the loss based on each third training sample and its annotation, and adjusting the model parameters of the specified language model based on the loss.

5. The method as described in claim 2 or 3, characterized in that, Based on the annotations of each second training sample, the target template parameters are readjusted, specifically including: taking each second training sample as input and inputting it into the target language model to obtain the execution results corresponding to each second training sample output by the target language model; adjusting the target template parameters using an evolutionary strategy, and iteratively determining each second training sample and its execution result based on the adjusted target template parameters, until the execution result of each second training sample is consistent with the annotation.

6. A language processing device, characterized in that, The device includes: a response module, configured to respond to a language processing task and determine template parameters corresponding to the language processing task and text information corresponding to the language processing task based on the type of the language processing task; a determination module, configured to determine a template for the language processing task based on the template parameters and a pre-determined mapping matrix, wherein the mapping matrix is ​​used to map template parameters corresponding to different language processing tasks to different templates; and an execution module, configured to combine the template and the text information, and input the combination result as input to a pre-trained language model to obtain an execution result of the text information output by the language model. The target mapping matrix and target template parameters corresponding to each type are determined in the following manner: Model parameters of a specified language model pre-trained based on several unlabeled text information are obtained and used as model parameters of the target language model; undetermined template parameters corresponding to each type are obtained, and undetermined templates corresponding to each type are determined according to the undetermined template parameters and the undetermined mapping matrix; each undetermined template and each text information are combined to determine each first training sample, and the corresponding annotations for each first training sample are determined; the loss is determined according to each first training sample and its annotations, and the undetermined mapping matrix and the undetermined template parameters corresponding to each type are adjusted, with the adjustment results used as the target mapping matrix corresponding to the target language model and the target template parameters corresponding to each type. The target template for the language processing task is determined based on the target template parameters and the pre-determined target mapping matrix. Specifically, this includes: determining the feature vector corresponding to the language processing task based on the target template parameters and the target mapping matrix; determining the manual annotation corresponding to each type from the pre-determined manual annotations corresponding to each type; and determining the target template for the language processing task based on the manual annotations and the feature vector.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 5.

Images