Training method of natural language processing model, natural language processing method and device

Abstract

The embodiment of the present specification provides a natural language processing model training method and device, and a natural language processing method and device. The training method comprises: generating an original sample set by using an original data set of the natural language; generating an anchor data set by using the original data set of the natural language; generating an adversarial sample set by using the original sample set of the natural language; inputting the anchor data set and the adversarial sample set into a pre-training model respectively, calculating the loss of the pre-training model, adjusting the parameters of the pre-training model until the pre-training model converges, and taking the converged model as the natural language processing model. The embodiment of the present specification can ensure the recognition effect of the natural language processing model.

Description

Technical Field

[0001] This specification relates to electronic information technology in one or more embodiments, and more particularly to training methods and apparatus for natural language processing models, and natural language processing methods and apparatus. Background Technology

[0002] In recent years, deep learning based on deep neural networks has achieved great success in the field of natural language processing. However, the robustness of these models still has significant shortcomings. For example, in sentiment classification tasks, replacing certain words in the original text with synonyms can lead to large misclassifications in deep neural networks, resulting in incorrect sentiment classification results.

[0003] The most widely used method to improve the robustness of natural language processing models is adversarial training. However, current adversarial training methods typically treat the natural language processing model as a completely new model, train it from scratch using adversarial examples, and modify all of the model's parameters.

[0004] Considering the characteristics of natural language, such as how slight changes in semantics can lead to huge differences in recognition results, such as sentiment classification results, the current practice of using adversarial examples to train the model from scratch and modify all the parameters of the model will result in a significant reduction in the recognition performance of the trained model on normal data. Summary of the Invention

[0005] This specification describes one or more embodiments of a training method and apparatus for a natural language processing model, as well as a natural language processing method and apparatus, which can guarantee the recognition performance of the natural language processing model.

[0006] According to the first aspect, a method for training a natural language processing model is provided, wherein the method includes:

[0007] The raw dataset is obtained from historical business data of natural language; this raw dataset includes multiple raw data sets of natural language.

[0008] The original sample set is generated using the original dataset of the natural language; the original sample set includes multiple original data of the natural language and the label corresponding to each original data.

[0009] Anchor point datasets are generated using raw natural language datasets;

[0010] An adversarial sample set is generated using the original sample set of natural language; the adversarial sample set includes multiple adversarial samples obtained by adversarially perturbing multiple original data in the original sample set, as well as the label of each adversarial sample.

[0011] The anchor dataset and the adversarial example set are respectively input into the pre-trained model, the loss of the pre-trained model is calculated, and the parameters of the pre-trained model are adjusted until the pre-trained model converges. The converged model is then used as the natural language processing model.

[0012] The step of generating the anchor dataset using the raw dataset of natural language includes:

[0013] Randomly sample from the original dataset of natural language, and use each sampled original data as anchor data to form an anchor dataset.

[0014] The step of inputting the anchor dataset and the adversarial example set into the pre-trained model and calculating the loss of the pre-trained model includes:

[0015] The anchor dataset is input into the currently obtained pre-trained model to obtain the prediction result for each anchor data point. The prediction result for each anchor data point is used as the pseudo label for that anchor data point. The loss of the pre-trained model is calculated based on each anchor data point and its pseudo label.

[0016] The set of adversarial examples is input into the currently obtained pre-trained model, and the loss of the pre-trained model is calculated based on each adversarial example and its label.

[0017] The method further includes setting the loss function of the pre-trained model to include: cross-entropy adversarial loss and original anchor loss;

[0018] The step of calculating the loss of the pre-trained model based on each anchor data and its pseudo-label includes: calculating the cross-entropy adversarial loss of the pre-trained model based on each anchor data and its pseudo-label.

[0019] The step of calculating the loss of the pre-trained model based on each adversarial sample and its label includes: calculating the original anchor loss of the pre-trained model based on each adversarial sample and its label.

[0020] The loss function of the pre-trained model further includes log loss.

[0021] The calculation of the cross-entropy adversarial loss of the pre-trained model includes: when calculating the cross-entropy adversarial loss, inputting the union of the adversarial sample set and the original sample set into the currently obtained pre-trained model, and using each anchor data and its pseudo-label, and each original data and its label to calculate the cross-entropy adversarial loss.

[0022] According to the second aspect, a natural language processing method is provided, wherein the method includes:

[0023] Obtain the natural language corpus to be identified;

[0024] The natural language corpus to be identified is input into a pre-obtained natural language processing model;

[0025] The recognition results of the natural language corpus to be recognized are obtained from the output of the natural language processing model.

[0026] The natural language processing model is used in the task of sentiment classification of natural language.

[0027] The recognition result is an emotion classification result.

[0028] According to the third aspect, a training device for a natural language processing model is provided, wherein the device includes:

[0029] The raw dataset acquisition module is configured to obtain a raw dataset from historical business data in natural language; this raw dataset includes multiple raw data sets in natural language.

[0030] The original sample set acquisition module is configured to generate an original sample set using the original dataset of the natural language; wherein, the original sample set includes multiple original data of the natural language and the label of each original data;

[0031] The anchor dataset generation module is configured to generate anchor datasets using raw natural language datasets.

[0032] The adversarial sample set generation module is configured to generate an adversarial sample set using the original sample set of natural language; wherein, the adversarial sample set includes multiple adversarial samples obtained by adversarially perturbing multiple original data in the original sample set, and a label for each adversarial sample.

[0033] The training execution module is configured to input the anchor dataset and the adversarial example set into the pre-trained model, calculate the loss of the pre-trained model, and adjust the parameters of the pre-trained model until the pre-trained model converges, and use the converged model as the natural language processing model.

[0034] According to the fourth aspect, a natural language processing apparatus is provided, wherein the apparatus includes:

[0035] The input module for the corpus to be recognized is configured to obtain the natural language corpus to be recognized;

[0036] The recognition result acquisition module is configured to input the natural language corpus to be recognized into a pre-obtained natural language processing model; and obtain the recognition result of the natural language corpus to be recognized output by the natural language processing model.

[0037] According to a fifth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in any embodiment of this specification.

[0038] The training methods and apparatus for natural language processing models, and the methods and apparatus for natural language processing provided in the various embodiments of this specification, individually or in combination, have at least the following beneficial effects:

[0039] 1. Considering the characteristics of natural language, such as how slight changes in semantics can lead to huge differences in recognition results, such as sentiment classification results, the embodiments in this specification perform various reasonable transformations on the natural language dataset when using the pre-trained model. The pre-trained model is adjusted by transforming the original dataset into various datasets (including anchor datasets and adversarial example sets), so that the final trained model can improve robustness without affecting the recognition performance on the original data.

[0040] 2. In the training method of the natural language processing model proposed in the embodiments of this specification, a completely new model is not trained from scratch. Instead, an adjustment is made on the basis of an existing pre-trained model to obtain the natural language processing model. Therefore, it can reduce the consumption of resources, such as eliminating the need to crawl massive amounts of training data from the network or to perform massive training rounds, such as hundreds of thousands of rounds.

[0041] 3. The method in the embodiments of this specification uses the adversarial sample set obtained by adversarial perturbation as data augmentation to adjust the pre-trained model, thereby improving the robustness of the model. On the other hand, in order to solve the problem of catastrophic forgetting caused by the adjustment of the pre-trained model, that is, the decline in the model's performance on normal original data, the anchor points of the original data are also used to constrain the model's weights, so that the final trained model can improve its robustness without affecting its recognition performance on the original data.

[0042] 4. In the embodiments of this specification, by setting the loss function of the model to include the logarithmic function corresponding to the hyperparameters, the weights of the model are learnable during the training process, rather than being fixed. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1This is a schematic diagram of the system architecture used in one embodiment of this specification.

[0045] Figure 2 This is a flowchart of a training method for a natural language processing model in one embodiment of this specification.

[0046] Figure 3 This is a schematic diagram of a training method for a natural language processing model in one embodiment of this specification.

[0047] Figure 4 This is a flowchart of a natural language processing method in one embodiment of this specification.

[0048] Figure 5 This is a schematic diagram of the structure of a training device for a natural language processing model in one embodiment of this specification.

[0049] Figure 6 This is a schematic diagram of the structure of a natural language processing device in one embodiment of this specification. Detailed Implementation

[0050] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0051] First, it should be noted that the terminology used in the embodiments of this invention is for the purpose of describing specific embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0052] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0053] To facilitate understanding of the methods provided in this manual, the system architecture involved and applicable to this manual is first described. For example... Figure 1 As shown, the system architecture mainly includes three types of network nodes: web crawler module, data processing module, and training execution module.

[0054] The web crawler module is used to obtain the required raw natural language data from the web, such as the natural language input by users when contacting customer service on a webpage, or the text of a novel. The data processing module is used to generate various datasets for training the natural language processing model from the raw natural language data, including: the original dataset, the original sample set, the anchor dataset, and the adversarial example set. The training execution module is used to train the natural language processing model using the anchor dataset and adversarial example set generated by the data processing module.

[0055] It should be understood that Figure 1 The number of web crawler modules, data processing modules, and training execution modules shown is merely illustrative. Any number can be selected and deployed according to implementation needs.

[0056] Figure 2 This is a flowchart of a training method for a natural language processing model in one embodiment of this specification. The main entity executing this method is the training device for the natural language processing model. It is understood that this method can also be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. See also... Figure 2 The method includes:

[0057] Step 201: Obtain the raw dataset from the historical business data of natural language; the raw dataset includes multiple raw data of natural language.

[0058] Step 203: Generate an original sample set using the original dataset of the natural language; wherein the original sample set includes multiple original data of the natural language and the label corresponding to each original data.

[0059] Step 205: Generate an anchor dataset using the original dataset of natural language; wherein the anchor dataset includes multiple anchor data obtained by random sampling from the original dataset of natural language.

[0060] Step 207: Generate an adversarial sample set using the original sample set of natural language; wherein, the adversarial sample set includes multiple adversarial samples obtained by adversarially perturbing multiple original data in the original sample set, and the label of each adversarial sample.

[0061] Step 209: Input the anchor dataset and the adversarial example set into the pre-trained model respectively, calculate the loss of the pre-trained model, and adjust the parameters of the pre-trained model until the pre-trained model converges. Use the converged model as the natural language processing model.

[0062] See Figure 2 , Figure 3Considering the characteristics of natural language, such as how slight semantic changes can lead to significant differences in recognition results, such as sentiment classification results, the embodiments of this specification perform various transformations on the original natural language dataset for various purposes. The pre-trained model is adjusted using the transformed datasets (including anchor datasets and adversarial example sets) to achieve the training objective. Specifically, the method of this specification, on the one hand, uses the adversarial example set obtained through adversarial perturbation as data augmentation to adjust the pre-trained model, thereby improving the model's robustness. On the other hand, to address the catastrophic forgetting that results from adjusting the pre-trained model—that is, the model's performance on normal original data declines—the original anchors are used to constrain the model's weights. This ensures that the final trained model improves robustness without affecting its recognition performance on the original data.

[0063] In the training method of the natural language processing model proposed in the embodiments of this specification, instead of training a completely new model from scratch, the natural language processing model is obtained by adjusting an existing pre-trained model. Therefore, it can reduce the consumption of resources, such as eliminating the need to crawl massive amounts of training data from the network or to perform massive training rounds, such as hundreds of thousands of rounds.

[0064] The following describes specific embodiments and... Figure 3 ,right Figure 2 The process shown is explained below.

[0065] First, for step 201: obtain the raw dataset from the historical business data of natural language; the raw dataset includes multiple raw data of natural language.

[0066] In step 201, the above-mentioned methods can be used. Figure 1 The web crawler module shown obtains raw datasets by acquiring natural language corpora from various web pages. For example, natural language input by users when contacting customer service, or the text of a novel, can all serve as raw data.

[0067] Next, for step 203: generate an original sample set using the original dataset of natural language; wherein, the original sample set includes multiple original data of natural language and the label corresponding to each original data.

[0068] In step 203, each raw data point in the original dataset is labeled, thus generating the original sample set. In the sentiment classification task, the labels can be sentiment-related labels, such as angry, happy, nervous, sad, etc.

[0069] Next, for step 205: generate an anchor dataset using the original dataset of natural language; wherein, the anchor dataset includes: multiple anchor data obtained by randomly sampling from the original dataset of natural language.

[0070] In step 205, random sampling can be performed on the original natural language dataset. Each sampled original data point is used as anchor data, and then the anchor data points are combined to form an anchor dataset. For example, if the original dataset contains 1,000 original data points, random sampling can be performed on these 1,000 original data points to select 500 original data points. These 500 sampled original data points are the 500 anchor data points, and the resulting anchor dataset includes these 500 anchor data points.

[0071] Anchor data is essentially raw data and possesses the characteristics of raw data. Therefore, the purpose of obtaining anchor datasets is to fix the model's recognition performance on raw data and avoid catastrophic forgetting caused by fine-tuning made to improve model robustness.

[0072] Next, for step 207: generate an adversarial sample set using the original sample set of natural language; wherein, the adversarial sample set includes multiple adversarial samples obtained by adversarially perturbing multiple original data in the original sample set, and the label of each adversarial sample.

[0073] The purpose of generating adversarial example sets is to train the model adversarially, improve the model's recognition performance on perturbed data, and thus enhance the model's robustness.

[0074] In the embodiments of this specification, for example, the original dataset includes 1,000 original data points. Adversarial perturbations, such as synonym replacement and / or addition / deletion of modifiers, can be performed on 500 of these original data points to obtain 500 adversarial samples. Each adversarial sample has a label, and the label of an adversarial sample is the label of the original data point that the adversarial sample perturbs. In this way, the 500 adversarial samples and their labels form an adversarial sample set.

[0075] In step 207, the requirement for adversarial perturbation is that the adversarial sample generated after perturbation should maintain semantic consistency with the original data, that is, the added perturbation should not change the semantics of the original sentence as much as possible.

[0076] Next, for step 209: input the anchor dataset and the adversarial example set into the pre-trained model respectively, calculate the loss of the pre-trained model, and adjust the parameters of the pre-trained model until the pre-trained model converges, and use the converged model as the natural language processing model.

[0077] In step 209, the parameters of the pre-trained model are adjusted based on the calculated loss of the pre-trained model.

[0078] Using a pre-trained model involves training the model on a pre-set batch of data, and then further training or repurposing the model based on this pre-trained model. In one embodiment of this specification, all or some weights of a pre-trained model already used for other objectives are used; that is, the pre-trained model is used as the initial values ​​to begin training.

[0079] As mentioned earlier, since the adversarial sample set obtained from the adversarial perturbation and the anchor point dataset are used simultaneously in step 209 to continue training the pre-trained model, the final trained model can improve its robustness without affecting its recognition performance on the user's normal original data.

[0080] In one embodiment of this specification, the specific implementation process of step 209 includes:

[0081] Step 2091: Input the anchor dataset into the currently obtained pre-trained model so that the pre-trained model can obtain the prediction result for each anchor data, use the prediction result for each anchor data as the pseudo label for that anchor data, and calculate the loss of the pre-trained model based on each anchor data and its pseudo label.

[0082] In step 2091, the anchor data in the anchor dataset is essentially the original data. However, instead of using the existing labels corresponding to the original data represented by the anchor data to calculate the loss of the pre-trained model, the prediction results obtained by the pre-trained model to predict the anchor data are used to calculate the loss of the pre-trained model. In other words, the output of the pre-trained model is used as the pseudo-label of the anchor data, and the pseudo-label is used as a constraint on the anchor data, thereby fixing the recognition ability of the obtained pre-trained model on normal original data.

[0083] Step 2093: Input the adversarial sample set into the currently obtained pre-trained model, and use each adversarial sample and its label in the adversarial sample set to calculate the loss of the pre-trained model.

[0084] In one embodiment of this specification, to better train the model and ensure that the final trained model improves robustness without affecting its recognition performance on the original data, the loss function of the pre-trained model can be set to include: cross-entropy adversarial loss and original anchor loss before obtaining the pre-trained model and executing step 209. The cross-entropy adversarial loss aims to improve model robustness. The original anchor loss aims to maintain the model's performance on the original dataset. Accordingly, in step 2091, the process of calculating the loss of the pre-trained model based on each anchor data point and its pseudo-label includes: calculating the cross-entropy adversarial loss of the pre-trained model based on each anchor data point and its pseudo-label; in step 2093, the process of calculating the loss of the pre-trained model based on each adversarial sample and its label includes: calculating the original anchor loss of the pre-trained model based on each adversarial sample and its label.

[0085] In one embodiment of this specification, the loss function of the pre-trained model includes three parts: cross-entropy adversarial loss, original anchor loss, and log loss. The role of log loss is to balance the cross-entropy adversarial loss and the original anchor loss, making them equally important.

[0086] In one embodiment of this specification, when calculating the cross-entropy adversarial loss, the adversarial sample set can be input into the currently obtained pre-trained model to calculate the cross-entropy adversarial loss; alternatively, the union of the adversarial sample set and the original sample set can be input into the currently obtained pre-trained model to calculate the cross-entropy adversarial loss. Specifically, in one embodiment of this specification, if the loss function of the pre-trained model includes three parts: cross-entropy adversarial loss, original anchor loss, and logarithmic loss, the expression of the loss function can be:

[0087] L=α1·L CE (D∪D adv ;θ+δ)+α2·L MSE ({x, f) θ (x)} (x，y)∈D ;θ+δ)-log(α)

[0088] Where L represents the loss function; α1 and α2 are hyperparameters, where α1 represents the weight of the cross-entropy adversarial loss, and α2 represents the weight of the original anchor loss. CE Characterizing cross-entropy adversarial loss, L MSE The original anchor loss is represented by D, and the original sample set is represented by D. adv The set of adversarial examples is represented by the operator "∪", which represents the union of the sets. θ+δ represents the updated parameters of the model obtained during training. x represents each anchor data point in the anchor dataset, y is the label of the original data represented by the anchor data, and f...θ (x) represents the prediction made by the pre-trained model on the anchor data, θ represents the initial model parameters of the pre-trained model, and α represents the two-dimensional array composed of α1 and α2.

[0089] In the above expression, -log(α) is used to balance the two weights α1 and α2, which can be seen as -log(α) = -log(α1) - log(α2). This makes the model's weights learnable during training, rather than fixed. For example, the training task corresponding to the first term in the expression is relatively simple, while the training task corresponding to the second term is more difficult. The use of -log(α) makes the training process learnable, enabling dynamic adjustment.

[0090] In one embodiment of this specification, the following relationship can be satisfied:

[0091] α1·L cE (D∪D adv ;θ+δ)=1

[0092] α2·L MSE ({x, f) θ (x)} (x，y)∈D ;θ+δ)=1

[0093] As can be seen, the method described in the embodiments of this specification proposes an anchor-based adversarial weight fine-tuning process. This is a lightweight fine-tuning scheme designed to rapidly improve the robustness of a pre-trained model while maintaining its original performance. Specifically, based on an existing pre-trained model, random adversarial perturbations are used for data augmentation, allowing the model to be trained simultaneously on both the original data (specifically, the anchor data) and adversarial data. Then, for the adversarial sample data, labeled cross-entropy loss is used for training; for the anchor data representing the original data, the output of the pre-trained model is used as the anchor, and mean squared error loss is used to keep the output of the robust model unchanged.

[0094] At this point, a natural language processing model has been trained, which can then be used to process natural language, such as sentiment classification.

[0095] Figure 4 This is a flowchart of a natural language processing method in one embodiment of this specification. See also... Figure 4 The execution entity of this method is a natural language processing device. It can be understood that this method can also be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. See also... Figure 4 The method includes:

[0096] Step 401: Obtain the natural language corpus to be identified.

[0097] Step 403: Input the natural language corpus to be identified into a pre-obtained natural language processing model; wherein the natural language processing model is obtained using the training method in any embodiment of this specification;

[0098] Step 405: Obtain the recognition result of the natural language corpus to be recognized, output by the natural language processing model.

[0099] In one embodiment of this specification, a natural language processing model is used in a sentiment classification task of natural language; correspondingly, the recognition result is a sentiment classification result.

[0100] This specification also provides an embodiment of a training device for a natural language processing model, see [link to embodiment]. Figure 5 The device includes:

[0101] The raw dataset acquisition module 501 is configured to obtain the raw dataset from historical business data of natural language; the raw dataset includes multiple raw data of natural language.

[0102] The original sample set acquisition module 502 is configured to generate an original sample set using the original dataset of the natural language; wherein, the original sample set includes multiple original data of the natural language and a label for each of the original data.

[0103] Anchor dataset generation module 503 is configured to generate anchor datasets using raw natural language datasets;

[0104] The adversarial sample set generation module 504 is configured to generate an adversarial sample set using the original sample set of natural language; wherein, the adversarial sample set includes multiple adversarial samples obtained by adversarially perturbing multiple original data in the original sample set, and a label for each adversarial sample.

[0105] The training execution module 505 is configured to input the anchor dataset and the adversarial example set into the pre-trained model respectively, calculate the loss of the pre-trained model, and adjust the parameters of the pre-trained model until the pre-trained model converges, and use the converged model as the natural language processing model.

[0106] exist Figure 5 In one embodiment of the apparatus shown in this specification, the anchor dataset generation module 503 is configured to perform: random sampling from the raw dataset of natural language, using each sampled raw data as anchor data, and forming an anchor dataset from the anchor data.

[0107] exist Figure 5 In one embodiment of the apparatus shown in this specification, the training execution module 505 is configured to execute:

[0108] The anchor dataset is input into the currently obtained pre-trained model so that the pre-trained model can obtain the prediction result for each anchor data. The prediction result for each anchor data is used as the pseudo label for that anchor data. The loss of the pre-trained model is calculated based on each anchor data and its pseudo label.

[0109] The set of adversarial examples is input into the currently obtained pre-trained model, and the loss of the pre-trained model is calculated based on each adversarial example and its label.

[0110] exist Figure 5 In one embodiment of the device shown in this specification, the loss function of the pre-trained model includes: cross-entropy adversarial loss and original anchor loss;

[0111] Accordingly, the training execution module 505 is configured to perform: calculating the cross-entropy adversarial loss of the pre-trained model based on each anchor data and its pseudo-label; and calculating the original anchor loss of the pre-trained model based on each adversarial sample and its label.

[0112] exist Figure 5 In one embodiment of the apparatus shown in this specification, the training execution module 505 is configured to: when calculating the cross-entropy adversarial loss, input the union of the adversarial sample set and the original sample set into the currently obtained pre-trained model, and use each anchor data and its pseudo-label, each original data and its label to calculate the cross-entropy adversarial loss.

[0113] This specification also provides an embodiment of a natural language processing apparatus, see [link to embodiment]. Figure 6 The device includes:

[0114] The input module 601 for the corpus to be recognized is configured to obtain the natural language corpus to be recognized;

[0115] The recognition result acquisition module 602 is configured to input the natural language corpus to be recognized into a pre-obtained natural language processing model; wherein, the natural language processing model is obtained by the training device of the natural language processing model in this embodiment; and obtain the recognition result of the natural language corpus to be recognized output by the natural language processing model.

[0116] It should be noted that the above-mentioned devices are typically implemented on the server side. They can be set up on independent servers, or some or all of the devices can be combined and installed on the same server. This server can be a single server or a server cluster consisting of multiple servers. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. The above-mentioned devices can also be implemented on computer terminals with strong computing capabilities.

[0117] This specification provides, in one embodiment, a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods of any embodiment in the specification.

[0118] This specification provides a computing device according to one embodiment, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to perform the method of any embodiment of the specification.

[0119] It is understood that the structures illustrated in the embodiments of this specification do not constitute a specific limitation on the apparatus of the embodiments of this specification. In other embodiments of the specification, the above-described apparatus may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0120] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0121] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, widgets, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0122] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. Training methods for natural language processing models, among which, The method includes: The raw dataset is obtained from historical business data of natural language; the raw dataset includes multiple raw data of natural language; the natural language includes: natural language corpus or text; The original dataset of the natural language is used to generate an original sample set; wherein the original sample set includes multiple original data of the natural language and the label corresponding to each original data. Anchor datasets are generated using the original dataset of natural language processing; the anchor datasets include a portion of the original data from the original dataset. An adversarial sample set is generated using the original sample set of natural language. The adversarial sample set includes multiple adversarial samples obtained by adversarially perturbing multiple original data in the original sample set, and a label for each adversarial sample. The label of an adversarial sample is the label of the original data that was perturbed by the adversarial sample. The training samples are: training sample pairs consisting of adversarial samples and their labels, and training sample pairs consisting of anchor data and pseudo-labels; the anchor dataset and the adversarial sample set are respectively input into the pre-trained model, the loss of the pre-trained model is calculated, and the parameters of the pre-trained model are adjusted until the pre-trained model converges, and the converged model is used as the natural language processing model. The step of inputting the anchor dataset and the adversarial example set into the pre-trained model and calculating the loss of the pre-trained model includes: The anchor dataset is input into the currently obtained pre-trained model to obtain the prediction result for each anchor data point. The prediction result for each anchor data point is used as the pseudo label for that anchor data point. The loss of the pre-trained model is calculated based on each anchor data point and its pseudo label. The set of adversarial examples is input into the currently obtained pre-trained model, and the loss of the pre-trained model is calculated based on each adversarial example and its label. The loss function of the pre-trained model includes: cross-entropy adversarial loss, original anchor loss, and a logarithmic term; wherein, the logarithmic term is configured such that the weights of the cross-entropy adversarial loss and the original anchor loss are learnable during training, thereby balancing the weights of the cross-entropy adversarial loss and the original anchor loss and achieving dynamic adjustment.

2. The method according to claim 1, wherein, The process of generating anchor datasets using raw natural language datasets includes: Randomly sample from the original dataset of natural language, and use each sampled original data as anchor data to form an anchor dataset.

3. Natural language processing methods, among which, The method includes: Obtain the natural language corpus to be identified; The natural language corpus to be identified is input into a pre-obtained natural language processing model; wherein the natural language processing model is trained using the method described in any one of claims 1 to 2; The recognition results of the natural language corpus to be recognized are obtained from the output of the natural language processing model.

4. The method according to claim 3, wherein, The natural language processing model is used in the task of sentiment classification of natural language. The recognition result is an emotion classification result.

5. A training device for a natural language processing model, wherein, The device includes: The raw dataset acquisition module is configured to obtain a raw dataset from historical business data of natural language; the raw dataset includes multiple raw data of natural language; the natural language includes: natural language corpus or text; The original sample set acquisition module is configured to generate an original sample set using the original dataset of the natural language; wherein, the original sample set includes multiple original data of the natural language and the label of each original data; The anchor dataset generation module is configured to generate an anchor dataset using the original natural language dataset; wherein the anchor dataset includes a portion of the original data from the original dataset. The adversarial example set generation module is configured to generate an adversarial example set using the original sample set of natural language. The adversarial example set includes multiple adversarial examples obtained by adversarially perturbing multiple original data in the original sample set, and a label for each adversarial example. The label of an adversarial example is the label of the original data that was perturbed by the adversarial example. The training samples are: training sample pairs consisting of adversarial examples and their labels, and training sample pairs consisting of anchor data and pseudo-labels. The training execution module is configured to input the anchor dataset and the adversarial example set into the pre-trained model respectively, calculate the loss of the pre-trained model, and adjust the parameters of the pre-trained model until the pre-trained model converges, and use the converged model as the natural language processing model. The training execution module is configured to execute: The anchor dataset is input into the currently obtained pre-trained model so that the pre-trained model can obtain the prediction result for each anchor data point. The prediction result for each anchor data point is used as the pseudo label for that anchor data point. The loss of the pre-trained model is calculated based on each anchor data point and its pseudo label. The adversarial sample set is input into the currently obtained pre-trained model, and the loss of the pre-trained model is calculated based on each adversarial sample and its label. The loss function of the pre-trained model includes: cross-entropy adversarial loss, original anchor loss, and a logarithmic term; wherein, the logarithmic term is configured such that the weights of the cross-entropy adversarial loss and the original anchor loss are learnable during training, thereby balancing the weights of the cross-entropy adversarial loss and the original anchor loss and achieving dynamic adjustment.

6. A natural language processing device, wherein, The device includes: The input module for the corpus to be recognized is configured to obtain the natural language corpus to be recognized; The recognition result acquisition module is configured to input the natural language corpus to be recognized into a pre-obtained natural language processing model; wherein the natural language processing model is obtained using the device described in claim 5; and to obtain the recognition result of the natural language corpus to be recognized output by the natural language processing model.

7. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-4.

Images