Text processing method, text processing device, and electronic device

Abstract

This invention provides a text processing method, a text processing device, and an electronic device, relating to the field of artificial intelligence technology. The method includes: acquiring multiple original samples, the original samples including text and labels; generating adversarial samples for each of the original samples; determining positive and negative samples for each of the original samples based on the adversarial samples; performing supervised contrastive learning and adversarial training based on the positive and negative samples to obtain a text classification model; and using the text classification model to classify the text to be processed. This invention improves the robustness of the text classification model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a text processing method, a text processing device, and an electronic device. Background Technology

[0002] Text classification models based on deep neural networks face the threat of adversarial examples. Adversarial examples are input samples with added subtle perturbations that cause the model to produce an incorrect output. The emergence of adversarial examples reveals the shortcomings of deep learning models in terms of robustness. Summary of the Invention

[0003] This invention provides a text processing method, a text processing device, and an electronic device to address the shortcomings of existing technologies in defending against adversarial text and to improve the robustness of text classification models.

[0004] This invention provides a text processing method, comprising:

[0005] Obtain multiple raw samples, including text and labels;

[0006] Generate adversarial samples corresponding to each original sample, and determine the positive and negative samples of each original sample based on the adversarial samples;

[0007] A text classification model is obtained by supervised contrastive learning and adversarial training based on the positive and negative samples, and the text to be processed is classified using the text classification model.

[0008] According to one embodiment of the present invention, the text of the original sample includes multiple words, and the generation of adversarial samples corresponding to each original sample includes:

[0009] Determine the set of synonyms for each word in the original sample;

[0010] The words and the words in the synonym set are converted into word embedding representations to obtain the embedding space corresponding to each original sample;

[0011] Based on the embedding space, the embedding space of the adversarial sample corresponding to the original sample is determined.

[0012] According to one embodiment of the present invention, determining the positive and negative samples of each of the original samples based on the adversarial examples includes:

[0013] Based on the labels of the original samples, determine the set of samples of the same type with the same labels, and regard the original samples in the same set of samples as samples of the same type, and regard the original samples in different sets of samples as samples of different types.

[0014] For the first sample in the original samples, the adversarial sample, the same sample, and the adversarial sample corresponding to the first sample are taken as positive samples of the first sample.

[0015] The outlier samples of the first sample, and the adversarial samples corresponding to the outlier samples, are taken as the negative samples of the first sample.

[0016] According to one embodiment of the present invention, the step of obtaining a text classification model by performing supervised contrastive learning and adversarial training based on the positive samples and the negative samples includes:

[0017] The supervised contrast loss is determined based on the positive and negative samples.

[0018] The text classification model is trained based on the supervised contrastive loss.

[0019] According to one embodiment of the present invention, training the text classification model based on the supervised contrastive loss includes:

[0020] The adversarial loss is determined based on the aforementioned supervised contrastive loss;

[0021] Determine the maximum value of the adversarial loss, and perform adversarial training on the text classification model by optimizing the maximum value of the adversarial loss.

[0022] The present invention also provides a text processing apparatus, comprising:

[0023] A sample acquisition module is used to acquire multiple raw samples, including text and labels;

[0024] An adversarial generation module is used to generate adversarial samples for each of the original samples, and to determine positive and negative samples for each of the original samples based on the adversarial samples;

[0025] The model training module is used to obtain a text classification model by performing supervised contrastive learning and adversarial training based on the positive samples and the negative samples, and then using the text classification model to classify the text to be processed.

[0026] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the text processing methods described above.

[0027] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text processing method as described above.

[0028] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the text processing methods described above.

[0029] The text processing method, text processing device, and electronic device provided by this invention construct adversarial examples from original samples and add them to the training data, which can improve the model's performance on adversarial examples, enhance the model's defense capabilities, and improve the model's robustness. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the text processing method provided by the present invention;

[0032] Figure 2 This is a schematic diagram of the structure of the text processing device provided by the present invention;

[0033] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] In this specification, the terms "first," "second," "third," etc., are used only as markers and are not intended to limit the number or order of objects.

[0038] The text processing method, text processing apparatus, and electronic device of the present invention are described below with reference to the accompanying drawings.

[0039] This invention first provides a text processing method. Exemplarily, this text processing method can be applied to various electronic devices such as personal computers (PCs), tablet computers, virtual reality devices, augmented reality devices, and servers; this embodiment does not impose any limitations on it.

[0040] Figure 1 A flowchart of the text processing method in this embodiment is shown. Figure 1 As shown, the text processing method may include the following steps:

[0041] Step 10: Obtain multiple raw samples, which include text and labels.

[0042] The original samples refer to text with category labels. The text may include multiple sentences or words. A certain number of texts are pre-acquired to form the training dataset. For example, the original samples can be represented as data pairs of (x, y), where x is the text and y is the category label corresponding to the text. The training dataset can be represented as: N is the number of original samples in the training dataset, and N can be any positive integer, such as 100, 1000, etc.; i represents the i-th original sample. Each original sample x i It can include multiple words, for example, L words, where L can be any positive integer; this implementation does not impose any limitation on this. Original sample x i It can be represented as:

[0043] Step 20: Generate adversarial samples corresponding to each original sample, and determine the positive and negative samples of each original sample based on the adversarial samples.

[0044] In an exemplary implementation, the initial values ​​of the adversarial sample can be the same as those of its corresponding original sample. That is, copying the original sample yields the initial adversarial sample. The adversarial sample gradually changes as the model is trained, thus differing from the original sample. First, the discrete words in the text are converted into continuous word embeddings, thereby transforming the text into vectors. Then, these word embeddings are copied to obtain the adversarial sample of the original sample.

[0045] To make the word embedding representation space of the text more compact, this embodiment can obtain the set of synonyms for each word in the text (i.e., the original sample), and convert all words in the original sample and their synonyms into word embedding representations. The embedding representation space of the original sample is then obtained based on all the converted word embedding representations. This embedding representation space is also the embedding representation space of the adversarial sample. For example, for the original sample x... i Each word in the dictionary can be used to obtain multiple synonyms, resulting in a synonym set. (Using...) To represent the original sample x i The j-th word The set of synonyms, then words The word embedding is represented as:

[0046]

[0047] Where T is the size of the synonym set, w k It is the weight of the synonyms in the synonym set, then v(·) is a word embedding function used to map synonyms in a set of synonyms to word embedding representations. The convex hull formed by the word embedding representations of all words in the original sample and their synonyms constitutes the embedding space of the original sample. For example, v(x) i ) represents the original sample x i The embedding space. In this embodiment, the embedding space of the text is made more compact by constructing the word embedding representation of synonyms as a convex hull.

[0048] Furthermore, for a batch of data of size B in the original sample: Its adversarial example embedding space can be represented as The embedding space of the original sample v(x) i The embedding space of its corresponding adversarial examples The initial values ​​are the same. To simplify the representation, we can merge the original sample with its adversarial sample as follows: The original samples correspond to elements 1 through B, and the adversarial samples are a set. The elements from the (i+B)th to the 2Bth, i.e.

[0049] After determining the adversarial sample for each original sample, the corresponding positive and negative samples for each original sample can be identified, thus forming positive sample pairs and negative sample pairs. For example, first, based on the labels of the original samples, determine the set of samples of the same class with the same labels. Original samples within the same set of samples of the same class are considered to be of the same class, while original samples not in the same set are considered to be of different classes. For any original sample (such as the first sample), the adversarial sample, the same class sample, and the adversarial sample corresponding to the same class sample are considered as positive samples of the first sample; the different class sample and the adversarial sample corresponding to the different class sample are considered as negative samples of the first sample.

[0050] Specifically, samples with the same label in the original samples are grouped into the same set to obtain a set of samples of the same class. For example, samples with label 1 are grouped into one set of samples of the same class, samples with label 2 are grouped into another set of samples of the same class, and so on. In this embodiment, the embedding space of the original samples is constructed by using the set of synonyms of the original samples, which can increase the inter-class distance of the samples, thereby helping to improve the classification accuracy of the model.

[0051] In summary, for each original sample v(x) i If the original sample v(x) is used as the anchor sample, then its corresponding adversarial sample, similar samples with the same label, and the adversarial sample corresponding to the similar sample are positive samples of the anchor sample. Therefore, the original sample v(x) is considered as a positive sample. i The set of positive samples is {v(x)} j |i≠j,y i =y j The negative sample set is {v(x,j∈[1,2B]},j∈[1,2B]}. j |i≠j,y i ≠y j ,j∈[1,2B]}. In this embodiment, constructing positive and negative samples using the original samples and adversarial samples is more conducive to training the model to distinguish between original samples and adversarial samples of the same type.

[0052] Step 30: Based on the positive samples and the negative samples, perform supervised contrastive learning and adversarial training to obtain a text classification model, and use the text classification model to classify the text to be processed.

[0053] Positive and negative sample pairs, consisting of anchor samples and their corresponding positive and negative samples, are input into a text classification model. Supervised contrastive learning and adversarial training are then performed on this model. The text classification model can be a neural network model of various structures; for example, it may include an encoder and a classifier. The encoder encodes the original samples and extracts features from the encoded samples. The extracted features are then classified by the classifier, which may employ a softmax function to output the probability of each feature as a class label.

[0054] For example, after determining the positive and negative samples of each original sample, the text classification model can be trained using positive sample pairs (formed by the original sample and its positive samples) and negative sample pairs (formed by the original sample and its negative samples). Specifically, a supervised contrastive loss can be determined based on the positive and negative samples, and the text classification model is trained by optimizing this supervised contrastive loss. The supervised contrastive loss can be expressed as follows:

[0055]

[0056] Where θ represents the parameters of the text classification model; v(x) represents the input sample; y represents the label of v(x); and N represents the size of the positive sample set. For indicator functions, Represents the set of positive samples. φ represents the fused set of positive and negative samples; φ is the encoder of the text classification model; τ is the temperature coefficient used to adjust the smoothness of the function. Optimizing this supervised contrastive loss function can reduce the distance between positive sample pairs while increasing the distance between negative sample pairs.

[0057] In an exemplary implementation, the text classification model further includes an adversarial loss. That is, the loss function of the text classification model can include two parts: a supervised contrastive loss and an adversarial loss. The adversarial loss can be determined based on the supervised contrastive loss. For example, the adversarial loss can be the sum of the supervised contrastive loss and the KL divergence, as shown in formula (3):

[0058]

[0059] in, Given the values ​​of the adversarial sample corresponding to the original sample x in the embedding space, combined with formula (1), then... f is the text classification model; parameter β is the weight parameter used to balance the supervised contrastive loss and KL divergence; the KL function is used to calculate the KL divergence between the distributions of the original samples and adversarial samples; L SCL To supervise the contrastive loss, the adversarial loss L is calculated for each adversarial example. ADVWe can identify the adversarial sample that maximizes the adversarial loss. Using this adversarial sample to train a text classification model can make the model more robust.

[0060] For example, the optimization objective of a text classification model is as follows:

[0061]

[0062] Among them, L CE This represents the cross-entropy loss of the text classification model on the original samples. The optimization objective involves maximizing the inner adversarial sample to find the sample that maximizes the adversarial loss, i.e., the sample with the highest probability of model error. Minimizing the outer adversarial sample optimizes the maximum adversarial loss, minimizing the model's loss on the original samples; in other words, making the model robust enough to defend against adversarial attacks.

[0063] A trained text classification model can be used to recognize text. Specifically, the text to be processed can be input into the trained text classification model to obtain its corresponding category label. The text to be processed can be one or more sentences, words, etc., and this implementation does not impose any special limitations on this. The text classification model obtained by incorporating an adversarial training method with supervised contrastive learning can guarantee the inter-class distance of the original samples, thereby achieving a better balance between the model's classification performance and robustness.

[0064] Furthermore, this embodiment also provides a text processing apparatus that can be used to perform the above-described text processing method. Figure 2 A block diagram of the text processing device is shown. Figure 2 As shown, the text processing device 20 may include: a sample acquisition module 21, used to acquire multiple original samples, the original samples including text and labels; an adversarial generation module 22, used to generate adversarial samples corresponding to each original sample, and determine positive and negative samples for each original sample based on the adversarial samples; and a model training module 23, used to perform supervised contrastive learning and adversarial training based on the positive and negative samples to obtain a text classification model, and use the text classification model to classify the text to be processed.

[0065] In one embodiment of the present invention, the text of the original sample includes multiple words, and the adversarial generation module 22 specifically includes: a synonym determination module, used to determine the set of synonyms for each word in the original sample; an embedding representation module, used to convert the word and the words in the set of synonyms into word embedding representations to obtain the embedding space corresponding to each original sample; and an adversarial sample generation module, used to determine the embedding space of the adversarial sample corresponding to the original sample based on the embedding space.

[0066] In one embodiment of the present invention, the initial value of the adversarial sample is the same as that of the corresponding original sample.

[0067] In one embodiment of the present invention, the adversarial generation module 22 specifically includes: a sample classification module, used to determine a set of similar samples with the same label based on the label of the original sample, wherein the original samples in the same set of similar samples are similar samples, and the original samples not in the same set of similar samples are dissimilar samples; a positive sample module, used to take the adversarial samples corresponding to the first sample, the similar samples, and the adversarial samples corresponding to the similar samples as positive samples of the first sample; and a negative sample module, used to take the dissimilar samples of the first sample and the adversarial samples corresponding to the dissimilar samples as negative samples of the first sample.

[0068] In one embodiment of the present invention, the model training module 23 specifically includes: a contrast loss determination module, used to determine a supervised contrast loss based on the positive samples and the negative samples; and a contrast training module, used to train the text classification model based on the supervised contrast loss.

[0069] In one embodiment of the present invention, the model training module 23 specifically includes: an adversarial loss determination module, used to determine the adversarial loss based on the supervised contrastive loss; and an adversarial training module, used to determine the maximum value of the adversarial loss and perform adversarial training on the text classification model by optimizing the maximum value of the adversarial loss.

[0070] The functional modules of the text processing device in this embodiment correspond to the steps of the text processing method described above. For details of the text processing device, please refer to the implementation method of the text processing method described above.

[0071] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a text processing method, which includes: acquiring multiple original samples, the original samples including text and labels; generating adversarial samples for each original sample; determining positive and negative samples for each original sample based on the adversarial samples; performing supervised contrastive learning based on the positive and negative samples to obtain a text classification model; and using the text classification model to classify the text to be processed.

[0072] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the text processing method provided in the above embodiments. The method includes: acquiring multiple original samples, the original samples including text and labels; generating adversarial samples for each original sample; determining positive and negative samples for each original sample based on the adversarial samples; performing supervised contrastive learning based on the positive and negative samples; obtaining a robust text classification model through adversarial training; and classifying the text to be processed using the text classification model.

[0074] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the text processing methods provided in the above embodiments. The method includes: acquiring a plurality of original samples, the original samples including text and labels; generating adversarial samples for each of the original samples; determining positive and negative samples for each of the original samples based on the adversarial samples; performing supervised contrastive learning based on the positive and negative samples to obtain a text classification model; and using the text classification model to classify the text to be processed.

[0075] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text processing method, characterized in that, include: Obtain multiple raw samples, including text and labels; Generate adversarial samples corresponding to each original sample, and determine the positive and negative samples of each original sample based on the adversarial samples; The supervised contrast loss is determined based on the positive and negative samples. The adversarial loss is determined based on the supervised contrastive loss, wherein the adversarial loss is the sum of the supervised contrastive loss and the KL divergence: ； Let x be the value of the adversarial sample corresponding to the original sample x in the embedding space; For text classification models; parameters The weighting parameter is used to balance the supervised contrastive loss and the KL divergence; the KL function is used to calculate the KL divergence between the original sample and adversarial sample distributions. To monitor and compare losses; Here are the parameters of the text classification model; y is... Tags; Let x be the value of the original sample x in the embedding space; To combat the losses; Determine the maximum value of the adversarial loss, and perform adversarial training on the text classification model by optimizing the maximum value of the adversarial loss, wherein the optimization objective of the text classification model is: ； conv(x) is the cross-entropy loss of the text classification model on the original sample. conv(x) is the embedding space of the original sample x. The embedding space of the original sample x is the convex hull formed by the word embedding representations of the original sample x and its synonyms. The optimization objective is to maximize the inner part to find the adversarial sample that maximizes the adversarial loss, and minimize the outer part to optimize the maximum adversarial loss, so that the model's loss on the original sample is minimized. The text classification model described above is used to classify the text to be processed.

2. The text processing method according to claim 1, characterized in that, The original sample text includes multiple words, and generating adversarial samples corresponding to each original sample includes: Determine the set of synonyms for each word in the original sample; The words and the words in the synonym set are converted into word embedding representations to obtain the embedding space corresponding to each original sample; Based on the embedding space, the embedding space of the adversarial sample corresponding to the original sample is determined.

3. The text processing method according to claim 1, characterized in that, The initial values ​​of the adversarial sample are the same as those of the corresponding original sample.

4. The text processing method according to claim 1, characterized in that, The step of determining the positive and negative samples of each original sample based on the adversarial samples includes: Based on the labels of the original samples, determine the same set of samples with the same labels. Original samples in the same set of samples are considered to be of the same type, and original samples not in the same set of samples are considered to be of different types. For the first sample in the original samples, the adversarial sample, the same sample, and the adversarial sample corresponding to the first sample are taken as positive samples of the first sample. The outlier samples of the first sample, and the adversarial samples corresponding to the outlier samples, are taken as the negative samples of the first sample.

5. A text processing device, characterized in that, include: A sample acquisition module is used to acquire multiple raw samples, including text and labels; An adversarial generation module is used to generate adversarial samples for each of the original samples, and to determine positive and negative samples for each of the original samples based on the adversarial samples; The model training module specifically includes: A contrast loss determination module is used to determine the supervised contrast loss based on the positive samples and the negative samples; The adversarial loss determination module is used to determine the adversarial loss based on the supervised contrastive loss, wherein the adversarial loss is the sum of the supervised contrastive loss and the KL divergence: ； Let x be the value of the adversarial sample corresponding to the original sample x in the embedding space; For text classification models; parameters The weighting parameter is used to balance the supervised contrastive loss and the KL divergence; the KL function is used to calculate the KL divergence between the original sample and adversarial sample distributions. To monitor and compare losses; Here are the parameters of the text classification model; y is... Tags; Let x be the value of the original sample x in the embedding space; To combat the losses; An adversarial training module is used to determine the maximum value of the adversarial loss and to perform adversarial training on the text classification model by optimizing the maximum value of the adversarial loss. The optimization objective of the text classification model is: ； Let conv(x) be the cross-entropy loss of the text classification model on the original sample, and let conv(x) be the embedding space of the original sample x. The embedding space of the original sample x is the convex hull formed by the word embedding representations of the original sample x and its synonyms. The optimization objective is to maximize the internal part to find the adversarial sample that maximizes the adversarial loss, and minimize the external part to optimize the maximum adversarial loss, so that the model's loss on the original sample is minimized. The text classification model is used to classify the text to be processed.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the text processing method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text processing method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the text processing method as described in any one of claims 1 to 4.

Images