Regitive language mimicking with constrained examples

Abstract

Devices, systems, and methods for generating phrases that cause obfuscation to language classifiers are generally discussed herein. A method may include determining, by an LC, a first classification score (CS) of a cue, the first classification score indicating whether the cue is a first class or a second class; predicting possible next words and a corresponding probability for each of the possible next words based on the cues and by a pre-trained language model (PLM); determining, by the LC, a second CS for each of the possible next words; determining, by the antagonism classifier, a respective score for each of the possible next words, the respective score being determined based on the prompted first CS, the second CS of the possible next word, and the probability of the possible next word; and selecting, by the adversarial classifier, a next one of the possible next words based on the respective scores.

Description

Background Art

[0001] Toxic language detection systems can be used to help determine whether to take down a post or otherwise remove content. Toxic language detection systems often incorrectly mark text containing mentions of minority groups as toxic because those groups are often targets of online hate. This over-reliance on targets for hate relevance makes it difficult for systems to detect implicitly toxic language. Toxic language detectors often over-rely on minority identity mentions when marking statements as toxic, without considering the deeper semantic meaning of the statement. This can lead to severe under-detection of more subtle forms of hate and over-detection of benign statements (e.g., “child abuse is wrong, racism is wrong, sexism is wrong”). This bias in toxicity detection risks further marginalizing or censoring minority groups because some online systems use toxic language detection systems when determining whether to take down a post or otherwise remove content. Summary of the invention

[0002] An apparatus, system, method, and computer-readable medium configured for improved language classifier (LC) operation are provided. The LC may be configured to determine a classification of an input phrase. The LC may confuse a phrase that is a first class as a second class, and vice versa. That is, the LC incorrectly classifies a phrase that is a first class as a second class. An embodiment may help the LC define a boundary between features of the first class and the second class. An embodiment may achieve such an improved definition by generating a confusing phrase using an adversarial classifier that selects the next word of the phrase based on the current LC classification of the phrase and the actual classification of the phrase.

[0003] A method for generating a phrase that confuses an LC may include determining, by the LC, a first classification score (CS) of a prompt, the first classification score indicating whether the prompt is of the first category or the second category. A pre-trained language model (PLM) may determine a possible next word and a corresponding probability for each of the possible next words based on the prompt. The LC may determine a second CS for each of the possible next words. An adversarial classifier may determine a corresponding score for each of the possible next words, the corresponding score being determined based on the first CS of the prompt, the second CS of the possible next word, and the probability of the possible next word. The adversarial classifier may select a next word among the possible next words based on the corresponding score.

[0004] The corresponding scores may be determined such that: if the first CS indicates that the cue is more likely to be the second class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the first class, and if the first CS indicates that the cue is more likely to be the first class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the second class. Selecting the next word may include: if the first CS indicates that the cue is more likely to be the second class, selecting the next word that maximizes the mathematical combination of the second CS and one minus the probability, and if the first CS indicates that the cue is more likely to be the first class, selecting the next word that maximizes the mathematical combination of the second CS and the probability. The second CS and the probability may be weighted, or the second CS and one minus the probability may be weighted to balance the understandability and confusion of the phrase.

[0005] The LC may be trained based on phrases. The PLM may be an autoregressive language model. Generating a phrase may include iterating N words from the first level to the Nth level of a beam search tree. Generating a phrase may include predicting a possible next word and a corresponding probability for each of the possible next words based on the i-th selected word and by the PLM. Generating a phrase may include determining a second CS for each of the possible next words by the LC. Generating a phrase may include determining a corresponding score for each of the possible next words by an adversarial classifier. Generating a phrase may include selecting a next word in the possible next words based on the corresponding scores. Generating a phrase may include iterating to generate the next word of the phrase based on the third CS of the selected next word. The LC may be a toxic language classifier (TLC), the first class may be neutral, and the second class may be toxic. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A diagram showing an embodiment of a user operating a toxic language classifier (LC) is shown by way of example.

[0007] Figure 2 A diagram illustrating an embodiment of a user operating a pre-trained language model (PLM) is shown by way of example.

[0008] Figure 3 A flow chart illustrating an embodiment of initial operations of a method for generating obfuscated output using an adversarial classifier in a loop is shown by way of example.

[0009] Figure 4 A flow chart of an embodiment of a second operation of a method for generating obfuscated output using an adversarial classifier in a loop is shown by way of example.

[0010] Figure 5 A flowchart of an embodiment of the final operations of a method for generating obfuscated output using an adversarial classifier in a loop is shown by way of example.

[0011] Figure 6A flow chart of an embodiment of a method for adversarial classifier selection for the next word of an obfuscated output is shown by way of example.

[0012] Figure 7 A diagram showing an embodiment of a method for generating phrases that a LC will find confusing is shown by way of example.

[0013] Figure 8 is a block diagram of an example of an environment including a system for neural network training.

[0014] Fig. 9 A block diagram of an embodiment of a machine (eg, a computer system) for implementing one or more embodiments is shown by way of example. DETAILED DESCRIPTION

[0015] Embodiments provide automatically generated prompts in a constrained manner to challenge a language classifier (LC). The LC may be a toxicity LC (TLC), a sentiment classifier, a question-answering model, a paraphrase model, etc. Embodiments are described primarily with reference to TLCs and the corresponding outputs of TLCs, and embodiments are equally applicable to LCs where the user wishes to better delineate the boundaries between classes.

[0016] A prompt is text (e.g., a string) that is passed into the model that encourages a specific behavior from the model. This is in contrast to scraping internet posts, which has been done previously. An embodiment includes using a pre-trained language model (PLM) to provide predicted words based on a portion of the prompt. The prompt can be run through an LC to determine a first class for the prompt. If the classifier produces a high score (e.g., greater than 0.5 on a scale of 0.0 to 1.0), the prompt is considered to be of the first class, and if the classifier produces a low score (e.g., less than 0.5), the prompt is considered to be of a different class. Other scoring systems can be used, what matters is whether the prompt is considered to be of the first class or the second class.

[0017] The prompt may be fed to the PLM one or more words at a time. The k most likely (e.g., top k) predicted outputs from the PLM may be retained. Each of the top k predicted outputs is added to the current output, resulting in k updated outputs. In the first iteration, the selected output is empty, so the updated output is only the top k predicted output. Each of the top k updated outputs may be provided to the LC, which generates a classification for each of the k updated outputs. One of the top k updated outputs may be selected based on a classification score (CS). The output of the top k updated outputs that maximizes the difference between the CS of the entire prompt and the CS of the output may be identified and selected as the output of the PLM. The next one or more words of the prompt may be provided to the PLM, and the top k predicted outputs of the PLM may be added to the currently selected output to generate the next iteration of updated outputs. These updated outputs may be provided to the LC, and selected in the same manner as the previous iteration, and the process may be repeated until the entire prompt has been fed to the PLM and the output is selected based on the CS from the LC. The result is a hint of either the first or second class, but the LC indicates it is the second or first class, respectively. The examples presented herein include a beam search of width one (1) (retaining one output at each iteration), but larger beam widths may be used.

[0018] The adversarial classifier between PLM and LC generated sentences has at least several advantages, first, it can limit false identity-toxicity correlations by generating a controlled number of toxic / benign sentences for each demographic group, which includes those that are often overlooked in toxic language corpora (e.g., native Americans). Second, machine generation and careful prompting enable the generation of implicitly toxic sentences (e.g., without profanity or insults), which are difficult to detect or find by definition and are therefore often missed in toxic language corpora. The reduction of profanity or insults in sentences provides a training set (or test set) that is more nuanced than a similar set with more profanity or insults. When used to train or test TLCs, these sets produce improved TLCs that can more accurately identify the subtle differences between toxic and neutral sentences and detect the subtle differences. These TLCs are more accurate than other TLCs. Embodiments provide an adversarial cyclic classifier decoding technique. Embodiments use an adversarial classifier to control the classification of output text by clicking the LC relative to a text generator (e.g., PLM) during beam search decoding. Given a hint of the first class, the adversarial classifier can encourage text generation from the text generator to be less likely to be of the first class based on the score from the LC. Similarly, the adversarial classifier can steer a text generator provided with a hint of the second class toward a hint that is more likely to be of the first class.

[0019] Experimental results show that fine-tuning an existing classifier on sentences generated by using an adversarial classifier in the loop consistently improves the performance of the trained model on existing human-written implicitly toxic datasets (e.g., +7-19%). This indicates that using data generated by an adversarial classifier in the loop provides a step to improve TLC and can potentially be used downstream to address issues from biased machine generation or neutral toxicity degradation.

[0020] Figure 1 A diagram of an embodiment of a user 108 operating an LC 112 is shown by way of example. The user 108 operates a computing device 106 to provide a prompt 102 to the LC 112. The prompt 102 may be provided by a PLM 220 (see Figure 2 ), user generated, a combination thereof, etc. LC 112 has been trained to provide an output 104 indicating the category of prompt 102. Example LC 112 that is a TLC includes HateBERT, PerspectiveAPI, Open AI Content Filter, AI2 Delphi, and Roberta, etc. Other types of LC 112 include sentiment classifiers, paraphrase models, question-answering models, etc. The remaining description of the figure describes the details that LC 112 is an instance of TLC, so the LC is referred to as TLC 112.

[0021] The dataset currently used to train TLC 112 biases TLC 112 to indicate that prompt 102 is toxic if it contains certain keywords, rather than considering the context of those keywords in prompt 102. This results in neutral content being considered toxic because it includes one or more keywords, and toxic content being considered neutral because it does not include one or more keywords. This problem of mischaracterizing prompts can lead to censorship and suppression of content that uses keywords and the spread of toxic content.

[0022] Prompts 102 are messages delivered to PLM 220 (see Figure 2 ). Designing prompts is very challenging. While there are several approaches for prompting a pre-trained PLM, a recent and promising direction is demonstration-based prompting. In demonstration-based prompting, example sentences are passed to the PLM 220, thereby encouraging the PLM 220 to produce similar but different sentences. For example, the PLM 220 can be encouraged to produce neutral sentences about a specific group of people by being provided with neutral prompts about that group of people.

[0023] Figure 2A diagram of an embodiment of a user 108 operating a PLM 220 is shown by way of example. The user 108 operates a computing device 106 to provide a prompt 102 to the PLM 220. The PLM 220 has been trained to provide an output prompt 222 that is similar in style, language, and tense as the prompt 102. Typically, the output prompt 222, when provided to the TLC 112, will have a toxicity score 104 similar to the prompt 102. Example PLMs 220 include GPT-3 (generates a pre-trained language model), Wu Dao 2.0, GPT-NEO, GPT-J-6B, or another autoregressive language model.

[0024] Demonstration-based prompts alone consistently produce toxic and benign sentences about minority groups, but there is no guarantee that these sentences will be challenging for TLC 112. Adversarial classifiers in loop decoding provide a variant of constrained beam search during decoding (selecting the next word) that generates sentences that are adversarial to TLC 112. In some previous constrained beam search settings, constraints are added during beam search decoding to force the model to include or exclude specific words or phrases in the output. Instead of using such hard constraints, embodiments impose soft constraints on TS 104 from TLC 112 during beam search.

[0025] An embodiment utilizes PLM 220 to play a game against TLC 112 to generate prompts that confuse TLC 112 in terms of toxicity. In this context, confusing means that TLC 112 considers prompt 102 to be toxic when it is neutral, and vice versa.

[0026] Figure 3 A flow chart of an embodiment of the initial operation of a method 300 for generating obfuscated cues using an adversarial classifier 334 in a loop is shown by way of example. Figures 3 to 5 In order to simplify the description, a beam search with a width of one (1) is used, but embodiments may use larger beam widths. Without loss of generality, a beam width of one (1) is provided. Initial operations include providing prompt 102 to PLM 220. PLM 220 performs a beam search decoding technique to generate a resulting prompt that is phonetically similar to prompt 102.

[0027] Beam search uses breadth-first search to build a search tree. The search tree includes possible words as nodes on a layer and edges between nodes on the directly adjacent layer, which indicate the probability that the word of the next node follows the previous word. At each level of the tree, it generates multiple successors of the state at the current level, sorting them in increasing order of trial cost. There are only a predetermined number of best states at each level (called the beam width). Only those states are expanded next. The larger the beam width, the fewer states are pruned. In the case of infinite beam width, no states are pruned, and beam search is the same as breadth-first search. The beam width limits the memory required to perform the search. Figure 3 The example shows the first level of the search tree.

[0028] The PLM 220 provides possible next words 332 (e.g., top k next words, etc.) at the first level of the search based on the prompt 102. The possible next words 332 are words that are determined to be most likely to appear first, which are most like the word of the prompt 102. Most likely means that the next word (NWX) has a probability greater than other words with non-zero probabilities. Each possible next word 332 is provided to the TLC 112. The TLC 112 generates a toxicity score (TS) 104A, 104B, 104K for each NWX. TS104A corresponds to NW1, TS104B corresponds to NW2, and so on.

[0029] Each possible NWX 332 and the corresponding probability is provided to an adversarial classifier 334. Each of the TSs 104A, 104B, 104K is provided to the adversarial classifier 334. The adversarial classifier 334 includes a Figure 6 104A, 104B, 104K. In general, the adversarial classifier 334 determines a mathematical combination of the probability and TS 104A, 104B, 104K for each possible next word 332. For example, the adversarial classifier 334 can determine a score for each possible next word 332. The score can be determined based on whether the prompt 102 is considered toxic or neutral by the TLC 112. If the prompt 102 is considered neutral, the score can be determined according to Equation 1: Score = Weight1*Probability+Weight2*TS Equation 1

[0030] If prompt 102 is considered toxic, a score may be determined according to Equation 2: Score = weight 1 * (1-probability) + weight 2 * TS Equation 2

[0031] The adversarial classifier 334 may then select the NW of the possible next words 332 with the highest score as the selected next word 336. By selecting words in this manner, the adversarial classifier 334 biases the resulting phrase consisting of a series of selected words to have a higher TS 104 if the prompt 102 is considered neutral, and a lower TS 104 if the prompt 102 is considered toxic.

[0032] In the figures, reference numerals without letter suffixes generally refer to specific instances of the components that include the letter suffixes. For example, TS104 generally refers to toxicity score, while TS104A is a specific instance of the general TS104.

[0033] Figure 4 A flowchart of an embodiment of a second operation of the method 300 for generating obfuscation cues using the adversarial classifier 334 in a loop is shown by way of example. The second operation includes assuming that Figure 3 SNW 446 is selected at the first level in the beam search tree and moves to the second level of the beam search tree including the top k next words 442. The possible next words 442 are the words determined by the PLM 220 to be most likely to appear after the string including the SNW 336. Each possible next word 332 is provided to the TLC 112 as a string including the selected next word 336 and NWX (in that order). The TLC 112 generates a corresponding toxicity score (TS) 104A, 104B, 104K for each string, each string including the selected next word 336 followed by a corresponding NWX.

[0034] Each possible next word 442 (NWX) and the corresponding probability is provided to the adversarial classifier 334. Each of the TS 104A, 104B, 104K is provided to the adversarial classifier 334. The adversarial classifier 334 determines a mathematical combination of the probability and the TS 104A, 104B, 104K for each possible next word 442. For example, the adversarial classifier 334 can determine a score for each possible next word 332. As in Figure 3 In an initial operation of the TLC 112, a score may be determined based on whether the prompt 102 is considered toxic or neutral by the TLC 112.

[0035] The adversarial classifier 334 may then select the NWX of the possible next words 442 with the highest score as the selected next word 446. By selecting words in this manner, the adversarial classifier 334 will include a series of selected words (via Figure 3 and Figure 4The resulting phrases for prompt 102 (SNW 336 and SNW 446) are biased to have a higher toxicity score if prompt 102 is considered neutral, and a lower toxicity score if prompt 102 is considered toxic.

[0036] The method 300 may continue through the levels of the beam search tree until a stop token is encountered or a maximum string length is achieved. The maximum string length is a hyperparameter that defines the maximum number of tokens that can be in a result prompt (sometimes referred to as a sentence or phrase). Figure 5 The final operations of the method 300 for generating a phrase that confuses the TLC 112 are shown.

[0037] Figure 5 A flow chart of an embodiment of the final operation of the method 300 for the adversarial classifier 334 in the cyclic generation of obfuscated cues is shown by way of example. The beam search tree provides possible next words 552 based on the string including the cue 102 and the SNWs 336, 446 provided up to this final operation. The possible next words 552 are the most likely to appear after the phrase determined so far including the word selected in each iteration as determined by the PLM 220. Each possible next word 552 is provided to the TLC 112 as a string including the selected next word for each iteration up to this final iteration (e.g., SNW 336, SNW 446, and SNW in subsequent iterations). The TLC 112 generates a corresponding toxicity score (TS) 104A, 104B, 104K for each string, each string including the selected next word followed by a corresponding NWX.

[0038] Each possible next word 552 (NWX) and the corresponding probability is provided to the adversarial classifier 334. Each of the TS 104A, 104B, 104K is provided to the adversarial classifier 334. The adversarial classifier 334 determines a mathematical combination of the probability and the TS 104A, 104B, 104K for each possible next word 552. For example, the adversarial classifier 334 can determine a score for each possible next word 552. Figure 3 and Figure 4 In operation, a score may be determined based on whether the prompt 102 is considered toxic or neutral by the TLC 112.

[0039] Then, the adversarial classifier 334 may select the NWX of the possible next word 552 with the highest score as the selected next word 556. By selecting words in this manner, the adversarial classifier 334 will include a series of selected words (SNW336, SNW 446, ..., SNW 556 to SNW 556). Figures 3 to 5) is biased to have a higher toxicity score if prompt 102 is considered neutral, and a lower toxicity score if prompt 102 is considered toxic.

[0040] Figure 6 A flow chart of an embodiment of a method 600 for selecting a next word for the adversarial classifier 334 is shown by way of example. The method 600 as shown includes: at operation 660, receiving a toxicity score ("TS(prompt)") of the prompt 102; at operation 662, determining whether the TS(prompt) indicates that the prompt 102 is toxic; at operation 664, if the prompt 102 is toxic, selecting a next word that maximizes the score determined according to equation 1; and at operation 666, if the prompt is neutral, selecting a next word that maximizes the score determined according to equation 2.

[0041] Weights: Weight 1 (W1) and Weight 2 (W2) may be set to balance the understandability of the generated phrase and the confusion the generated phrase causes to the TLC 112. In some embodiments, W1 and W2 may sum to one (1) or another constant. The higher W1 is relative to W2, the easier the generated phrase will be to understand. The higher W2 is relative to W1, the more confusing the TLC 112 will find the generated phrase. If W1=W2, then the probability and TS are equally weighted, and the generated phrase will be understandable because it is not confusing to the TLC 112.

[0042] Because the PLM 220 tends to generate phrases with TS 104 that are similar to the prompt 102 without the adversarial classifier 334 in the loop, the phrases selected by the adversarial classifier 334 are more likely to confuse the TLC 112 than phrases generated without the adversarial classifier 334. This confusion makes the resulting phrases useful for training or testing the TLC 112. Training based on confused phrases can help the TLC 112 learn a more nuanced understanding of why language is considered toxic.

[0043] Figure 7A diagram of an embodiment of a method 700 for generating phrases that the LC 112 will find confusing is shown by way of example. The method 700 as shown includes: at operation 770, determining, by the LC, a first classification score (CS) of a prompt indicating whether the prompt is of the first category or the second category; at operation 772, predicting possible next words and corresponding probabilities for each of the possible next words based on the prompt and by a pre-trained language model (PLM); at operation 774, determining, by the LC, a second CS for each of the possible next words; at operation 776, determining, by the adversarial classifier, a corresponding score for each of the possible next words, the corresponding score being determined based on the first CS of the prompt, the second CS of the possible next words, and the probability of the possible next words; and at operation 778, selecting, by the adversarial classifier, a next word from the possible next words based on the corresponding scores.

[0044] The method 700 may also include, wherein the corresponding scores are determined such that: if the first CS indicates that the hint is more likely to be the second class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the first class, and if the first CS indicates that the hint is more likely to be the first class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the second class. The method 700 may also include, wherein selecting the next word includes: if the first CS indicates that the hint is more likely to be the second class, selecting the next word that maximizes a mathematical combination of the second CS and one minus the probability, and if the first CS indicates that the hint is more likely to be the first class, selecting the next word that maximizes a mathematical combination of the second CS and the probability.

[0045] Method 700 may also include training an LC based on a phrase. Method 700 may also include: wherein the PLM is an autoregressive language model. Method 700 may also include, wherein generating a phrase includes iterating N words from the first level to the Nth level of a beam search tree, including: predicting a more likely next word and a corresponding probability for each of the possible next words based on the i-th selected word and by the PLM; determining a second CS for each of the possible next words by the LC; determining a corresponding score for each of the possible next words by the adversarial classifier; selecting a next word from the possible next words based on the corresponding score; and iterating to generate a next word of the phrase based on a third CS of the selected next word. LC is a toxic language classifier (TLC), a sentiment classifier (SC), a paraphrase model, a question-answering model, etc. The first class may be neutral, the second class may be toxic, and vice versa.

[0046] Another alternative application of toxicity classification can include sentiment analysis. For example, a review stating "the movie is entertaining, intelligent, visually creative, and especially lively and entertaining" has a positive sentiment, and "the movie as a whole is cheap garbage and a disservice to their death-defending efforts" has a negative sentiment. Suppose a company builds a classifier to understand its customers' satisfaction with various movies using reviews. An adversarial classifier can attack the sentiment classifier, creating sentences whose sentiment the sentiment classifier cannot correctly identify. The company can use the data generated by the adversarial classifier to improve their movie review sentiment classifier.

[0047] Another candidate application for virulence classification could include paraphrasing. For example, a company is selling a product that summarizes meetings using transcriptions of them. An important part of this task is paraphrasing (e.g., identifying that “Yesterday, Taiwan reported 35 new infections, bringing the total number of cases to 418.” and “The region reported 35 additional possible cases yesterday, for a total of 418” means the same thing). Adversarial classifiers can help attack a classifier that is performing this task to generate more data and improve it.

[0048] Another alternative application of toxicity classification can include question-answer generation. For example, suppose a user types the question into their search engine: "How do I become a good geologist?", and some other user has searched for "What should I do to become a great geologist?", and the search engine has the answer for the second question. The search engine may have a way (usually using a ranker or classifier) ​​to understand that these two questions have the same meaning. An adversarial classifier can help improve the classifier / ranker by performing attacks on it and generating adversarial data.

[0049] Phrases generated using the adversarial classifier in the loop were more likely to generate phrases that were toxic to TLC 112 than PLM 220 alone. For 125 randomly selected prompts (62 toxic and 63 non-toxic), two sentences were generated: one with the adversarial classifier in the loop and one without the adversarial classifier in the loop. The toxicity of 250 generated sentences was evaluated manually and with TLC 112. Using adversarial classifier 334 successfully fooled TLC 112 more frequently than using PLM 220 alone to generate prompts. 26.4% of the phrases generated using adversarial classifier 334 fooled TLC 112, while only 16.8% of the phrases generated by PLM 220 without adversarial classifier 334 fooled TLC 112. Using adversarial classifier 334 produced more difficult and more ambiguous examples on average.

[0050] The sentences generated by the adversarial classifier 334 can be used to evaluate the robustness of the TLC 112 and to generate sentences for attacking the TLC 112. Fine-tuning the TLC 112 based on the sentences generated by the adversarial classifier 334 can improve performance on other toxicity datasets. Fine-tuning the TLC 112 can include providing supervised input-output examples generated using the adversarial classifier 334 in the loop and which are applied to the TLC 112. These examples help the TLC 112 learn a more nuanced understanding of what is toxic and non-toxic to improve classification after deployment.

[0051] Adversarial classifier 334 can be used to challenge content filters via adversarial games between two artificial intelligence (AI) systems. Embodiments and similar methods can be used to improve content filters and prevent large-scale attacks on sensitive platforms. Improving TLC to include less machine bias helps combat online harm. In the absence of such a system, minority groups may be targeted by current (biased) systems. Adversarial classifier 334 is an important step in advancing this toxicity classification task.

[0052] Embodiments may include receiving potentially toxic user content. The user content may include text. An enhanced TLC may be provided, received, or otherwise accessed. The enhanced TLC was previously improved by utilizing an adversarial classifier that is configured to select words that are more likely to be misclassified by the TLC before enhancing the TLC. The TLC may be enhanced using the operations and methods discussed herein to generate phrases that may confuse the TLC, and then the TLC is trained based on the generated phrases. The enhanced TLC may operate on the received user content to generate an output indicating whether the user content is toxic. The output of the enhanced TLC may be used to identify the user content as toxic or non-toxic. Actions such as removing user content from a website, document, etc. may be performed for the identified toxic content (e.g., automatically - without user interference).

[0053] AI is a field that deals with developing decision-making systems to perform cognitive tasks that traditionally require living actors such as humans. Neural networks (NNs) are computational structures loosely modeled on biological neurons. Typically, NNs encode information (e.g., data or decisions) via weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are the basis for many AI applications, such as text prediction, toxicity classification, content filtering, etc. A TLC 112 or PLM 220 may include one or more NNs.

[0054] Many NNs are represented as matrices of weights (sometimes called parameters) corresponding to the connections being modeled. A NN operates by accepting data into a set of input neurons, which typically have many output connections to other neurons. At each pass between neurons, the corresponding weights modify the inputs and are tested against a threshold at the target neuron. If the weighted value exceeds the threshold, the value is weighted again, or transformed by a nonlinear function, and transmitted to another neuron further downstream in the NN graph—if the threshold is not exceeded, the value is typically not transmitted to neurons downstream in the graph, and the synaptic connection remains inactive. The weighted testing process continues until an output neuron is reached; the pattern and value of the output neuron constitutes the result of the NN processing.

[0055] Optimal operation of most NNs relies on accurate weights. However, NN designers typically do not know which weights will work for a given application. NN designers typically choose multiple layers of neurons or specific connections between layers that include ring connections. A training process that can be improved using phrases generated by an adversarial classifier 334 that plays an adversarial game between the TLC 112 and the PLM 220 can be used to determine appropriate weights by selecting initial weights.

[0056] In some examples, the initial weights may be randomly selected. The training data is fed to the NN, and the results are compared to an objective function that provides an error indication. The error indication is a measure of how wrong the results of the NN are compared to the expected results. The error is then used to correct the weights. Over multiple iterations, the weights will converge together to encode the operational data into the NN. This process may be referred to as optimization of an objective function (e.g., a cost or loss function), whereby the cost or loss is minimized.

[0057] Gradient descent techniques are commonly used to perform objective function optimization. Gradients (e.g., partial derivatives) are calculated relative to layer parameters (e.g., aspects of weights) to provide the direction and possible degree of correction, but do not result in a single correction to set the weight to the "correct" value. That is, over several iterations, the weight will move toward the "correct" or operationally useful value. In some embodiments, the movement amount or step size is fixed (e.g., the same from iteration to iteration). Small step sizes tend to take a long time to converge, while large step sizes may oscillate around the correct value or exhibit other undesirable behaviors. Variable step sizes can be tried to provide faster convergence without the disadvantages of large step sizes.

[0058] Backpropagation is a technique for feeding training data forward through a NN, where "forward" means that the data starts at the input neuron and follows the directed graph of neuron connections until it reaches the output neuron, and the objective function is applied backward through the NN to correct the synaptic weights. At each step in the backpropagation process, the weights are corrected using the results of the previous step. Thus, the results of the output neuron correction are applied to the neurons connected to the output neuron, and so on, until the input neuron is reached. Backpropagation has become a popular technique for training various NNs. Any well-known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0059] Figure 8 805. The system includes an artificial NN (ANN) 805 trained using a processing node 810. The processing node 810 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other processing circuit. In an example, multiple processing nodes can be used to train different layers of the ANN 805, or even different nodes 807 within a layer. Therefore, a group of processing nodes 810 are arranged to perform the training of the ANN 805.

[0060] The group of processing nodes 810 is arranged to receive a training set 815 for the ANN 805 (e.g., including phrases generated using the adversarial classifier 334). The ANN 805 includes a group of nodes 807 (shown as multiple rows of nodes 807) arranged in layers and a group of inter-node weights 808 (e.g., parameters) between the nodes in the group of nodes. In the example, the training set 815 is a subset of the complete training set. Here, the subset can enable processing nodes with limited storage resources to participate in training the ANN 805.

[0061] The training data may include multiple values ​​representing a domain, such as words, symbols, numbers, other parts of speech, etc. After the ANN 805 is trained, each value of the training or input 817 to be classified is provided to a corresponding node 807 in the first layer or input layer of the ANN 805. These values ​​propagate through the layers and are changed by the objective function.

[0062] As described above, the group of processing nodes is arranged to train neural networks to create trained neural networks. For example, after ANN is trained, the data input into ANN will produce effective classification 820 (for example, input data 817 will be assigned to categories). The training performed by the group of processing nodes 807 is iterative. In an example, each iteration of training ANN 805 is performed independently between the layers of ANN 805. Therefore, two different layers can be processed in parallel by different members of the group of processing nodes. In an example, the different layers of ANN 805 are trained on different hardware. The members of different members of the group of processing nodes can be located in different packages, shells, computers, cloud-based resources, etc. In an example, each iteration of training is performed independently between the nodes in the group of nodes. This example is additional parallelization, thereby training individual nodes 807 (for example, neurons) independently. In an example, nodes are trained on different hardware.

[0063] Fig. 9 A block diagram of an embodiment of a machine 900 (e.g., a computer system) for implementing one or more embodiments is shown by way of example. The client device 106, adversarial classifier 334, TLC 112, PLM 220, or components thereof may include one or more components of the machine 900. One or more of the client device 106, methods 300, 700, adversarial classifier 334, TLC 112, PLM 220, or components or operations thereof may be implemented, at least in part, using components of the machine 900. An example machine 900 (in the form of a computer) may include a processing unit 902, a memory 903, a removable storage device 910, and a non-removable storage device 912. Although an example computing device is shown and described as machine 900, in different embodiments, the computing device may be in different forms. For example, the computing device may alternatively be a smartphone, a tablet, a smartwatch, or include a computer program product related to the invention. Fig. 9 Other computing devices having the same or similar elements as shown and described. Devices such as smart phones, tablet computers, and smart watches are generally referred to as mobile devices. In addition, although various data storage elements are shown as part of machine 800, the storage device may also or alternatively include a cloud-based storage device accessible via a network (such as the Internet).

[0064] The memory 903 may include volatile memory 914 and nonvolatile memory 908. The machine 900 may include a computing environment or may have access to a computing environment that includes various computer-readable media, such as volatile memory 914 and nonvolatile memory 908, removable storage 910, and non-removable storage 912. Computer storage includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices capable of storing computer-readable instructions for execution to perform the functions described herein.

[0065] The machine 900 may include or have access to a computing environment including input 906, output 904, and communication connection 916. The output 904 may include a display device, such as a touch screen, which may also be used as an input device. The input 906 may include a touch screen, a touch pad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated into the machine 800 or coupled to the machine 800 via a wired or wireless data connection, and one or more of other input devices. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers, such as a database server, including cloud-based servers and storage devices. The remote computer may include a personal computer (PC), a server, a router, a network PC, a peer device or other public network node, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), Bluetooth or other network.

[0066] Computer-readable instructions stored on a computer-readable storage device may be executed by a processing unit 902 (sometimes referred to as a processing circuit) of the machine 900. Hard drives, CD-ROMs, and RAM are some examples of articles of manufacture that include non-transitory computer-readable media (such as storage devices). For example, the computer program 918 may be used to cause the processing unit 902 to perform one or more methods or algorithms described herein.

[0067] In some embodiments, the operations, functions or algorithms described herein can be implemented in software. Software can include computer executable instructions stored on a computer or other machine-readable medium or storage device (such as one or more non-transitory memories (e.g., non-transitory machine-readable media) or other types of local or networked hardware-based storage devices). In addition, such functions can correspond to subsystems, which can be software, hardware, firmware, or a combination thereof. Multiple functions can be performed in one or more subsystems as needed, and the described embodiments are only examples. Software can be executed on a processing circuit, which can include, for example, a digital signal processor, an ASIC, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or other types of processors operating on a computer system (such as a personal computer, a server, or other computer system), thereby converting such a computer system into a specifically programmed machine. Additionally or alternatively, the processing circuit may include electrical and / or electronic components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates, buffers, caches, memories, GPUs, CPUs, field programmable gate arrays (FPGAs), etc.). The terms computer-readable medium, machine-readable medium, and storage device do not include carrier waves or signals, as long as carrier waves and signals are considered too transient.

[0068] Additional Notes and Examples

[0069] Example 1 may include a method for generating a phrase that confuses a language classifier (LC), the method comprising: determining, by the LC, a first classification score (CS) of a prompt, the first classification score indicating whether the prompt is of the first category or the second category; predicting a possible next word and a corresponding probability for each of the possible next words based on the prompt and by a pre-trained language model (PLM); determining, by the LC, a second CS for each of the possible next words; determining, by an adversarial classifier, a corresponding score for each of the possible next words, the corresponding score being determined based on the first CS of the prompt, the second CS of the possible next word, and the probability of the possible next word; and selecting, by the adversarial classifier, a next word from the possible next words based on the corresponding score.

[0070] In Example 2, Example 1 also includes, wherein the corresponding scores are determined such that: if the first CS indicates that the hint is more likely to be the second category, the adversarial classifier selects the next word to generate a phrase that is more likely to be the first category, and if the first CS indicates that the hint is more likely to be the first category, the adversarial classifier selects the next word to generate a phrase that is more likely to be the second category.

[0071] In Example 3, Example 2 also includes, wherein selecting the next word includes: if the first CS indicates that the prompt is more likely to be the second category, selecting the next word that maximizes the mathematical combination of the second CS and one minus the probability, and if the first CS indicates that the prompt is more likely to be the first category, selecting the next word that maximizes the mathematical combination of the second CS and the probability.

[0072] In Example 4, Example 3 also includes, wherein the second CS and the probability are weighted, or the second CS and one minus the probability are weighted to balance the intelligibility and confusion of the phrase.

[0073] In Example 5, at least one of Examples 1 to 4 further includes training the LC based on the phrase.

[0074] In Example 6, at least one of Examples 1 to 5 also includes, wherein the PLM is an autoregressive language model.

[0075] In Example 7, at least one of Examples 1 to 6 also includes, wherein generating a phrase includes iterating N words from the first level to the Nth level of the beam search tree by the following steps: predicting a possible next word and a corresponding probability for each of the possible next words based on the i-th selected word and by the PLM; determining a second CS for each of the possible next words by the LC; determining a corresponding score for each of the possible next words by the adversarial classifier; selecting a next word from the possible next words based on the corresponding scores; and iterating to generate the next word of the phrase based on a third CS of the selected next word.

[0076] In Example 8, at least one of Examples 1 to 7 further includes, wherein the LC is a toxic language classifier (TLC), the first class is neutral and the second class is toxic.

[0077] Example 9 includes a computing system comprising a memory, a processing circuit coupled to the memory, the processing circuit configured to perform the operations of the method of at least one of Examples 1-8.

[0078] Example 10 includes a machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the operations of the method of at least one of Examples 1-8.

[0079] Although some embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the accompanying drawings do not require the order or sequential order shown to achieve the desired results. Other steps can be provided, or steps can be eliminated from the described flows, and other components can be added to or removed from the described systems. Other embodiments may be within the scope of the appended claims.

Claims

1. A method for generating a phrase that confuses a language classifier (LC), the method comprising: determining, by the LC, a first classification score (CS) for a cue, the first classification score indicating whether the cue is a first category or a second category; Predicting top-k next words and corresponding probabilities for each of the top-k next words based on the prompt and by a pre-trained language model (PLM); determining, by the LC, a second CS for each of the first k next words; determining, by an adversarial classifier, a corresponding score for each of the top k next words, the corresponding score being determined based on the first CS of the prompt, the second CS of the top k next words, and the probabilities of the top k next words; as well as A next word among the top k next words is selected by an adversarial classifier based on the corresponding scores.

2. A method according to claim 1, wherein the corresponding scores are determined such that: if the first CS indicates that the prompt is more likely to be the second class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the first class, and if the first CS indicates that the prompt is more likely to be the first class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the second class.

3. The method of claim 2, wherein selecting the next word comprises: If the first CS indicates that the hint is more likely to be the second category, then select the next word that maximizes the mathematical combination of the second CS and one minus the probability, and if the first CS indicates that the hint is more likely to be the first category, then select the next word that maximizes the mathematical combination of the second CS and the probability.

4. The method of claim 3, wherein the second CS and the probability are weighted, or the second CS and one minus the probability are weighted to balance the understandability and confusion of the phrase. The method of claim 1 , further comprising training the LC based on the phrase. The method of claim 1 , wherein the PLM is an autoregressive language model.

7. The method of claim 1 , wherein generating the phrase comprises iterating N words from a first level to an Nth level of a beam search tree by: Based on the i-th selected word and by the PLM, predicting the top k next words and the corresponding probability for each of the top k next words; determining, by the LC, the second CS for each of the first k next words; determining, by the adversarial classifier, a corresponding score for each of the top k next words; selecting the next word among the top k next words based on the corresponding scores; as well as Iteration is performed to generate a next word of the phrase based on the selected third CS of the next word.

8. The method of claim 1, wherein the LC is a Toxicity Language Classifier (TLC), the first class is neutral and the second class is toxic.

9. A computing system comprising: Memory; a processing circuit coupled to the memory, the processing circuit being configured to: determining, by a language classifier (LC), a first classification score (CS) for a prompt, the first classification score indicating whether the prompt is of the first category or the second category; Predicting top-k next words and corresponding probabilities for each of the top-k next words based on the prompt and by a pre-trained language model (PLM); determining, by the LC, a second CS for each of the first k next words; determining, by an adversarial classifier, a corresponding score for each of the top k next words, the corresponding score being determined based on the first CS of the prompt, the second CS of the top k next words, and the probabilities of the top k next words; as well as A next word among the top k next words is selected by an adversarial classifier based on the corresponding scores.

10. A computing system according to claim 9, wherein the corresponding scores are determined such that: if the first CS indicates that the prompt is more likely to be the second class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the first class, and if the first CS indicates that the prompt is more likely to be the first class, the adversarial classifier selects the next word to generate a phrase that is more likely to be the second class.

11. The computing system of claim 10, wherein selecting the next word comprises: If the first CS indicates that the hint is more likely to be the second category, then select the next word that maximizes the mathematical combination of the second CS and one minus the probability, and if the first CS indicates that the hint is more likely to be the first category, then select the next word that maximizes the mathematical combination of the second CS and the probability.

12. The computing system of claim 11, wherein the second CS and the probability are weighted, or the second CS and one minus the probability are weighted to balance understandability and confusion of the phrase.

13. The computing system of claim 9, further comprising training the LC based on the phrase.

14. The computing system of claim 9, wherein the PLM is an autoregressive language model.

15. A machine-readable medium comprising instructions which, when executed by a machine, cause the machine to perform the method according to any one of claims 1 to 8.

Images