A rendering-based adversarial sample generation method

Abstract

The application provides a rendering-based adversarial sample generation method, comprising the following steps: S1, evaluating the influence of each word on the output result of a text classification model; selecting the words having a certain influence on the model to classify and store, and generating a key word table of each category; S2, for the content in the text title, a modification method based on visually imperceptible insertion is adopted; S3, for the content in the beginning and end of the multi-paragraph text, a modification method based on visually imperceptible separation is adopted; S4, for the content in the middle of the multi-paragraph text and the first and last sentences of the single-paragraph text, a modification method based on visually imperceptible reverse sequence is adopted; and S5, for the content in the middle of the multi-paragraph text and the middle sentences of the single-paragraph text except the first and last sentences, a modification method based on visually imperceptible replacement is adopted. The application provides a beneficial reference for the semantic reservation and indistinguishability of the text adversarial sample and the improvement of the security of the text classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a rendering-based adversarial sample generation method. BACKGROUND

[0002] Adversarial samples refer to modifying input samples so that the model gives an incorrect output result with a high probability, while the sample can still be correctly understood and used by human users. There have been many explorations of adversarial samples in the image field, but constructing adversarial samples in the text field is still a challenge at present. First, due to the discrete nature of the text space, it is difficult to use the image gradient direction perturbation method. Second, existing text adversarial sample generation methods mostly use insertion, deletion, and replacement methods for words and phrases, or sentence-level paraphrasing methods. These methods cannot guarantee the fluency of the generated samples, making it easy for humans to misinterpret the sample meaning and affecting the human user's ability to restore the sample information.

[0003] Unicode supports most languages, allowing the same rendering presentation to be represented by various encoding sequences, and exposing some security problems such as homograph, bidirectional display, and number ambiguity. However, existing text classification models do not consider these rendering-based problems when encoding. SUMMARY

[0004] The present application aims to provide a rendering-based adversarial sample generation method that can create effective adversarial samples while preserving the semantic and visual information of the sample, greatly improving the quality of adversarial samples.

[0005] The technical solution of the present application is a rendering-based adversarial sample generation method, comprising:

[0006] S1, using a confidence method to score each word in each sample in the training set to evaluate the influence of each word on the output result of the text classification model; selecting words with a certain influence on the model to store and generate a list of key words;

[0007] S2, based on the key word list, for a piece of text, first process the content in the text title, use a visual-unnoticed insertion modification method, match the words in the modification table contained in the sample title as inserted words, and insert an invisible backspace control character after each word or character of the inserted word;

[0008] S3, based on the key word table, for the content in the beginning and end of the multi-paragraph text, using the visual imperceptible separation modification method, matching the words in the modification table contained in the beginning and end of the sample as the separation word, inserting zero-width space character randomly into any position except the beginning and end of the separation word;

[0009] S4, based on the key word table, for the content in the middle of the multi-paragraph text and the first and last sentences of the single-paragraph text, using the visual imperceptible reverse modification method, matching the words in the modification table contained in the middle of the sample and the first and last sentences of the single-paragraph text as the reverse word, using invisible control character to complete the reverse;

[0010] S5, based on the key word table, for the content in the middle of the multi-paragraph text and the middle sentences of the single-paragraph text except the beginning and end, using the visual imperceptible replacement modification method, matching the words in the modification table contained in the middle of the sample and the middle sentences of the single-paragraph text except the beginning and end as the replaced word, using the homograph replacement.

[0011] Preferably, S1 specifically comprises:

[0012] S11, taking each sample in the training set as a detection sample, for each input model detection sample, judging the detection sample category by model output, and using the confidence evaluation method to obtain the score C of the contribution degree of each word in the sample to the text classification model F ;

[0013] S12, arranging the words in the detection sample in the order of the contribution degree score from large to small, and extracting the top three words from each sample to store in the initial word table corresponding to the category of the sample;

[0014] S13, processing all detection samples, respectively removing the duplicate words in the initial word table of each category; and in each category initial word table, only storing the difference set of the initial word table and other category initial word table, to constitute the key word table of each category, wherein all stored are the words with great contribution to the classification confidence of each category of data.

[0015] Preferably, the score C of the contribution degree of each word in the text classification model in S11 F The specific calculation includes:

[0016]

[0017] Wherein, F represents the model, x i represents the i i th original sample, y k is the category label of the i k th sample, w k is the k k th word of the sample, x i represents the sample without the k i th word.

[0018] Preferably, S2 specifically comprises:

[0019] S21, matching the words in the modification table contained in the sample title as the inserted word, randomly selecting a word from the non-sample class key word table as the inserted word, inserting a non-visible backspace control character (BS character) after each character of the inserted word to form a backspace word;

[0020] S22, randomly inserting the backspace word in front of or behind the inserted word in the original sample, so that the inserted word is not noticeable in rendering.

[0021] Preferably, S3 specifically comprises:

[0022] Randomly inserting a zero-width space character (ZWSP character) into any position except the beginning and end of the separated word.

[0023] Preferably, the non-visible control character in S4 specifically comprises: embedding and rewriting termination character (PDF character), left-to-right override character (LRO character), right-to-left override character (RLO character), left-to-right isolation character (LRI character), isolation termination character (PDI character); wherein the left-to-right override character (LRO character) and the right-to-left override character (RLO character) will force the direction attribute of the characters after the control character to be overwritten to the corresponding direction; the embedding and rewriting termination character (PDF character) is an end mark, used to end the scope of the left-to-right override character (LRO character) and the right-to-left override character (RLO character); the left-to-right isolation character (LRI character) is used to make the string after it escape from the global direction of the parent element; the isolation termination character (PDI character) is used to end the scope of the left-to-right isolation character (LRI character);

[0024] Preferably, the specific operation steps of the non-visible control character are:

[0025] S41, taking out the reversed word in the original sample and reversing the order of the characters or words in the reversed word;

[0026] S42, adding the left-to-right override character (LRO character), the left-to-right isolation character (LRI character), and the right-to-left override character (RLO character) in order before the reversed word that has been reversed, and adding the embedding and rewriting termination character (PDF character), the isolation termination character (PDI character), and the embedding and rewriting termination character (PDF character) after the reversed word that has been reversed, which are used in pairs with the three non-visible control characters added before the word to end their scope, to form a reversed string;

[0027] S43, put the reverse order string back to the original position in the sample, so that the words in the reverse order are visually correctly rendered in the coding order.

[0028] The present application has the beneficial effect that the rendering-based adversarial sample generation method intuitively presents the security threat and vulnerability of the text encoding of the text classification model, thereby proposing an effective adversarial sample generation method, and providing effective technology for visual indistinguishability and semantic preservation of text adversarial sample generation. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A rendering-based adversarial sample generation method provided by an embodiment of the present application is shown in the overall schematic diagram.

[0030] Figure 2 A rendering-based adversarial sample generation method provided by an embodiment of the present application is shown in the schematic diagram of using the confidence method to measure the influence of words.

[0031] Figure 3 A rendering-based adversarial sample generation method provided by an embodiment of the present application is shown in the specific flowchart of generating adversarial samples based on rendering for a given text sample. DETAILED DESCRIPTION

[0032] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, and the embodiments of the present application are not limited thereto.

[0033] Embodiment 1

[0034] The present application is divided into two modules, including: a word importance evaluation module, which scores each word in each sample in the training set using a confidence method to evaluate the influence of each word on the output result of the text classification model. Select the words with certain influence on the model to store and generate a key word table for each category; an adversarial sample generation module: based on the key word table, according to the different positions of the words in the sample, automatically select the method corresponding to the position in multiple visual indistinguishable modification methods, modify the related words in the sample, and generate adversarial samples.

[0035] As shown in Figure 1 A rendering-based adversarial sample generation method, comprising:

[0036] S1, using a confidence method, scoring each word in each sample in the training set to evaluate the influence of each word on the output result of the text classification model; select the words with certain influence on the model to store and generate a key word table for each category;

[0037] S2, based on the key word list, for a piece of text first process the content in the title, using the visual unperceived based on the insertion of the modified method, matching the words contained in the modification table in the sample title as the inserted word, after each word or character of the inserted word, each inserted an invisible backspace control character;

[0038] S3, based on the key word list, for the content in the beginning and end of the multi-paragraph text, using the visual unperceived based on the separation of the modified method, matching the words contained in the modification table in the sample beginning and end as the separation word, inserting a zero-width space character randomly into any position except the beginning and end of the separation word;

[0039] S4, based on the key word list, for the content in the middle section of the multi-paragraph text and the first and last sentences of the single paragraph text, using the visual unperceived based on the reverse order of the modified method, matching the words contained in the modification table in the sample middle section and the first and last sentences of the single paragraph text as the reverse order word, using invisible control characters to complete the reverse order;

[0040] S5, based on the key word list, for the content in the middle section of the multi-paragraph text and the middle sentence of the single paragraph text except the beginning and end, using the visual unperceived based on the replacement of the modified method, matching the words contained in the modification table in the sample middle section and the middle sentence of the single paragraph text except the beginning and end as the replaced word, using similar characters to replace.

[0041] S1 specifically includes:

[0042] S11, each sample in the training set is taken as a detection sample, for each input model detection sample, the detection sample category is judged by the model output, and the score C of the contribution degree of each word in the sample to the text classification model is obtained by using the confidence evaluation method F ;

[0043] S12, the words in the detection sample are arranged in the order of the contribution degree score from large to small, and the top three words with the highest score are extracted from each sample and stored in the initial word table corresponding to the category;

[0044] S13, process all detection samples, respectively, to the initial word table of each category, and only store the difference set of the initial word table and other category initial word tables in each category initial word table to constitute the key word table of each category, wherein all the stored are the words with great contribution to the classification confidence of each category of data.

[0045] S2 specifically includes:

[0046] S21, match the words in the modification table contained in the sample title as the inserted word, randomly select a word from the non-sample belonging class key word table as the inserted word, insert a non-visible backspace control character (BS character) after each character of the inserted word, to form a backspace word; for example, the word "science", then add "sports" after it, and add a BS after "body" and "education", that is, add the backspace word like "body+BS education+BS".

[0047] S22, randomly insert the backspace word in front of or behind the inserted word in the original sample, so that the inserted word is not noticeable in rendering. For example, when the text input model is inserted after the backspace word, the model will understand the word according to the encoding of the word, that is, "science sports", while the human visual is still "science"; complete the insertion invisible to the human eye and only visible to the model, make the text classification model invalid while fully preserving the semantic and visual information.

[0048] S3 specifically includes:

[0049] Randomly insert a zero-width space character (ZWSP character) into any position except the first and last positions of the separated word. For example, insert a ZWSP character after the "run" word in "athlete", to form a string like "run+ZWSP athlete"; when the disturbed word is input into the model, the model will understand the word according to the encoding of the word, that is, from the perspective of the model, "run" and "athlete" are separated and become two different verbs, while the human visual is still the noun "athlete", from the human perspective, it does not cause any impact on the semantics; complete the separation invisible to the rendering and only visible to the model, make the model's understanding wrong, make the text classification model invalid while fully preserving the semantic and visual information.

[0050] The invisible control character in S4 specifically includes: embedded and rewritten termination character (PDF character), left-to-right override character (LRO character), right-to-left override character (RLO character), left-to-right isolate character (LRI character), isolate termination character (PDI character); wherein, the left-to-right override character (LRO character) and the right-to-left override character (RLO character) will forcibly cover the direction attribute of the characters after the control symbol as the corresponding direction; the embedded and rewritten termination character (PDF character) is an end mark, used to end the action range of the left-to-right override character (LRO character) and the right-to-left override character (RLO character); the left-to-right isolate character (LRI character) is used to make the string after it escape from the global direction of the parent element; the isolate termination character (PDI character) is used to end the action range of the left-to-right isolate character (LRI character);

[0051] The specific operation steps of the invisible control character are:

[0052] S41, the original sample is taken out and the order of the words or characters in the original sample is changed to the original reverse order; for example, the word "Jieqi", first arrange the words in the original reverse order, that is, "Jieqi".

[0053] S42, add the left-to-right override character (LRO character), left-to-right isolate character (LRI character), right-to-left override character (RLO character) in sequence to the already reversed word before the reversed word, and add the embedding and override termination character (PDF character), isolate termination character (PDI character), embedding and override termination character (PDF character) after the already reversed word, use the three invisible control characters added before the word to end their scope, and form a reverse order string; for example, add invisible direction control characters around the reversed word to form a reverse order string like "LRO+LRI+RLO+Jieqi+PDF+PDI+PDF".

[0054] S43, put the reverse order string back to the original position in the sample, so that the word with reverse order of encoding is correctly rendered visually. For example, when the reverse order string is input into the model, the model will understand the word according to the encoding order, that is, "Jieqi", while the human eye is correct "Jieqi"; make the word complete the reverse order that is invisible to the human eye and only visible to the model; so as to make the text classification model invalid without changing the visual information of the data and completely preserving the semantic information of the text.

[0055] In S5, for example, the structure of the characters greatly determines the understanding of the text semantics by humans, and characters with the same or similar structure will be easily recognized by humans. For example: the rendering effect of Latin letter "a" (Unicode: 0061) and Cyrillic letter "a" (Unicode: 0430) is the same. They will be judged by humans as lowercase Latin letter "a". The similar characters in Chinese characters also have the same effect: Chinese characters "Hu" (Unicode: 58f8) and "Hu" (Unicode: 58f6) also have similar rendering effects, and the rendering effects of "Zeng" (Unicode: 58ab) and "Wu" (Unicode: 58ff) are even almost the same. However, when processing text, the model usually regards a word as a discrete unit composed of a single character, and the change of a single character can easily make the model incorrectly identify a word as another word, resulting in errors of the model. For the content in the middle of a multi-paragraph text and the middle of a single-paragraph text except the beginning and end, the present application matches the words in the modification table contained therein as modified words. The present application uses similar characters to replace the characters in the modified words, which can make the model make mistakes while ensuring that the user can normally restore the sample information.

[0056] As Figure 2As shown, if deleting a word from a text significantly alters the confidence probability of the text's classification, it demonstrates that the word has a greater impact on the model's output.

[0057] This invention uses a confidence level method to measure the influence of words, through a formula. Calculate the contribution score of each word in the input sample to the model output.

[0058] Here, F represents the model, a concept in the field of artificial intelligence. The model refers to learning some functions (f) that provide a mapping relationship between input values ​​x and output values ​​y. The model is also the target of this invention.

[0059] x i Let y represent the i-th original sample. i w is the class label of the i-th sample. k For the k-th word in the sample, This represents the sample after removing the k-th word.

[0060] Figure 2 This invention uses confidence scores to rank a text, and the ranked text is used to generate adversarial examples for the recognition model. For example, in the text field "Huawei's sales this spring far exceeded expectations", "Huawei" has the highest confidence score (0.7591) and ranks first, while sales revenue has the second highest confidence score (0.5863). The remaining fields are ranked in order of confidence score.

[0061] like Figure 3 As shown, for a given text sample, this invention matches and modifies words in the modification table contained in the given sample. For each matched word, based on its position, it automatically selects the method corresponding to that position from multiple visually imperceptible modification methods as the modification method for that word: for content in the text title, a visually imperceptible insertion modification method is used; for content in the beginning and end paragraphs of multi-paragraph text, a visually imperceptible separation modification method is used; for content in the middle paragraphs of multi-paragraph text and the first and last sentences of single-paragraph text, a visually imperceptible reverse order modification method is used; for content in the middle paragraphs of multi-paragraph text and the middle sentences of single-paragraph text excluding the beginning and end, a visually imperceptible replacement modification method is used. Following these steps, this invention modifies all matched words in the given sample sequentially according to the modification method corresponding to their position, thus completing the generation of adversarial examples.

[0062] Figure 3An example of generating adversarial samples by visual rendering. For example, the text field "HP sales in spring this year far exceeded expectations", first the invention sorts this text field by confidence to get a new order "HP, sales, spring, expectations, far exceeded, this year"); second, the invention uses different visual imperceptible modification methods to modify the different positions of the words in the text, including imperceptible reverse order, imperceptible separation, imperceptible insertion and imperceptible replacement, and the adversarial sample of this text is "this year spring HP house BS house BS sales far exceeded expectations", and the visual rendering effect is still "this year spring HP sales far exceeded expectations"

[0063] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the flow in the drawings is not necessarily necessary for the implementation of the present application.

Claims

1. A rendering-based adversarial example generation method, characterized in that, include: S1 uses the confidence method to score each word in each sample in the training set to evaluate the influence of each word on the output of the text classification model. Select words that have a certain influence on the model for classification and storage, and generate various key word lists; S2, based on the key word list, first processes the content in the text title of a text, and adopts a visually imperceptible insertion modification method. It matches the words in the modification list contained in the sample title as the words to be inserted, and inserts an invisible backspace control character after each word or character of the inserted word. S3, based on the key word list, for the content in the beginning and end paragraphs of multi-paragraph text, adopts a visually imperceptible separation modification method, matches the words in the modification list contained in the beginning and end paragraphs of the sample as the separated words, and randomly inserts zero-width space characters into any position of the separated words except the beginning and end; S4, based on the key word list, for the content in the middle paragraphs of multi-paragraph texts and the first and last sentences of single-paragraph texts, adopts a visually imperceptible reverse modification method, matching the words in the modification list contained in the middle paragraphs and the first and last sentences of single-paragraph texts as the reversed words, and using invisible control characters to complete the reverse order; S5, based on the key word list, for the content in the middle paragraphs of multi-paragraph texts and the middle sentences of single-paragraph texts excluding the first and last, adopts a visually imperceptible replacement modification method, matching the words in the modification list contained in the middle paragraphs and middle sentences of sample single-paragraph texts excluding the first and last as the words to be replaced, and using similar-looking characters for replacement.

2. The adversarial example generation method based on rendering according to claim 1, characterized in that, S1 specifically includes: S11 uses each sample in the training set as a probe sample. For each probe sample input into the model, the model output determines the category of the probe sample, and the confidence score C is used to evaluate the contribution of each word in the sample to the text classification model. F ; S12: Arrange the words in the probe samples in descending order of their contribution to the model, and extract the three words with the highest scores from each sample and store them in the initial word list of the corresponding category of the sample. S13, process all probe samples, remove duplicates from the initial vocabulary lists of each category; and in each category's initial vocabulary list, store only the difference between the initial vocabulary list and the initial vocabulary lists of other categories to form the key vocabulary lists of each category, which contain only words that significantly contribute to the confidence of the classification of each category of data.

3. The adversarial example generation method based on rendering according to claim 2, characterized in that, The score C for each word in S11 reflects its contribution to the text classification model. F The specific calculations include: Where F represents the model, x i Let y represent the i-th original sample. i w is the class label of the i-th sample. k For the k-th word in the sample, This represents the sample after removing the k-th word.

4. The adversarial example generation method based on rendering according to claim 1, characterized in that, S2 specifically includes: S21, match the words in the modification table contained in the sample title as the inserted words, randomly select a word from the key word table of the non-sample class as the inserted word, and insert an invisible backspace control character (BS character) after each word or character of the inserted word to form a backspace word; S22, randomly inserts backspace words before or after the word to be inserted in the original sample, so that the inserted words are not noticeable in the rendering.

5. The adversarial example generation method based on rendering according to claim 1, characterized in that, S3 specifically includes: Insert zero-width space characters (ZWSP characters) randomly into any position of the separated word, except for the beginning and end.

6. The adversarial example generation method based on rendering according to claim 1, characterized in that, In S4, invisible control characters specifically include: Embedded and Rewrite Termination Character (PDF character), Left-to-Right Rewrite Character (LRO character), Right-to-Left Rewrite Character (RLO character), Left-to-Right Separation Character (LRI character), and Separation Termination Character (PDI character). The Left-to-Right Rewrite Character (LRO character) and Right-to-Left Rewrite Character (RLO character) force the direction attribute of the character following the control character to be overridden to the corresponding direction. The Embedded and Rewrite Termination Character (PDF character) is an end marker used to terminate the scope of the Left-to-Right Rewrite Character (LRO character) and Right-to-Left Rewrite Character (RLO character). The Left-to-Right Separation Character (LRI character) is used to remove the following string from the global direction of the parent element. The Separation Termination Character (PDI character) is used to terminate the scope of the Left-to-Right Separation Character (LRI character).

7. The adversarial example generation method based on rendering according to claim 6, characterized in that, The specific steps for operating invisible control characters are as follows: S41, extract the reversed words from the original sample and change the order of the characters or symbols in them back to the original reverse order; S42 adds the left-to-right rewrite character (LRO character), the left-to-right isolation character (LRI character), and the right-to-left rewrite character (RLO character) in sequence before the reversed word. It adds the embedding and rewrite termination character (PDF character), the isolation termination character (PDI character), and the embedding and rewrite termination character (PDF character) after the reversed word. These characters are used in pairs with the three invisible control characters added before the word to end their scope, thus forming the reversed string. S43, put the reversed string back into its original position in the sample so that words with reversed encoding order are rendered correctly visually.

Images