Model attack resistance testing method and system
By calculating the importance of word vectors and synonyms, a variety and targeted attack samples are generated, which solves the problem of low generation efficiency of attack samples in black box attacks and improves the efficiency of model resistance to attack testing.
Patent Information
- Application Number
- CN202510023689.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-07
- Publication Date
- 2025-04-18
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In black box attack scenarios, how to generate diverse and targeted attack samples based on limited output information to improve the efficiency of model anti-attack testing.
By dividing the test sentences into words, obtaining word vectors, calculating the synonyms and their importance, randomly selecting synonyms to replace the original words, generating attack samples, and testing the model's resistance to attack.
It improves the diversity and targetedness of attack samples, reduces the search space for generating adversarial samples, improves attack efficiency, and effectively avoids the use of invalid or poor synonyms.
Smart Images
Figure CN119848876A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a model anti-attack testing method and system. Background Art
[0002] Artificial intelligence models are the main approach in natural language processing. However, while these models improve performance, they also expose vulnerabilities to malicious attacks. These attacks may cause the model to output incorrect results. For example, adversarial sample attacks can cause the model to produce incorrect classification results by making small perturbations to the input data. Depending on the attacker's understanding of the model information, attacks can be divided into white-box attacks, black-box attacks, and gray-box attacks. Unlike white-box attacks, black-box attacks assume that the attacker cannot access the internal structure, parameters, and gradient information of the target model and can only attack by inputting samples and observing the output of the model. In many cases, the internal details of the model are confidential, and black-box attack scenarios are closer to practical applications. In black-box attacks, input samples need to be carefully constructed to induce the model to produce incorrect prediction results without knowing the internal structure and parameters of the target model. There can be an unlimited number of attack samples, but the time to test the model is limited, and it is impossible to use an unlimited number of attack samples to test the model. How to generate diverse and targeted attack samples based on limited output information is an urgent problem to be solved in model anti-attack testing. Summary of the invention
[0003] In order to improve the efficiency of model anti-attack test, in a first aspect of the present invention, a model anti-attack test method is provided, the method comprising the following steps: Input the test sentence into the target model, obtain the label and confidence output by the model, and use the label with the highest confidence as the true label of the test sentence; divide the test sentence into multiple words, obtain the word vectors of the words, and obtain the synonym set of the word and the similarity between each synonym and the word according to the similarity of the word vectors; Counting the number of times the synonyms in the synonym set appear in each non-real label training set and the number of times they appear in the entire training set, and obtaining the importance of the synonyms in the synonym set based on the number of appearances and the similarity; The synonym set is filtered according to the importance to obtain an updated synonym set, at least one synonym set is randomly selected, and a synonym is randomly selected from the selected new synonym set to replace the corresponding word in the test sentence to obtain an attack sample; the attack sample is used to test the anti-attack ability of the model.
[0004] Optionally, obtaining the importance of synonyms in the synonym set based on the number of occurrences and the similarity is specifically: Calculate the ratio of the number of times synonyms appear in the training set of all non-real labels to the number of times they appear in the entire training set; The product or sum of the ratio and the similarity is taken as the importance of the synonyms.
[0005] Optionally, obtaining the importance of synonyms in the synonym set based on the number of occurrences and the similarity is specifically: Get the confidence of all non-real labels output by the model; Calculate the ratio of the number of times synonyms appear in the training set of each non-real label to the number of times they appear in the entire training set; The confidence of the non-real label is used to perform weighted summation on the ratio to obtain the influence of the synonym; The product or sum of the influence and the similarity is taken as the importance of the synonym.
[0006] Optionally, filtering the synonym set according to the importance to obtain an updated synonym set is specifically: sorting the synonyms in the synonym set according to the importance; Iterate over each word in the test sentence; Starting from the first synonym in the synonym set of the word, use the first synonym in the synonym set of the word to replace the word in the test sentence, generate a new sentence, input the new sentence into the target model, obtain the label with the highest confidence, if the label with the highest confidence in the new sentence is different from the true label, or the new label is the same as the true label but the confidence of the true label minus the confidence of the label with the highest confidence is greater than a preset value, then mark the synonym in the synonym set, and continue to select the next word from the synonym set of the word to replace the word in the test sentence, otherwise jump to the next word in the test sentence; Continue iterating until the last word of the test sentence; Filter out the unmarked synonyms in the synonym set to obtain an updated synonym set.
[0007] Optionally, the use of the attack sample to test the anti-attack performance of the model is specifically: The changes in the attack success rate and attack failure confidence are displayed in a graphical form.
[0008] In a second aspect of the present invention, a model anti-attack test system is provided, the system comprising the following modules: The synonym acquisition module is used to input the test sentence into the target model, obtain the label and confidence output by the model, and use the label with the highest confidence as the true label of the test sentence; divide the test sentence into multiple words, obtain the word vector of the word, and obtain the synonym set of the word and the similarity between each synonym and the word according to the similarity of the word vector; A synonym ranking module is used to count the number of times a synonym in the synonym set appears in each non-real label training set and the number of times it appears in the entire training set, and obtain the importance of the synonym in the synonym set based on the number of occurrences and the similarity; The attack test module is used to filter the synonym set according to the importance to obtain an updated synonym set, randomly select at least one synonym set, and randomly select a synonym from the selected new synonym set to replace the corresponding word in the test sentence to obtain an attack sample; and use the attack sample to test the anti-attack ability of the model.
[0009] Optionally, obtaining the importance of synonyms in the synonym set based on the number of occurrences and the similarity is specifically: Calculate the ratio of the number of times synonyms appear in the training set of all non-real labels to the number of times they appear in the entire training set; The product or sum of the ratio and the similarity is taken as the importance of the synonyms.
[0010] Optionally, obtaining the importance of synonyms in the synonym set based on the number of occurrences and the similarity is specifically: Get the confidence of all non-real labels output by the model; Calculate the ratio of the number of times synonyms appear in the training set of each non-real label to the number of times they appear in the entire training set; The confidence of the non-real label is used to perform weighted summation on the ratio to obtain the influence of the synonym; The product or sum of the influence and the similarity is taken as the importance of the synonym.
[0011] Optionally, filtering the synonym set according to the importance to obtain an updated synonym set is specifically: sorting the synonyms in the synonym set according to the importance; Iterate over each word in the test sentence; Starting from the first synonym in the synonym set of the word, use the first synonym in the synonym set of the word to replace the word in the test sentence, generate a new sentence, input the new sentence into the target model, obtain the label with the highest confidence, if the label with the highest confidence in the new sentence is different from the true label, or the new label is the same as the true label but the confidence of the true label minus the confidence of the label with the highest confidence is greater than a preset value, then mark the synonym in the synonym set, and continue to select the next word from the synonym set of the word to replace the word in the test sentence, otherwise jump to the next word in the test sentence; Continue iterating until the last word of the test sentence; Filter out the unmarked synonyms in the synonym set to obtain an updated synonym set.
[0012] Optionally, the use of the attack sample to test the anti-attack performance of the model is specifically: The changes in the attack success rate and attack failure confidence are displayed in a graphical form.
[0013] In a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.
[0014] Finally, the present invention also provides a computer program, which, when executed by a processor, implements the method described in the first aspect.
[0015] Since the model converts words into word vectors for calculation, the present invention determines the synonym set based on the similarity of word vectors rather than semantics or synonym libraries, and can better select synonyms; in addition, the similarity between synonyms and original words and the distribution of synonyms in training sets of different categories are comprehensively considered to calculate the importance of synonyms, so that words that can confuse the model are ranked higher, and by filtering the synonym set, only those more important synonyms are retained, which reduces the search space for subsequent generation of adversarial samples, improves attack efficiency, and avoids the use of some invalid or ineffective synonyms. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 This is a schematic diagram of the process of Embodiment 1; Figure 2 The output of the model for the statement to be tested is shown; Figure 3 This is a schematic diagram of the process of generating a synonym set; Figure 4 A screenshot of the attack test effect page is shown. DETAILED DESCRIPTION
[0017] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0018] Figure 1 A flow chart of a first embodiment of the present invention is shown. Figure 1 As shown, the method comprises the following steps: S1, input the test sentence into the target model, obtain the label and confidence output by the model, and use the label with the highest confidence as the true label of the test sentence; divide the test sentence into multiple words, obtain the word vectors of the words, and obtain the synonym set of the word and the similarity between each synonym and the word according to the similarity of the word vectors; In the attack test, the input is slightly disturbed to change the label output by the model, thereby achieving the purpose of deceiving the model. In order to determine whether the attack is successful, first obtain the original prediction result of the model when it is not attacked. This original prediction result is the true label. The confidence corresponding to the true label is the maximum confidence of all labels output by the model. Specifically, input the sentence that needs to be tested for offensiveness into the model, and the model will output the prediction result, such as Figure 2 As shown. The output of the model usually consists of two parts: the predicted label and the confidence. The label refers to the category to which the sentence predicted by the model belongs. For example, for a sentiment classification model, the label may be "positive", "negative" or "neutral". The higher the confidence, the more the model believes that its prediction result is correct. The confidence value ranges from 0 to 1. There is a confidence for each label. For example, the confidence of a positive label is 0.7, the confidence of a neutral label is 0.1, and the confidence of a negative label is 0.2. The model is not limited to sentiment classification models, but can also be other classification models, such as news classification, spam filtering, harmful content identification, etc.
[0019] Use a word segmentation tool such as Jieba word segmenter to decompose the test sentence into independent words, and then use a word vector model such as Word2Vec, GloVe, etc. to convert each word into a vector Token representation. In one embodiment, different word vector models are used to obtain multiple tokens of the word segmentation, and for each token of the word segmentation, a synonym subset of the word segmentation is obtained according to the similarity of the token, and all synonym subsets are added to the synonym set, such as Figure 3 As shown, the similarity between the synonyms and the participles is calculated. By calculating the similarity between word vectors, for example, using cosine similarity, other words that are close to the target word on the Token are found. For example, a participle of a test sentence is "wonderful". Word2Vec and GloVe are used to convert "wonderful" into word vectors respectively. For the two word vectors obtained, the synonyms are determined by similarity calculation methods respectively, and the synonyms are added to the synonym set. If a synonym is similar to both word vectors, the maximum similarity value is used as the similarity of the synonym. The present invention uses word vectors as a way to obtain synonyms, rather than semantics or other ways to obtain synonyms. The main purpose is to convert participles into word vectors in the model. Only when the word vectors are similar can the model be better misled, and the greater the similarity, the easier it is to make the model go wrong.
[0020] S2, counting the number of times the synonyms in the synonym set appear in each non-real label training set and the number of times they appear in the entire training set, and obtaining the importance of the synonyms in the synonym set based on the number of appearances and the similarity; In the training set, each training sample will have a label, and the training samples with the same label constitute the training set of this label. For example, if a label is "positive", all training samples with the label positive constitute the training set of the "positive" label. Non-real labels are labels other than the real label. Each label has a training set. For each synonym in the synonym set, the number of times the synonym appears in the training set of the non-real label is counted, and the number of times it appears in the entire training set is counted. Assuming that the real label of the test sentence is "positive", then the non-real label is a label that is not "positive", that is, the non-real label is "neutral" or "negative". Count the number of times a synonym appears in the training set of each non-real label and the total number of times the synonym appears in all training samples. If a synonym appears frequently in the training set of the non-real label, but appears less frequently in the training set of the real label, replacing the synonym will make it easier for the model to make prediction errors.
[0021] In an optional embodiment, obtaining the importance of synonyms in the synonym set based on the number of occurrences and the similarity is specifically as follows: Calculate the ratio of the number of times synonyms appear in the training set of all non-real labels to the number of times they appear in the entire training set; The product or sum of the ratio and the similarity is taken as the importance of the synonyms.
[0022] Count the total number of times the synonym appears in all training samples whose labels are not the true labels. For example, if the true label is "positive", count the number of times the synonym appears in all "negative" and "neutral" training samples. Count the total number of times the synonym appears in all training samples, including samples with the true label and all other labels. Divide the former by the latter to get a ratio, which is between [0,1]. The larger the ratio, the more likely the synonym appears in a category different from the true label, and the more likely it is that the model will make an incorrect prediction when it is replaced by a synonym. Multiply the above ratio with the similarity between the synonym and the segmentation in the test sentence to obtain the importance of the synonym, or add the above ratio to the similarity between the synonym and the original word to obtain the importance of the synonym. Assuming the ratio is 0.6 and the similarity between the synonym and the corresponding segmentation in the test sentence is 0.5, the importance is 0.3 or 1.1. In a more specific embodiment, the ratio and similarity are normalized respectively, and then the importance is calculated after normalization.
[0023] The model may have a low prediction confidence for some non-real labels. Even if the synonyms appear frequently in the training set of these labels, replacing them is unlikely to cause the model to make incorrect predictions. In order to more effectively select replacement words, in another embodiment, the importance of the synonyms in the synonym set is obtained based on the number of occurrences and the similarity, specifically: Get the confidence of all non-real labels output by the model; Calculate the ratio of the number of times synonyms appear in the training set of each non-real label to the number of times they appear in the entire training set; The confidence of the non-real label is used to perform weighted summation on the ratio to obtain the influence of the synonym; The product or sum of the influence and the similarity is taken as the importance of the synonym.
[0024] Specifically, after the test sentence is input into the target model, the model will output a probability distribution, indicating the possibility that the sentence belongs to each category or label, and extract the probability values corresponding to all non-real labels. These probability values are the confidence of the model for these labels. The higher the confidence, the more the model tends to think that the sentence belongs to the label. Calculate the number of times the synonym appears in the training set of each non-real label. For example, if the non-real labels are "negative" and "neutral", count the number of times the synonym appears in the "negative" training set and the "neutral" training set respectively. Then, count the total number of times the synonym appears in all training samples. For each non-real label, divide the number of times the synonym appears in the training set of the label by the number of times it appears in the entire training set to obtain a ratio. For a synonym, obtain the ratio corresponding to the training set of a non-real label, then obtain the confidence of the non-real label, multiply the two to obtain the influence of the synonym on the non-real label, and then add the influence of all non-real labels to obtain the influence of the synonym. After introducing the confidence, the model can be made more inclined to predict the wrong label through weighted summation. If the model has a high confidence in a non-real label, even if the synonym does not appear most frequently in the label training set, its influence will be amplified, thereby more accurately reflecting the potential attack effect of the synonym. After obtaining the influence, the same method as the above embodiment is adopted, that is, the product or sum of the influence and similarity is used as the importance of the synonym. In a more specific embodiment, the influence and similarity are first normalized separately, and then the importance is calculated after normalization.
[0025] S3, filtering the synonym set according to the importance to obtain an updated synonym set, randomly selecting at least one synonym set, and randomly selecting a synonym from the selected new synonym set to replace the corresponding word in the test sentence to obtain an attack sample; using the attack sample to test the anti-attack ability of the model.
[0026] Each word segment in the test sentence will have a synonym set, but the importance of different synonyms is different, and if the synonym set is too large, more attack samples will be generated later, and the test cannot be completed efficiently. In one embodiment, a threshold is set to retain synonyms whose importance is greater than the threshold, or to retain the K most important synonyms. After obtaining the filtered synonym set, at least one word is randomly selected from the test sentence, and its corresponding filtered synonym set is obtained. Only one word can be selected for replacement, or multiple words can be selected for replacement at the same time. For each selected word, a synonym is randomly selected from its corresponding filtered synonym set, and the selected synonym is used to replace the corresponding word in the test sentence, thereby generating a new sentence, i.e., an adversarial sample. In one embodiment, after the attack sample is generated, the attack sample is further disturbed, such as modifying the word order, deleting or adding some unimportant words, etc., where unimportant words include but are not limited to modal particles.
[0027] In yet another embodiment, filtering the synonym set according to the importance to obtain an updated synonym set is specifically: sorting the synonyms in the synonym set according to the importance; Iterate over each word in the test sentence; Starting from the first synonym in the synonym set of the word, use the first synonym in the synonym set of the word to replace the word in the test sentence, generate a new sentence, input the new sentence into the target model, obtain the label with the highest confidence, if the label with the highest confidence in the new sentence is different from the true label, or the new label is the same as the true label but the confidence of the true label minus the confidence of the label with the highest confidence is greater than a preset value, then mark the synonym in the synonym set, and continue to select the next word from the synonym set of the word to replace the word in the test sentence, otherwise jump to the next word in the test sentence; Continue iterating until the last word of the test sentence; Filter out the unmarked synonyms in the synonym set to obtain an updated synonym set.
[0028] For each word in the test sentence, its corresponding synonym set is sorted from high to low in importance, and then each word in the test sentence is traversed, and the synonyms in the synonym set of the word are used to replace it in turn. Specifically, according to the previously sorted order, the first synonym is selected from the synonym set for trial, and the selected synonym is used to replace the corresponding word in the test sentence to generate a new sentence. The newly generated sentence is input into the target model to obtain the label with the highest confidence predicted by the model. If the replaced sentence causes the model to predict a different label than the original sentence's true label, it means that the replacement has successfully fooled the model. Even if the model predicts the true label after the replacement, if the confidence of the real label drops significantly, that is, the drop exceeds the preset value, it is considered that the replacement has a certain effect. The preset value is a positive number. If any of the above two conditions is met, the currently attempted synonym is marked in the synonym set of the current word. The marked synonym is useful for attacking and is retained in the updated synonym set. Then, continue to try the next synonym in the synonym set of the word. If neither condition is met, it means that the current synonym is not a perfect synonym. If the replacement of a word does not achieve the expected effect, no further attempts will be made for the remaining important synonyms in the synonym set, and the next word in the test sentence will be directly jumped to continue the replacement attempt. The above process is repeated until all the words in the test sentence have been tried. After traversing all the words, the synonyms that are not marked in the synonym set are filtered out, and the remaining is the updated synonym set. These synonyms are all verified by the model and are more likely to be effectively used to generate adversarial samples. In this way, more aggressive synonyms can be effectively screened out, improving the efficiency and success rate of adversarial attacks. Since the selection is made directly based on the feedback of the target model, it is more effective than simply performing Top K selection or threshold filtering based on importance.
[0029] The generated adversarial sample is input into the model, and the prediction result of the model for the attack sample is compared with the prediction result for the original sentence. If the prediction result of the model for the attack sample is different from the true label, the attack is considered successful. The number of samples with successful attacks is counted, and the attack success rate is calculated to evaluate the model's anti-attack ability. In one embodiment, the changes in the attack success rate and the attack failure confidence are displayed in the form of a chart, Figure 4 A screenshot of an attack test effect page is shown.
[0030] In a second aspect of the present invention, a model anti-attack test system is provided, the system comprising the following modules: The synonym acquisition module is used to input the test sentence into the target model, obtain the label and confidence output by the model, and use the label with the highest confidence as the true label of the test sentence; divide the test sentence into multiple words, obtain the word vector of the word, and obtain the synonym set of the word and the similarity between each synonym and the word according to the similarity of the word vector; A synonym ranking module is used to count the number of times a synonym in the synonym set appears in each non-real label training set and the number of times it appears in the entire training set, and obtain the importance of the synonym in the synonym set based on the number of occurrences and the similarity; The attack test module is used to filter the synonym set according to the importance to obtain an updated synonym set, randomly select at least one synonym set, and randomly select a synonym from the selected new synonym set to replace the corresponding word in the test sentence to obtain an attack sample; and use the attack sample to test the anti-attack ability of the model.
[0031] Optionally, obtaining the importance of synonyms in the synonym set based on the number of occurrences and the similarity is specifically: Calculate the ratio of the number of times synonyms appear in the training set of all non-real labels to the number of times they appear in the entire training set; The product or sum of the ratio and the similarity is taken as the importance of the synonyms.
[0032] Optionally, obtaining the importance of synonyms in the synonym set based on the number of occurrences and the similarity is specifically: Get the confidence of all non-real labels output by the model; Calculate the ratio of the number of times synonyms appear in the training set of each non-real label to the number of times they appear in the entire training set; The confidence of the non-real label is used to perform weighted summation on the ratio to obtain the influence of the synonym; The product or sum of the influence and the similarity is taken as the importance of the synonym.
[0033] Optionally, filtering the synonym set according to the importance to obtain an updated synonym set is specifically: sorting the synonyms in the synonym set according to the importance; Iterate over each word in the test sentence; Starting from the first synonym in the synonym set of the word, use the first synonym in the synonym set of the word to replace the word in the test sentence, generate a new sentence, input the new sentence into the target model, obtain the label with the highest confidence, if the label with the highest confidence in the new sentence is different from the true label, or the new label is the same as the true label but the confidence of the true label minus the confidence of the label with the highest confidence is greater than a preset value, then mark the synonym in the synonym set, and continue to select the next word from the synonym set of the word to replace the word in the test sentence, otherwise jump to the next word in the test sentence; Continue iterating until the last word of the test sentence; Filter out the unmarked synonyms in the synonym set to obtain an updated synonym set.
[0034] Optionally, the use of the attack sample to test the anti-attack performance of the model is specifically: The changes in the attack success rate and attack failure confidence are displayed in a graphical form.
[0035] In a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.
[0036] Finally, the present invention also provides a computer program, which, when executed by a processor, implements the method described in the first aspect.
[0037] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0038] As another aspect, the present invention further provides a computer-readable medium, wherein the computer-readable medium carries one or more programs, and when the one or more programs are executed by a device or a processor, the method described in the first embodiment is implemented.
[0039] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A model anti-attack testing method, characterized in that: The method comprises the following steps: Input the test sentence into the target model, obtain the label and confidence output by the model, and use the label with the highest confidence as the true label of the test sentence; divide the test sentence into multiple words, obtain the word vectors of the words, and obtain the synonym set of the word and the similarity between each synonym and the word according to the similarity of the word vectors; Counting the number of times the synonyms in the synonym set appear in each non-real label training set and the number of times they appear in the entire training set, and obtaining the importance of the synonyms in the synonym set based on the number of appearances and the similarity; The synonym set is filtered according to the importance to obtain an updated synonym set, at least one synonym set is randomly selected, and a synonym is randomly selected from the selected new synonym set to replace the corresponding word in the test sentence to obtain an attack sample; the attack sample is used to test the anti-attack ability of the model.
2. The method according to claim 1, characterized in that The importance of the synonyms in the synonym set is obtained based on the number of occurrences and the similarity, specifically: Calculate the ratio of the number of times synonyms appear in the training set of all non-real labels to the number of times they appear in the entire training set; The product or sum of the ratio and the similarity is taken as the importance of the synonyms.
3. The method according to claim 1, characterized in that The importance of the synonyms in the synonym set is obtained based on the number of occurrences and the similarity, specifically: Get the confidence of all non-real labels output by the model; Calculate the ratio of the number of synonyms appearing in the training set of each non-real label to the number of synonyms appearing in the entire training set; The confidence of the non-real label is used to perform weighted summation on the ratio to obtain the influence of the synonym; The product or sum of the influence and the similarity is taken as the importance of the synonym.
4. The method according to claim 1, characterized in that The filtering of the synonym set according to the importance to obtain an updated synonym set is specifically: sorting the synonyms in the synonym set according to the importance; Iterate over each word in the test sentence; Starting from the first synonym in the synonym set of the word, use the first synonym in the synonym set of the word to replace the word in the test sentence, generate a new sentence, input the new sentence into the target model, obtain the label with the highest confidence, if the label with the highest confidence in the new sentence is different from the true label, or the new label is the same as the true label but the confidence of the true label minus the confidence of the label with the highest confidence is greater than a preset value, then mark the synonym in the synonym set, and continue to select the next word from the synonym set of the word to replace the word in the test sentence, otherwise jump to the next word in the test sentence; Continue iterating until the last word of the test sentence; Filter out the unmarked synonyms in the synonym set to obtain an updated synonym set.
5. The method according to claim 1, characterized in that The attack sample is used to test the anti-attack performance of the model, specifically: The changes in the attack success rate and attack failure confidence are displayed in a graphical form.
6. A model anti-attack test system, characterized in that: The system includes the following modules: The synonym acquisition module is used to input the test sentence into the target model, obtain the label and confidence output by the model, and use the label with the highest confidence as the true label of the test sentence; Divide the test sentence into multiple words, obtain the word vectors of the words, and obtain the synonym set of the word and the similarity between each synonym and the word according to the similarity of the word vectors; A synonym ranking module is used to count the number of times a synonym in the synonym set appears in each non-real label training set and the number of times it appears in the entire training set, and obtain the importance of the synonym in the synonym set based on the number of occurrences and the similarity; The attack test module is used to filter the synonym set according to the importance to obtain an updated synonym set, randomly select at least one synonym set, and randomly select a synonym from the selected new synonym set to replace the corresponding word in the test sentence to obtain an attack sample; and use the attack sample to test the anti-attack ability of the model.
7. The system according to claim 6, characterized in that The importance of the synonyms in the synonym set is obtained based on the number of occurrences and the similarity, specifically: Calculate the ratio of the number of times synonyms appear in the training set of all non-real labels to the number of times they appear in the entire training set; The product or sum of the ratio and the similarity is taken as the importance of the synonyms.
8. The system according to claim 6, characterized in that The importance of the synonyms in the synonym set is obtained based on the number of occurrences and the similarity, specifically: Get the confidence of all non-real labels output by the model; Calculate the ratio of the number of synonyms appearing in the training set of each non-real label to the number of synonyms appearing in the entire training set; The confidence of the non-real label is used to perform weighted summation on the ratio to obtain the influence of the synonym; The product or sum of the influence and the similarity is taken as the importance of the synonym.
9. The system according to claim 6, characterized in that The filtering of the synonym set according to the importance to obtain an updated synonym set is specifically: sorting the synonyms in the synonym set according to the importance; Iterate over each word in the test sentence; Starting from the first synonym in the synonym set of the word, use the first synonym in the synonym set of the word to replace the word in the test sentence, generate a new sentence, input the new sentence into the target model, obtain the label with the highest confidence, if the label with the highest confidence in the new sentence is different from the true label, or the new label is the same as the true label but the confidence of the true label minus the confidence of the label with the highest confidence is greater than a preset value, then mark the synonym in the synonym set, and continue to select the next word from the synonym set of the word to replace the word in the test sentence, otherwise jump to the next word in the test sentence; Continue iterating until the last word of the test sentence; Filter out the unmarked synonyms in the synonym set to obtain an updated synonym set.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.