Attack defense methods, devices, electronic devices and computer-readable storage media

Abstract

This disclosure provides an attack defense method, apparatus, electronic device, and computer-readable storage medium. The attack defense method includes: acquiring a text to be predicted that belongs to a category to be protected and a confidence level of the text belonging to the category to be protected; acquiring a category perturbation term corresponding to the category to be protected; perturbing the text to be predicted using the category perturbation term to obtain a perturbed text to be predicted; obtaining a confidence level of the perturbed text to be predicted that belongs to the category to be protected; and determining whether the text to be predicted is risky or risk-free based on the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the perturbed text to be predicted belonging to the category to be protected. Embodiments of this disclosure can accurately determine whether the text to be predicted is risky or risk-free.

Description

Technical Field

[0001] This disclosure relates to the field of computer and Internet technology, and in particular to an attack defense method and apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] With the development of the internet, the number of training parameters and training datasets for text classification models are increasing, making it extremely difficult to train a text classification model from scratch. Therefore, a common approach is to purchase or download a pre-trained system from a third-party system developer for use.

[0003] However, some recent studies have pointed out that some third-party system developers may implant backdoors during the model training process. Once the system deployer goes live with the model purchased from them, they can use the backdoor trigger to activate this hidden backdoor and cause attacks and harm to the system deployer.

[0004] For text classification models, a backdoor trigger might be a piece of text. Therefore, when using a text classification model purchased from a third party to classify the text to be predicted, it becomes very important to identify risky text that may contain backdoor triggers.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0006] The purpose of this disclosure is to provide an attack defense method, apparatus, electronic device, and computer-readable storage medium that can accurately identify risky text in the text to be predicted, so as to defend against attacks on the text classification model based on the risky text.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0008] This disclosure provides an attack defense method, comprising: obtaining a text to be predicted that belongs to a category to be protected and the confidence level of the text to be predicted belonging to the category to be protected; obtaining a category perturbation item corresponding to the category to be protected; perturbing the text to be predicted using the category perturbation item to obtain a text to be predicted that is perturbed; obtaining the confidence level of the text to be predicted that is perturbed belonging to the category to be protected; and determining whether the text to be predicted is risky or risk-free based on the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted that is perturbed belonging to the category to be protected.

[0009] This disclosure provides an attack defense device, including: a first confidence level acquisition module, a category perturbation item acquisition module, a perturbation text generation module to be predicted, a second confidence level acquisition module, and a risk prediction module.

[0010] The first confidence level acquisition module is used to acquire the text to be predicted that belongs to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected; the category perturbation item acquisition module is used to acquire the category perturbation item corresponding to the category to be protected; the text to be predicted perturbation generation module is used to perturb the text to be predicted by the category perturbation item to obtain the text to be predicted perturbation; the second confidence level acquisition module is used to acquire the confidence level of the text to be predicted belonging to the category to be protected; and the risk prediction module is used to determine whether the text to be predicted is a risky text or a risk-free text based on the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted perturbation belonging to the category to be protected.

[0011] In some embodiments, the risk prediction module includes: a risk-free text determination submodule and a risk text determination submodule.

[0012] Specifically, the risk-free text determination submodule is configured to determine the text to be predicted as risk-free if the difference between the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected is greater than or equal to a confidence decrease threshold; and the risk text determination submodule is configured to determine the text to be predicted as risky if the difference between the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected is less than the confidence decrease threshold.

[0013] In some embodiments, the first confidence acquisition module includes a text classification model acquisition submodule and a first confidence determination submodule, and the second confidence acquisition module includes a second confidence determination submodule.

[0014] The text classification model acquisition submodule is used to acquire a text classification model; the first confidence determination submodule is used to input the text to be predicted into the text classification model and determine the confidence level of the text to be predicted belonging to the category to be protected; the second confidence determination submodule is used to input the perturbation text to be predicted into the text classification model and determine the confidence level of the perturbation text to be predicted belonging to the category to be protected.

[0015] In some embodiments, the attack defense device includes: a training sample acquisition module, a category perturbation term determination module, a training perturbation text acquisition module, a third confidence determination module, a fourth confidence acquisition module, a target loss value determination module, and an update module.

[0016] The system comprises the following modules: a training sample acquisition module for acquiring training text labeled as the category to be protected; a category perturbation term determination module for determining the category perturbation term corresponding to the category to be protected based on the training text; a training perturbation text acquisition module for perturbing the training text with the target category perturbation term to obtain training perturbation text; a third confidence determination module for classifying the training text using the training network model to determine the confidence level of the training text belonging to the category to be protected; a fourth confidence acquisition module for classifying the training perturbation text using the training network model to determine the confidence level of the training perturbation text belonging to the category to be protected; a target loss value determination module for determining the target loss value of the training network model based on the confidence level of the training text belonging to the category to be protected and the confidence level of the training perturbation text belonging to the category to be protected; and an update module for updating the word vector parameters of the category perturbation term in the training network model based on the target loss value to obtain the text classification model.

[0017] In some embodiments, the target loss value determination module includes: a confidence difference determination submodule, a first difference determination submodule, a first target value determination submodule, a second difference determination submodule, a second target value determination submodule, and a target loss value determination submodule.

[0018] The confidence difference determination submodule is used to determine the confidence difference between the confidence that the training text belongs to the category to be protected and the confidence that the training perturbation text belongs to the category to be protected; the first difference determination submodule is used to perform subtraction on the confidence difference using a first value to obtain a first difference; the first target value determination submodule is used to determine a maximum value between the first difference and a third value as a first target value; the second difference determination submodule is used to perform subtraction on a second value using the confidence difference to obtain a second difference; the second target value determination submodule is used to determine a maximum value between the second difference and the third value as a second target value; and the target loss value determination submodule is used to determine the target loss value based on the first target value and the second target value.

[0019] In some embodiments, the text to be trained includes multiple texts; the attack defense device further includes: a total perturbation processing module, a probability difference determination module, a sorting module, and a percentile determination module.

[0020] The perturbation processing module is used to perturb each training text separately using the category perturbation item to obtain the training perturbation text corresponding to each training text; the probability difference determination module is used to classify each training text and the corresponding training perturbation text separately using the training network model to determine the probability difference of each training text before and after perturbation in the category to be protected; the sorting module is used to sort the probability differences of each training text before and after perturbation to obtain the probability difference sorting result; the percentile determination module is used to determine a confidence decrease threshold in the difference sorting result based on the target percentile, so as to perform attack defense on the text to be predicted based on the confidence decrease threshold, where the target percentile is a pre-set probability that the text classification model is allowed to misdetect risk-free text as risky text.

[0021] In some embodiments, the network model to be trained includes multiple network layers, the multiple network layers include an embedding layer, the embedding layer includes word vector parameters corresponding to the category perturbation term; wherein, the update module includes: a training gradient value determination submodule, a target gradient value determination submodule, and a parameter modification submodule.

[0022] The training gradient value determination submodule is used to perform backpropagation processing on the network model to be trained based on the target loss value to determine the backpropagation gradient value of each parameter in each network layer; the target gradient value determination submodule is used to determine the target gradient value of the word vector parameter corresponding to the category perturbation term in the embedding layer from the backpropagation gradient values ​​in each network layer; the parameter modification submodule is used to backpropagate the word vector parameter corresponding to the category perturbation term using the target gradient value, while keeping the parameters in the network model to be trained unchanged except for the word vector parameter corresponding to the category perturbation term, so as to obtain the text classification model.

[0023] In some embodiments, the parameter changing submodule further includes: a first perturbation word vector determination unit, a second perturbation word vector determination unit, and a norm control unit.

[0024] The first perturbation word vector determination unit is used to obtain a first perturbation word vector obtained by processing the category perturbation item with the word vector parameters corresponding to the category perturbation item before reverse training; the second perturbation word vector determination unit is used to obtain a second perturbation word vector obtained by processing the category perturbation item with the word vector parameters corresponding to the category perturbation item after reverse training; the norm control unit is used to perform norm control processing on the second perturbation word vector according to the first perturbation word vector, so that the norm of the second perturbation word vector is consistent with the norm of the first perturbation word vector, so as to obtain the text classification model.

[0025] In some embodiments, the category perturbation item determination module includes: a low-frequency word determination submodule and a relevance determination submodule.

[0026] The low-frequency word determination submodule is used to determine multiple low-frequency words in the text to be trained that appear less than a frequency threshold; the relevance determination submodule is used to determine a target low-frequency word among the low-frequency words that has a relevance to the text to be trained that is lower than a similarity threshold, and to use the target low-frequency word as the category perturbation term.

[0027] In some embodiments, the target category to be protected includes a first category and a second category, and the category perturbation item includes a first category perturbation item corresponding to the first category and a second category perturbation item corresponding to the second category.

[0028] This disclosure provides an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the attack defense method described above.

[0029] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the attack defense method as described in any of the preceding embodiments.

[0030] This disclosure provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned attack defense method.

[0031] The attack defense method, apparatus, electronic device, and computer-readable storage medium provided in this disclosure accurately determine whether a text to be predicted is a risky text by determining whether the text to be predicted belongs to a protected category and whether the text to be predicted, which is a perturbation text to be predicted, belongs to a protected category, so as to perform attack defense based on the risk assessment result for the text to be predicted.

[0032] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0034] Figure 1 A schematic diagram of an exemplary system architecture applied to an attack defense method or attack defense device according to embodiments of this disclosure is shown.

[0035] Figure 2 This is a flowchart illustrating an attack defense method according to an exemplary embodiment.

[0036] Figure 3 This is a training method for a text classification model according to an exemplary embodiment.

[0037] Figure 4 This is a method for determining a target loss value according to an exemplary embodiment.

[0038] Figure 5 This is a reverse training method illustrated according to an exemplary embodiment.

[0039] Figure 6 This is a structural diagram illustrating an attack defense method according to an exemplary embodiment.

[0040] Figure 7 This is a block diagram illustrating an attack defense device according to an exemplary embodiment.

[0041] Figure 8 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0042] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0043] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0044] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0045] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0046] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of one or more elements / components / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0047] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0048] Figure 1 A schematic diagram of an exemplary system architecture that can be applied to the attack defense method or attack defense device in the embodiments of this disclosure is shown.

[0049] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0050] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0051] For example, a user can use terminals 101, 102, or 103 to obtain the text to be predicted that belongs to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected; obtain the category perturbation item corresponding to the category to be protected; perturb the text to be predicted by the category perturbation item to obtain the text to be predicted by perturbation; obtain the confidence level of the text to be predicted by perturbation belonging to the category to be protected; and determine whether the text to be predicted is risky or risk-free based on the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted by perturbation belonging to the category to be protected.

[0052] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0053] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0054] Server 105 may, for example, obtain the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected; server 105 may, for example, obtain the category perturbation item corresponding to the category to be protected; server 105 may, for example, perturb the text to be predicted through the category perturbation item to obtain the text to be predicted perturbed; server 105 may, for example, obtain the confidence level of the text to be predicted perturbed belonging to the category to be protected; server 105 may, for example, determine whether the text to be predicted is risky or risk-free based on the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted perturbed belonging to the category to be protected.

[0055] It should be noted that the server can work in conjunction with the aforementioned terminal devices to implement the attack defense method provided in this disclosure, or it can work independently. This disclosure does not impose any restrictions on this. Multiple servers and terminal devices can form a blockchain, and the servers or terminal devices can be nodes on the blockchain.

[0056] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0057] It should be understood that any number of elements in the accompanying drawings of this disclosure is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning. For ease of understanding, the terms used in the embodiments of this disclosure are explained below.

[0058] Text classification model: A neural network model whose input is a text and whose output is the probability that the text belongs to a preset category.

[0059] Word segmentation: dividing a piece of text into individual words (or phrases) according to certain rules.

[0060] Word vectors: The input to a text classification model is a text (or words). After word segmentation, a sequence of words is formed. Each word corresponds to a unique feature vector in the model, which serves as the feature replacement for the input model. These vectors are called word vectors.

[0061] System developer: A person who develops a system, such as someone who designs, trains, and open-sources a text classification model.

[0062] System deployer: The person who downloads and deploys the pre-trained model provided by the system developer for downstream tasks.

[0063] System users: System users provide input and expect to get the correct output from the model.

[0064] Backdoor: A backdoor refers to a "Trojan horse" or "virus" that malicious system developers implant into a model during training. Normally, the harm of this virus will not be apparent; however, when certain specific signals are input, the backdoor is activated and harms the model's performance.

[0065] Risk model: A text classification model with backdoors.

[0066] Risk-free model: A text classification model without backdoors.

[0067] Backdoor trigger: A special signal used by system developers to trigger the backdoors they have hidden. For text classification models, the input is a piece of text, so the backdoor trigger can be a word, sentence, or symbol, etc.

[0068] Target category: Target label. For text classification models, the model's output can be labels for different categories. Malicious system developers aim to cause the model to classify all inputs into this target category when a backdoor trigger is present and activated, regardless of the input's true category.

[0069] The category to be protected: the protected label, is the category that the system deployer wants to protect from attacks, and it is likely the target category that the attacker wants to attack.

[0070] Model output confidence: The probability of a model outputting a certain class. For example, in a binary classification model, the output confidence for one class is a probability between 0 and 1. The higher the probability, the greater the probability that the sample belongs to that class.

[0071] Risk text: Sample containing backdoor triggers.

[0072] Risk-free samples: Samples that do not contain backdoor triggers.

[0073] Online backdoor defense technology refers to the deployment of a system by the deployer after obtaining a trained model from the system developer. However, it is unclear whether the model contains backdoors. Therefore, the goal is to detect and reject risky text containing backdoor triggers as much as possible during the online operation phase, while minimizing the number of risk-free samples that are mistakenly detected as risky text to reduce the adverse impact on users.

[0074] Abbreviations:

[0075] NLP: Natural Language Processing.

[0076] Clean Acc: Clean Accuracy refers to the classification accuracy of a backdoor / risk-free model on risk-free samples.

[0077] ASR: Attack Success Rate, which is the probability that a text classification model will classify a sample containing a backdoor trigger into the target category.

[0078] FRR: False Rejection Rate, the probability of falsely rejecting a risk-free sample, i.e., the probability that a defense mechanism incorrectly detects a risk-free sample as risky text.

[0079] FAR: False Acceptance Rate, the probability of incorrectly accepting risky text, that is, the probability that the defense mechanism incorrectly detects risky text as a risk-free sample.

[0080] RAP: Robustness-Aware Perturbation, a robustness-based perturbation term used in this disclosure to distinguish risky text from risk-free samples.

[0081] In order to accurately and efficiently identify which texts are risky and which are risk-free when making predictions, this disclosure provides the following technical solutions.

[0082] Figure 2 This is a flowchart illustrating an attack defense method according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be executed by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0083] Reference Figure 2 The attack defense method provided in this disclosure may include the following steps.

[0084] Step S202: Obtain the text to be predicted that belongs to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected.

[0085] The category to be protected can be any one or more categories in the text classification model, and this disclosure does not impose any restrictions on it. The text classification model can be a binary classification model or a multi-class classification model, and this disclosure does not impose any restrictions on it.

[0086] The text classification model can be generated by retraining a model obtained from a third party. The training process will be described in other embodiments below, and will not be repeated in this embodiment.

[0087] The text to be predicted is the text that needs to be input into the text classification model to predict its category.

[0088] In some embodiments, the confidence level of the text to be predicted belonging to the category to be protected can be obtained by: obtaining a text classification model; inputting the text to be predicted into the text classification model, and determining the confidence level of the text to be predicted belonging to the category to be protected.

[0089] In other embodiments, the confidence level of the predicted text belonging to the category to be protected can also be determined by the correlation between the predicted text and the category to be protected. For example, the correlation between the predicted text and the category to be protected can be determined by the textual correlation, semantic correlation, etc. between the predicted text and the text in the category to be protected. The method for determining the confidence level of the predicted text belonging to the category to be protected in this disclosure is not limited.

[0090] Step S204: Obtain the category perturbation term corresponding to the category to be protected.

[0091] The aforementioned perturbation items can be text, words, punctuation marks, etc., and this disclosure does not impose any restrictions on them.

[0092] In some embodiments, the category perturbation term corresponding to the category to be protected may be a perturbation term with low correlation to the category to be protected, or a perturbation term with low correlation to the text of the input text classification model (which may be a text correlation or a semantic correlation, etc.), that is, the category perturbation term has a small impact on the semantics of the text to be predicted.

[0093] In some embodiments, the category perturbation term of the category to be protected can be determined by the following method: obtaining a text set that may be input to the text classification model, such as a validation dataset provided by a third party that provides the original training model for the text classification model; identifying some low-frequency words that appear less than the target value in the text set; and then selecting words with low relevance to the text set from the low-frequency words as perturbation terms of the category to be protected.

[0094] It is understood that the relationship between the category to be protected and the category protection item can be one-to-one or many-to-many, and this disclosure does not impose any restrictions on this.

[0095] Step S206: Perturb the text to be predicted by the category perturbation term to obtain the perturbated text to be predicted.

[0096] In some embodiments, a category perturbation term may be inserted into the text to be predicted to perturb the text, for example, by inserting the category perturbation term at the beginning or end of the text to be predicted. This disclosure does not limit the method of inserting the category perturbation term.

[0097] Step S208: Obtain the confidence level that the perturbation text to be predicted belongs to the category to be protected.

[0098] In some embodiments, the perturbation text to be predicted can be input into a text classification model to determine the confidence level that the perturbation text to be predicted belongs to the category to be protected.

[0099] In other embodiments, the confidence level of the predicted perturbation text belonging to the category to be protected can also be determined by the correlation between the predicted perturbation text and the category to be protected. For example, the correlation between the predicted perturbation text and the category to be protected can be determined by the textual correlation, semantic correlation, etc. between the predicted perturbation text and the text in the category to be protected.

[0100] Step S210: Based on the confidence level that the text to be predicted belongs to the category to be protected and the confidence level that the disturbed text to be predicted belongs to the category to be protected, determine whether the text to be predicted is a risky text or a risk-free text.

[0101] In some embodiments, the confidence level of the predicted perturbation text belonging to the category to be protected and the confidence level difference between the two can be determined. If the confidence level difference between the two is greater than or equal to the confidence level decrease threshold δ, the text to be predicted is risk-free text; if the confidence level difference between the two is less than the confidence level decrease threshold δ, the text to be predicted is risky text.

[0102] In other embodiments, the confidence ratio between the confidence level of the predicted perturbation text belonging to the category to be protected and the confidence level of the predicted text belonging to the category to be protected can also be determined. If the confidence ratio between the confidence level of the predicted perturbation text belonging to the category to be protected and the confidence level of the predicted text belonging to the category to be protected is greater than a ratio threshold, then the predicted text is risk-free text; if the confidence ratio between the confidence level of the predicted perturbation text belonging to the category to be protected and the confidence level of the predicted text belonging to the category to be protected is less than or equal to the ratio threshold, then the predicted text is risky text.

[0103] The technical solution provided in this embodiment can not only defend against backdoor attacks where the trigger is a rare word, but also defend against attacks where the trigger is a sentence. On the other hand, when judging risky text, only two predictions are needed, without repeatedly operating and predicting for the same input. Therefore, it saves a lot of computing resources and improves the efficiency of judging risky text during the detection stage.

[0104] This disclosure presents an online attack defense technique for text classification models, aiming to address the risk of attacks due to backdoors embedded in models obtained from third parties. When the input is a risk-free sample, the technical solution provided by the embodiments of this disclosure will not detect it as risky text; however, when the input is risky text containing backdoor triggers, the technical solution provided by the embodiments of this disclosure can detect these risky texts, achieving the defense objective.

[0105] Figure 3 This is a training method for a text classification model according to an exemplary embodiment.

[0106] refer to Figure 3 The training method for the above text classification model may include the following steps.

[0107] Step S302: Obtain the training text whose text label is the category to be protected.

[0108] In some embodiments, while obtaining the model to be trained from a third party, some datasets with known labels as the category to be protected can be obtained, or some datasets with known labels as the category to be protected can be collected from the Internet, or some datasets with known labels as the category to be protected can be collected through other channels as the training text mentioned above.

[0109] Step S304: Determine the category perturbation term corresponding to the category to be protected based on the text to be trained.

[0110] In some embodiments, the category perturbation item corresponding to the category to be protected can be any content item that will not affect the text semantics of the text to be trained, such as a perturbation item with low relevance to the category to be protected or a perturbation item with low relevance to the text to be trained (which can be a textual relevance or a semantic relevance, etc.).

[0111] In some embodiments, the category perturbation term of the category to be protected can be determined by the following method: identifying multiple low-frequency words in the text to be trained that appear less than a frequency threshold; identifying a target low-frequency word among the low-frequency words that has a relevance to the text to be trained that is lower than a similarity threshold, and using the target low-frequency word as the category perturbation term.

[0112] Step S306: Perturb the training text by using the target category perturbation term to obtain the perturbation text to be trained.

[0113] In some embodiments, a category perturbation term may be inserted into the text to be trained to perturb the text, for example, by inserting the category perturbation term at the beginning or end of the text to be trained. This disclosure does not limit the method of inserting the category perturbation term.

[0114] Step S308: The training text is classified using the network model to be trained in order to determine the confidence level of whether the training text belongs to the category to be protected.

[0115] Among them, the network model to be trained can be some pre-trained risky models that may contain "backdoors".

[0116] In some embodiments, the text to be trained can be input into a text classification model to determine the confidence level that the text belongs to the category to be protected.

[0117] Step S310: The training perturbation text is classified using the network model to be trained in order to determine the confidence level of whether the training perturbation text belongs to the category to be protected.

[0118] In some embodiments, the perturbation text to be trained can be input into a text classification model to determine the confidence level that the perturbation text to be trained belongs to the category to be protected.

[0119] Step S312: Determine the target loss value of the network model to be trained based on the confidence level of the training text belonging to the category to be protected and the confidence level of the training perturbation text belonging to the category to be protected.

[0120] Step S314: Update the word vector parameters of the category perturbation term in the network model to be trained according to the target loss value to obtain the text classification model.

[0121] The technical solution provided in this embodiment aims to propose an efficient text classification model attack defense technology. This technical solution only requires special modification to a small number of word vector parameters (i.e., word vector parameters corresponding to category perturbation terms) in the text classification model. It will not affect the normal operation of the text classification model after it is launched, and can also efficiently detect risky samples containing backdoor triggers during deployment. Furthermore, it only requires two classifications of the same sample to be predicted to determine whether it contains a backdoor trigger, thus saving a lot of computing resources.

[0122] Figure 4 This is a method for determining a target loss value according to an exemplary embodiment.

[0123] refer to Figure 4 The method for determining the target loss value may include the following steps.

[0124] Step S402: Determine the confidence difference between the confidence level of the training text belonging to the category to be protected and the confidence level of the training perturbation text belonging to the category to be protected.

[0125] In some embodiments, the confidence level f of the training text belonging to the category to be protected can be determined first. θ (x;y T ) and the perturbation text f to be trained θ (x+t;y T The confidence difference f between the confidence levels of the categories to be protected θ (x;y T )-f θ (x+t;y T ).

[0126] Among them, f θ (x;y T () is a text classification model that determines the protection category y of the input training text x. T The confidence probability on f θ (x+t;y T () is the text classification model's assessment of the input perturbation text x+t in the category y to be protected. T The confidence level probability.

[0127] Step S404: Use the first value to perform subtraction on the confidence difference to obtain the first difference value.

[0128] The first value can be a pre-defined lower bound c for the confidence level decrease. low .

[0129] In some embodiments, the first value c can be used. low For confidence difference f θ (x;y T )-f θ (x+t;y T Perform a subtraction operation to obtain the first difference value c. low -f θ (x;y T )+f θ (x+t;y T ).

[0130] Step S406: Determine a maximum value between the first difference and the third value as the first target value.

[0131] The third value can be a pre-set value, such as 0.

[0132] Step S408: The second value is subtracted using the confidence difference to obtain the second difference value.

[0133] The second value can be a pre-defined upper bound c for the confidence level decrease. up .

[0134] In some embodiments, confidence difference f can be used. θ (x;y T )-f θ (x+t;y T For the second value c up Perform a subtraction operation to obtain the second difference value f. θ (x;y T )-f θ (x+t;y T )-c up .

[0135] Step S410: Determine a maximum value between the second difference and the third value as the second target value.

[0136] Step S412: Determine the target loss value based on the first target value and the second target value.

[0137] In some embodiments, the target loss value can be determined according to formula (1).

[0138]

[0139] Where y T It is a category to be protected, f θ (x;y T () is a text classification model that determines the protection category y of the input training text x. T The confidence probability on f θ (x+t;y T () is a text classification model that determines the protection category y of the input training text x. T The confidence probability on [x] + =max{z,x}. ​​z is a set third value, for example, it can be 0. Where c... low It is the lower bound of the decrease in confidence level of the disturbance term, c up This is the upper bound of the confidence decrease. During training, we hope that inserting this perturbation word will reduce the confidence of risk-free samples by at least c. low However, it cannot drop too much beyond c. up Otherwise, this disturbance may cause a decrease in confidence for risky texts, at least c lowλ is an adjustable hyperparameter used to adjust the weights of the two loss components, typically set to 1. After calculating the loss, gradient backpropagation is performed. However, this disclosure only uses gradient descent to update the word vector parameters of the selected low-frequency words, while maintaining the updated word vector norm unchanged; otherwise, the word vector norm would become increasingly larger. Updates continue until the word vector parameters converge.

[0140] The technical solution provided in this embodiment trains a text classification model using a target loss value. This ensures that after the trained model classifies a given sample and its corresponding perturbation sample, the confidence level of the perturbation sample belonging to the protected category is significantly lower than the confidence level of the original sample belonging to the protected category (i.e., the decrease should be greater than the first value), but not excessively large (i.e., the decrease should not be greater than the second value). Since text classification model developers design backdoors with the intention of creating strong backdoors, ensuring a 100% success rate for future attacks, the probability of a risky sample belonging to the protected category is not significantly altered after adding a category perturbation term to the risky sample.

[0141] Based on the above, when classifying the text to be predicted and the text to be predicted as perturbation, if the text to be predicted is a risk-free text, the confidence level of the text to be predicted as perturbation belonging to the category to be protected will change significantly compared to the confidence level of the text to be predicted as perturbation belonging to the category to be protected. If the text to be predicted is a risky text, the confidence level of the text to be predicted as perturbation belonging to the category to be protected will not change significantly compared to the confidence level of the text to be predicted as perturbation belonging to the category to be protected.

[0142] In other embodiments, a method for determining a confidence decline threshold is also provided, specifically including the following methods: perturbing each training text in multiple training texts using a category perturbation term to obtain training perturbed texts corresponding to each training text; classifying each training text and its corresponding training perturbed text using a training network model to determine the probability difference between each training text before and after perturbation belonging to the category to be protected; sorting the probability differences of each training text before and after perturbation to obtain a probability difference sorting result; determining a confidence decline threshold based on a target percentile in the difference sorting result, and using the confidence decline threshold to defend against attacks on the text to be predicted, where the target percentile is a pre-set probability that the text classification model is allowed to misdetect a risk-free text as a risky text.

[0143] Figure 5 This is a reverse training method illustrated according to an exemplary embodiment.

[0144] The network model to be trained includes multiple network layers, and each network layer includes an embedding layer, which includes word vector parameters corresponding to category perturbation terms.

[0145] refer to Figure 5 The above-mentioned reverse training method may include the following steps.

[0146] Step S502: Perform backpropagation on the network model to be trained based on the target loss value to determine the backpropagation gradient values ​​of each parameter in each network layer.

[0147] Step S504: Determine the target gradient value of the word vector parameter corresponding to the category perturbation term in the embedding layer from the back training gradient values ​​in each network layer.

[0148] Step S506: Update the word vector parameters corresponding to the category perturbation term in reverse using the target gradient value, and keep the parameters in the network model to be trained unchanged except for the word vector parameters corresponding to the category perturbation term, so as to obtain a text classification model.

[0149] Understandably, the embedding layer sets a word vector parameter for each phrase (which may include symbols) so that the phrase can be converted into a word vector based on the word vector parameter.

[0150] This embodiment only performs reverse updates on the word vector parameters corresponding to the perturbation terms in order to perform transfer training on the text to be trained.

[0151] In addition, while updating the word vector parameters corresponding to the category perturbation terms in reverse, the norm of the word vectors corresponding to the category perturbation terms before and after the reverse update can be kept unchanged, so as to minimize the storage space of the category perturbation terms and reduce the impact of changes in the word vectors of the category perturbation terms on the text classification performance of the text classification model.

[0152] Maintaining the norm of the word vectors corresponding to the category perturbation terms before and after the reverse update can include the following steps:

[0153] Obtain the first perturbation word vector after processing the category perturbation term with the word vector parameters corresponding to the category perturbation term before back-training; obtain the second perturbation word vector after processing the category perturbation term with the word vector parameters corresponding to the category perturbation term after back-training; perform norm control processing on the second perturbation word vector according to the first perturbation word vector so that the norm of the second perturbation word vector is consistent with the norm of the first perturbation word vector, so as to obtain the text classification model.

[0154] This solution achieves its goal of protecting a text classification model from backdoor attacks by making only minor parameter adjustments (i.e., only to the word vector parameters corresponding to the category perturbation terms). Furthermore, because only a very small number of parameters are adjusted, the model's performance on risk-free text remains unaffected, ensuring its good post-deployment performance. Compared to current online attack defense techniques, this solution offers better protection and incurs lower computational costs during testing.

[0155] It is important to note that a text classification model can include multiple categories; for example, the category to be protected can include a first category and a second category.

[0156] In some embodiments, a category perturbation item can be set for each category using the technical solution improved in this embodiment. For example, the category perturbation item provided in this disclosure may include a first category perturbation item and a second category perturbation item. The first category may correspond to the first category perturbation item, and the second category may correspond to the second category perturbation item.

[0157] In some embodiments, the technical solutions provided in this disclosure can be used to update the word vector parameters corresponding to each category of perturbation terms through reverse training, so that risky texts can be judged for different categories to be protected when used online.

[0158] Figure 6 This is a structural diagram illustrating an attack defense method according to an exemplary embodiment. Referring to this structural diagram, the attack defense method provided in this disclosure may include the following parts.

[0159] First, define the capabilities of a text classification model deployer, who is also a defender: obtain a training model to be deployed from the text classification model developer, and a small portion of risk-free validation dataset to test the model's performance on the risk-free dataset.

[0160] In related technologies, text classification models will classify samples containing backdoor triggers into the target category with high confidence. That is, when this disclosure adds perturbations (e.g., adding new polar words) to risky text, the model's output confidence remains almost unchanged. This is because third parties add strong backdoors to the model to ensure they can trigger them 100% if they want to attack later. However, for risk-free samples, such as a positive movie review, adding negative words causes a drastic drop in the model's confidence. Based on this finding, this disclosure constructs a perturbation that, when applied to risk-free samples, lowers the model's output confidence to a certain extent (e.g., by 0.1), but when applied to risky text, it does not decrease to that extent. Therefore, this disclosure aims to determine whether a text is risky by comparing the original confidence level with the model's confidence level after adding the perturbation during online testing, for the same input.

[0161] Although it is assumed that the risk-free validation set cannot be used to continue training the model, otherwise it would cause the model to "catastrophically forget", this disclosure can achieve the purpose of this disclosure by only changing the word vector parameters of certain words. The reasons are: (1) For text classification models, they are all sentences composed of words, so the perturbations to be created are also words or phrases. Therefore, controlling the word vector parameters of these words is enough to make them special "perturbations"; (2) In order to minimize the impact on the performance of the model to be deployed on risk-free samples, the perturbation can be selected as a low-frequency word. If only the word vector of this low-frequency word is changed to make it the special perturbation required by this disclosure, the performance of the model will not be harmed. Since the perturbation created by this disclosure is based on the perturbation resistance of risky text and risk-free samples, that is, the robustness is different, this perturbation is referred to as the robustness-aware perturbation (RAP) below.

[0162] The following disclosure will combine Figure 6 This section describes the overall defense process in implementing this solution:

[0163] Construction Phase: Select a low-frequency word (e.g., mb) and the target category to be protected (sometimes for a text classification model, only one category needs to be protected from backdoor attacks, such as the "non-spam" category in a spam classification model. However, the scheme disclosed herein can also defend against all categories sequentially). Then, select samples from the risk-free validation set whose original labels are the categories to be protected as a small training set. This disclosure adopts a batch training method, in which a batch of data is extracted from the small training set each time, and then the model is used to obtain the model confidence of each batch. Then, for each text to be trained in the batch, the low-frequency words selected in this embodiment are inserted at the beginning of the text to be trained, and then the model outputs a new confidence. Based on the two outputs, the loss function value can be calculated by formula (1). After the loss function value is calculated, gradient backpropagation is performed. However, this disclosure only uses gradient descent to update the word vector parameters corresponding to the low-frequency words selected in this disclosure. At the same time, this disclosure keeps the norm of the updated word vector unchanged. The update continues until the word vector parameters converge or reach a certain preset number of times.

[0164] Selecting an appropriate threshold: After the above construction process is completed, an additional step is required. Although during the construction phase, this disclosure aims to reduce the confidence of all risk-free samples by at least c after adding the perturbation words. low This refers to the threshold δ mentioned earlier. However, in reality, it's not possible to ensure that the confidence level of all samples decreases by c. low Assuming that this disclosure allows the detection mechanism to have an error rate of a% on risk-free samples, i.e., the probability of mistakenly detecting a risk-free sample as risky text (false rejection rate, FRR), then this disclosure takes the threshold δ as a small training set. The percentile a of the confidence difference before and after adding the disturbance word to all samples is arranged from smallest to largest.

[0165] Online Defense Phase: After all the above processes are completed, the model can be deployed online. During online operation, for an original input that is classified into a category to be protected, this disclosure retains the confidence score of the model output regarding the category to be protected. Then, a perturbation word (i.e., a low-frequency word) selected by this disclosure is inserted at the beginning of the input, and the model is then processed again to output a new confidence score. If the difference between the two confidence scores is less than the threshold δ, then it is a risky text; otherwise, it is determined to be a risk-free text. A good defense mechanism needs to minimize the false acceptance rate (FAR) and the false rejection rate (FRR) on online test samples. These two metrics can be used as indicators to measure the defense effectiveness of this scheme.

[0166] The technical solution provided in this embodiment achieves the following technical objectives: (1) When the input is a risk-free sample, the text classification model can determine that it is a normal sample; (2) When the input is a risky text containing a backdoor trigger, the text classification model needs to successfully detect that it is a risky text.

[0167] The technical solution provided in this implementation aims to defend against a new threat to text classification models: backdoor attacks. This technical solution can greatly reduce the damage caused by text classification model developers using backdoors they have left in their models to attack those who purchase and deploy text classification models.

[0168] Compared to some existing technologies, the technical solution provided in this embodiment does not require any assumptions about the form of the backdoor trigger. It provides excellent defense regardless of whether rare words or long sentences are used as backdoor triggers. Furthermore, compared to existing technologies, this solution has a very low computational cost during detection, resulting in improved detection speed. Finally, this solution only requires modifying the word vector parameters of any low-frequency word in the trained model to deploy the model and make it run, without affecting its performance on normal samples.

[0169] Figure 7 This is a block diagram illustrating an attack defense device according to an exemplary embodiment. (Refer to...) Figure 7 The attack defense device 700 provided in this embodiment may include: a first confidence level acquisition module 701, a category perturbation item acquisition module 702, a perturbation text generation module 703 to be predicted, a second confidence level acquisition module 704, and a risk prediction module 705.

[0170] The first confidence level acquisition module 701 can be used to acquire the text to be predicted that belongs to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected; the category perturbation item acquisition module 702 can be used to acquire the category perturbation item corresponding to the category to be protected; the text to be predicted perturbation generation module 703 can be used to perturb the text to be predicted through the category perturbation item to obtain the text to be predicted perturbation; the second confidence level acquisition module 704 can be used to acquire the confidence level of the text to be predicted belonging to the category to be protected; and the risk prediction module 705 can be used to determine whether the text to be predicted is a risky text or a risk-free text based on the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted perturbation belonging to the category to be protected.

[0171] In some embodiments, the risk prediction module 705 may include a risk-free text determination submodule and a risk text determination submodule.

[0172] The risk-free text determination submodule can be used to determine if the text to be predicted is risk-free if the difference between the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected is greater than or equal to the confidence level decrease threshold. The risk text determination submodule can be used to determine if the text to be predicted is risky if the difference between the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected is less than the confidence level decrease threshold.

[0173] In some embodiments, the first confidence level acquisition module 701 may include a text classification model acquisition submodule and a first confidence level determination submodule, and the second confidence level acquisition module 704 may include a second confidence level determination submodule.

[0174] The text classification model acquisition submodule can be used to acquire a text classification model; the first confidence determination submodule can be used to input the text to be predicted into the text classification model and determine the confidence level of the text to be predicted belonging to the category to be protected; the second confidence determination submodule can be used to input the perturbation text to be predicted into the text classification model and determine the confidence level of the perturbation text to be predicted belonging to the category to be protected.

[0175] In some embodiments, the attack defense device includes: a training sample acquisition module, a category perturbation term determination module, a training perturbation text acquisition module, a third confidence determination module, a fourth confidence acquisition module, a target loss value determination module, and an update module.

[0176] The system comprises the following modules: a training sample acquisition module for acquiring training text labeled as the category to be protected; a category perturbation term determination module for determining the category perturbation term corresponding to the category to be protected based on the training text; a training perturbation text acquisition module for perturbing the training text with the target category perturbation term to obtain the training perturbation text; a third confidence determination module for classifying the training text using the training network model to determine the confidence that the training text belongs to the category to be protected; a fourth confidence acquisition module for classifying the training perturbation text using the training network model to determine the confidence that the training perturbation text belongs to the category to be protected; a target loss value determination module for determining the target loss value of the training network model based on the confidence that the training text belongs to the category to be protected and the confidence that the training perturbation text belongs to the category to be protected; and an update module for updating the word vector parameters of the category perturbation term in the training network model based on the target loss value to obtain the text classification model.

[0177] In some embodiments, the target loss value determination module may include: a confidence difference determination submodule, a first difference determination submodule, a first target value determination submodule, a second difference determination submodule, a second target value determination submodule, and a target loss value determination submodule.

[0178] The module includes several submodules: a confidence difference determination submodule, a first difference determination submodule, and a target loss determination submodule. The first target loss determination submodule is used to determine the confidence difference between the confidence of the training text belonging to the protected category and the confidence of the training perturbation text belonging to the protected category. The first difference determination submodule is used to calculate the difference between the confidence differences using a first value to obtain a first difference value. The first target loss determination submodule is used to determine a maximum value between the first difference value and a third value to obtain a first target value. The second difference determination submodule is used to calculate the difference between the second value and a second value to obtain a second difference value. The second target loss determination submodule is used to determine a maximum value between the second difference value and a third value to obtain a second target value. The target loss value determination submodule is used to determine a target loss value based on the first target value and the second target value.

[0179] In some embodiments, the text to be trained includes multiple texts; the attack defense device also includes: a total perturbation processing module, a probability difference determination module, a sorting module, and a percentile determination module.

[0180] The system includes several modules: a perturbation processing module to perturb each training text by category perturbation terms, resulting in perturbed training texts; a probability difference determination module to classify each training text and its corresponding perturbed training text using the training network model, determining the probability difference between the training texts before and after perturbation in the protected category; a ranking module to rank the probability differences of each training text before and after perturbation, resulting in a probability difference ranking result; and a percentile determination module to determine a confidence decrease threshold based on the target percentile in the difference ranking result, using this threshold to defend against attacks on the predicted text. The target percentile is a pre-set probability that the text classification model is allowed to misdetect a risk-free text as a risky text.

[0181] In some embodiments, the network model to be trained includes multiple network layers, and the multiple network layers include an embedding layer, which includes word vector parameters corresponding to category perturbation terms; wherein, the update module may include: a training gradient value determination submodule, a target gradient value determination submodule, and a parameter modification submodule.

[0182] The training gradient value determination submodule can be used to perform backpropagation on the network model to be trained based on the target loss value to determine the backpropagation gradient values ​​of each parameter in each network layer; the target gradient value determination submodule can be used to determine the target gradient value of the word vector parameter corresponding to the category perturbation term in the embedding layer from the backpropagation gradient values ​​in each network layer; the parameter modification submodule can be used to backpropagate the word vector parameter corresponding to the category perturbation term using the target gradient value, while keeping the parameters in the network model to be trained unchanged except for the word vector parameter corresponding to the category perturbation term, so as to obtain a text classification model.

[0183] In some embodiments, the parameter changing submodule further includes: a first perturbation word vector determination unit, a second perturbation word vector determination unit, and a norm control unit.

[0184] The first perturbation word vector determination unit can be used to obtain the first perturbation word vector obtained by processing the category perturbation term with the word vector parameters corresponding to the category perturbation term before reverse training; the second perturbation word vector determination unit can be used to obtain the second perturbation word vector obtained by processing the category perturbation term with the word vector parameters corresponding to the category perturbation term after reverse training; the norm control unit can be used to perform norm control processing on the second perturbation word vector according to the first perturbation word vector, so that the norm of the second perturbation word vector is consistent with the norm of the first perturbation word vector, so as to obtain a text classification model.

[0185] In some embodiments, the category perturbation item determination module includes a low-frequency word determination submodule and a relevance determination submodule.

[0186] Specifically, the low-frequency word determination submodule can be used to identify multiple low-frequency words in the text to be trained that appear less than a frequency threshold; the relevance determination submodule can be used to identify a target low-frequency word among the low-frequency words that has a relevance to the text to be trained that is lower than a similarity threshold, and use the target low-frequency word as a category perturbation term.

[0187] In some embodiments, the target category to be protected includes a first category and a second category, and the category perturbation items include a first category perturbation item corresponding to the first category and a second category perturbation item corresponding to the second category.

[0188] Since the functions of the device 700 have been described in detail in their respective method embodiments, they will not be repeated here.

[0189] The modules and / or sub-modules and / or units described in the embodiments of this application can be implemented in software or hardware. The described modules and / or sub-modules and / or units can also be located in a processor. The names of these modules and / or sub-modules and / or units do not, in some cases, constitute a limitation on the module and / or sub-module and / or unit itself.

[0190] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0191] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0192] Figure 8 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 8 The illustrated electronic device 800 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0193] like Figure 8 As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0194] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0195] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this application.

[0196] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0197] In another aspect, this application also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs, which, when executed by the device, enable the device to perform the following functions: obtaining a text to be predicted belonging to a category to be protected and the confidence level of the text to be predicted belonging to the category to be protected; obtaining a category perturbation term corresponding to the category to be protected; perturbing the text to be predicted using the category perturbation term to obtain a perturbed text to be predicted; obtaining the confidence level of the perturbed text to be predicted belonging to the category to be protected; and determining whether the text to be predicted is risky or risk-free based on the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the perturbed text to be predicted belonging to the category to be protected.

[0198] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0199] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or smart device, etc.) to execute the method according to the embodiments of this disclosure, for example... Figure 2 , Figure 3 , Figure 4 or Figure 5 One or more of the steps shown in the diagram.

[0200] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0201] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. An attack defense method, characterized in that, include: The text classification model is used to obtain the text to be predicted that belongs to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected. Obtain the category perturbation term corresponding to the category to be protected; The text to be predicted is perturbed by the category perturbation item to obtain the perturbed text to be predicted; The confidence level of the perturbation text to be predicted belonging to the category to be protected is obtained through the text classification model. Based on the confidence level that the text to be predicted belongs to the category to be protected, and the confidence level that the disturbed text to be predicted belongs to the category to be protected, the text to be predicted is determined to be either risky text or risk-free text. The text classification model is trained in the following way: Obtain the training text whose text label is the category to be protected, and obtain the training perturbation text corresponding to the training text; Obtain the confidence difference between the confidence score of the training text belonging to the protection category and the confidence score of the training perturbation text belonging to the protection category; The confidence difference is subtracted using the first value to obtain the first difference value; A maximum value is determined between the first difference and the third value, and this maximum value is used as the first target value. The second value is subtracted using the confidence difference to obtain the second difference value; A maximum value is determined between the second difference and the third value, and this maximum value is used as the second target value. Determine the target loss value based on the first target value and the second target value; The training network model is updated based on the target loss value to obtain the text classification model.

2. The method according to claim 1, characterized in that, Based on the confidence level that the text to be predicted belongs to the category to be protected, and the confidence level that the disturbed text to be predicted belongs to the category to be protected, the text to be predicted is determined to be risky text or risk-free text, including: If the confidence difference between the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected is greater than or equal to the confidence decrease threshold, then the text to be predicted is a risk-free text. If the confidence difference between the confidence level of the text to be predicted belonging to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected is less than the confidence decrease threshold, then the text to be predicted is a risky text.

3. The method according to claim 1, characterized in that, The text classification model is used to obtain the text to be predicted that belongs to the category to be protected, and the confidence level of the text to be predicted belonging to the category to be protected, including: Obtain the text classification model; The text to be predicted is input into the text classification model to determine the confidence level of the text to be predicted belonging to the category to be protected; The confidence level of the perturbation text to be predicted belonging to the category to be protected is obtained through the text classification model, including: The text to be predicted is input into the text classification model to determine the confidence level of whether the text to be predicted belongs to the category to be protected.

4. The method according to claim 1, characterized in that, Obtaining the perturbation text corresponding to the text to be trained includes: Based on the text to be trained, determine the category perturbation term corresponding to the category to be protected; The text to be trained is perturbed by the category perturbation term to obtain the perturbed text to be trained; Obtaining the confidence difference between the confidence score of the training text belonging to the protection category and the confidence score of the training perturbation text belonging to the protection category includes: The text to be trained is classified using a network model to determine the confidence level of the text to be trained belonging to the category to be protected. The perturbation text to be trained is classified using the network model to be trained, so as to determine the confidence level of the perturbation text to be trained belonging to the category to be protected; Determine the confidence difference between the confidence level of the training text belonging to the category to be protected and the confidence level of the training perturbation text belonging to the category to be protected; Update the network model to be trained based on the target loss value, including: The word vector parameters of the category perturbation term in the network model to be trained are updated according to the target loss value to obtain the text classification model.

5. The method according to claim 4, characterized in that, The text to be trained includes multiple texts; the method also includes; The perturbation terms of the categories are used to perturb each text to be trained, so as to obtain the perturbation text to be trained corresponding to each text to be trained. The training network model is used to classify each training text and the corresponding training perturbation text to determine the probability difference between each training text before and after the perturbation in the category to be protected. The probability difference of each training text before and after the perturbation is sorted to obtain the probability difference sorting result; A confidence decrease threshold is determined based on the target percentile in the difference ranking results, and attack defense is performed on the text to be predicted based on the confidence decrease threshold. The target percentile is a pre-set probability that the text classification model is allowed to misdetect non-risk text as risky text.

6. The method according to claim 4, characterized in that, The network model to be trained includes multiple network layers, each including an embedding layer, which includes word vector parameters corresponding to the category perturbation term. Updating the word vector parameters of the category perturbation term in the network model to be trained based on the target loss value to obtain the text classification model includes: The network model to be trained is backpropagated based on the target loss value to determine the backpropagation gradient values ​​of each parameter in each network layer. The target gradient value of the word vector parameter corresponding to the category perturbation term in the embedding layer is determined from the back training gradient values ​​in each network layer; The text classification model is obtained by back-updating the word vector parameters corresponding to the category perturbation term using the target gradient value and keeping the parameters in the network model to be trained unchanged except for the word vector parameters corresponding to the category perturbation term.

7. The method according to claim 6, characterized in that, The text classification model is obtained by back-updating the word vector parameters corresponding to the category perturbation term using the target gradient value, while keeping all parameters in the network model under training unchanged except for the word vector parameters corresponding to the category perturbation term. The model further includes: Obtain the first perturbation word vector by processing the category perturbation term using the word vector parameters corresponding to the category perturbation term before reverse training; The second perturbation word vector is obtained by processing the category perturbation term using the word vector parameters corresponding to the category perturbation term after reverse training; The second perturbation word vector is subjected to norm control processing based on the first perturbation word vector so that the norm of the second perturbation word vector is consistent with the norm of the first perturbation word vector, thereby obtaining the text classification model.

8. The method according to claim 4, characterized in that, Determining the category perturbation term corresponding to the category to be protected based on the text to be trained includes: In the text to be trained, identify multiple low-frequency words that appear less than a threshold number of times; Among the various low-frequency words, a target low-frequency word whose relevance to the text to be trained is lower than a similarity threshold is identified, and the target low-frequency word is used as the category perturbation term.

9. The method according to claim 4, characterized in that, The category to be protected includes a first category and a second category, and the category disturbance item includes a first category disturbance item corresponding to the first category and a second category disturbance item corresponding to the second category.

10. An attack defense device, characterized in that, The device includes: The first confidence level acquisition module is used to obtain the text to be predicted that belongs to the category to be protected and the confidence level of the text to be predicted belonging to the category to be protected through a text classification model. The category perturbation item acquisition module is used to acquire the category perturbation item corresponding to the category to be protected; The module for generating perturbation text to be predicted is used to perturb the text to be predicted using the category perturbation item to obtain the perturbation text to be predicted. The second confidence acquisition module is used to obtain the confidence level of the perturbation text to be predicted belonging to the category to be protected through the text classification model; The risk prediction module is used to determine whether the text to be predicted is a risky text or a risk-free text based on the confidence level that the text to be predicted belongs to the category to be protected and the confidence level that the disturbed text to be predicted belongs to the category to be protected. The text classification model is trained in the following way: Obtain the training text whose text label is the category to be protected, and obtain the training perturbation text corresponding to the training text; Obtain the confidence difference between the confidence score of the training text belonging to the protection category and the confidence score of the training perturbation text belonging to the protection category; The confidence difference is subtracted using the first value to obtain the first difference value; A maximum value is determined between the first difference and the third value, and this maximum value is used as the first target value. The second value is subtracted using the confidence difference to obtain the second difference value; A maximum value is determined between the second difference and the third value, and this maximum value is used as the second target value. Determine the target loss value based on the first target value and the second target value; The training network model is updated based on the target loss value to obtain the text classification model.

11. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being used to execute the attack defense method as described in any one of claims 1-9 based on instructions stored in the memory.

12. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the attack defense method as described in any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor of a computer device reads from the computer-readable storage medium and executes the computer program, causing the computer device to perform the attack defense method according to any one of claims 1-9.

Images