Robustness evaluation method and device for operation safety of machine learning model

Abstract

The invention discloses a robustness evaluation method and device for operation safety of a machine learning model, and relates to the field of artificial intelligence and big data or other related technical fields, and the method comprises the steps: obtaining a sample data set; performing cross validation on the sample data set based on a pre-constructed robustness evaluation framework, and screening out a target trigger and a target forgetting instance which enable a loss function of the robustness evaluation framework to be minimum based on the cross validation; executing forgetting operation on the machine learning model based on the target forgetting instance, and inputting the target trigger into the machine learning model executing the forgetting operation to obtain a category label output by the machine learning model; and calculating a backdoor attack success rate based on the category label output by the machine learning model, and evaluating the robustness of the operation security of the machine learning model based on the backdoor attack success rate. According to the method and the device, the technical problem that an evaluation mechanism for the operation safety of a machine learning model is lacked for backdoor attacks in a machine forgetting scene in the related technology is solved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and big data or other related technical fields. Specifically, it relates to a robustness evaluation method and apparatus for the safe operation of machine learning models. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, machine learning models have been widely applied in various industries, including but not limited to financial services, healthcare, autonomous driving, and image recognition. These models rely on large amounts of user data for training to achieve high-precision prediction and decision-making capabilities. However, as public awareness of privacy protection increases, individual users have the right to demand that their data be forgotten from machine learning models. This has led to "machine forgetting" becoming an important requirement and technical issue in the management of machine learning models.

[0003] In related technologies, research on machine forgetting mainly focuses on improving the effectiveness and efficiency of forgetting, but neglects the security challenges it brings and lacks an evaluation mechanism for the security of machine learning operations. In machine forgetting scenarios, some users contributing training data may have malicious intent. They may submit deceptive forgetting requests to induce the forgotten model to behave maliciously, thereby launching an attack. The most prominent of these is backdoor attacks, where attackers aim to inject hidden backdoors into the machine learning model through methods such as polluting its training set, so as to change the prediction results of the attacked model when a specific trigger occurs during the inference phase.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a robustness assessment method and apparatus for the operation security of machine learning models, thereby addressing the technical problem in related technologies where, in the context of machine forgetting scenarios, there is a lack of an assessment mechanism for the operational security of machine learning models in the face of backdoor attacks.

[0006] According to one aspect of the present invention, a robustness evaluation method for the safe operation of a machine learning model is provided, comprising: obtaining a sample dataset from an authorized user, wherein the sample data types in the sample dataset are normal datasets and trigger datasets, the normal datasets include datasets that will not trigger backdoor attacks, the trigger datasets include datasets used to trigger backdoor attacks, each training sample in the sample datasets includes a forgotten instance and a category label, the forgotten instance being a training sample that the user requests the machine learning model to forget; performing cross-validation on the sample datasets based on a pre-built robustness evaluation framework, and selecting, based on the cross-validation, a target trigger and a target forgotten instance that minimize the loss function of the robustness evaluation framework; performing a forgetting operation on the machine learning model based on the target forgotten instance, and inputting the target trigger into the machine learning model performing the forgetting operation to obtain the category label output by the machine learning model; calculating the backdoor attack success rate based on the category label output by the machine learning model, and evaluating the robustness of the machine learning model's operation based on the backdoor attack success rate; and performing image recognition based on the machine learning model that has passed the robustness evaluation.

[0007] Furthermore, the robustness evaluation framework is pre-built. The steps of building the robustness evaluation framework include: establishing a loss function for the machine learning model on the normal dataset; establishing a loss function for the machine learning model on the triggered dataset; constructing a loss function for the number of triggered forgotten instances based on the arctangent function approximating the number of triggered forgotten instances, wherein the triggered forgotten instances represent forgotten instances selected by the attacker to trigger a backdoor attack; constructing a loss function for the trigger detection probability based on the gradient difference generated by the forward propagation of the triggered data input and the normal data input in the machine learning model; and constructing the loss function of the robustness evaluation framework based on the loss function of the machine learning model on the normal dataset, the loss function of the machine learning model on the triggered dataset, the loss function for the number of triggered forgotten instances, and the loss function for the trigger detection probability, thus obtaining the robustness evaluation framework.

[0008] Furthermore, the loss function of the machine learning model on the normal dataset is expressed as: ,in, For hyperparameters, For machine learning models after forgetting, and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the loss function of the machine learning model. These are the model parameters for the machine learning model after forgetting.

[0009] Furthermore, the loss function of the machine learning model on the trigger dataset is expressed as: ,in, For hyperparameters, It is a trigger.

[0010] Furthermore, the loss function for triggering the number of forgotten instances is expressed as: , in, For hyperparameters, The probability of an instance being forgotten is selected for the attacker.

[0011] Furthermore, the loss function for the trigger detection probability is expressed as: , in, For hyperparameters, Represents the parameter gradient matrix, Indicates the gradient anomaly threshold. For the sample dataset, For the detection function, output trigger The probability of being identified as a backdoor attack.

[0012] Further, the step of cross-validating the sample dataset based on a pre-constructed robustness evaluation framework, and selecting the target trigger and target forgotten instance that minimizes the loss function of the robustness evaluation framework based on cross-validation, includes: splitting the sample dataset into K mutually exclusive data subsets, and constructing multiple rounds of cross-validation based on the data subsets, wherein in each round of cross-validation, one subset is used as the test set, and the remaining K-1 sub-machines are used as the training set; for each round of cross-validation, iteratively training the machine learning model using the training set, and calculating the parameter gradient matrix and Hessian matrix during the training process, wherein the parameter gradient matrix is ​​the first derivative matrix corresponding to the loss function of the robustness evaluation framework, and the Hessian matrix is ​​the second derivative matrix corresponding to the loss function of the robustness evaluation framework; updating the model parameters of the forgetting model based on the Hessian matrix, the parameter gradient matrix, and the learning rate of the machine learning model; and determining the sample data containing the trigger and forgotten instance that minimizes the loss function of the robustness evaluation framework based on the updated model parameters, thereby obtaining the target trigger and the target forgotten instance.

[0013] According to another aspect of the present invention, a robustness evaluation apparatus for the safe operation of a machine learning model is also provided, comprising: an acquisition unit, configured to acquire a sample dataset from an authorized user, wherein the sample data types in the sample dataset are normal datasets and trigger datasets, the normal datasets include datasets that will not trigger backdoor attacks, the trigger datasets include datasets used to trigger backdoor attacks, each training sample in the sample datasets includes a forgotten instance and a category label, the forgotten instance being a training sample that the user requests the machine learning model to forget; a filtering unit, configured to perform cross-validation on the sample datasets based on a pre-built robustness evaluation framework, and filter out target triggers and target forgotten instances that minimize the loss function of the robustness evaluation framework based on the cross-validation; an execution unit, configured to perform a forgetting operation on the machine learning model based on the target forgotten instance, and input the target trigger to the machine learning model performing the forgetting operation to obtain the category label output by the machine learning model; and an evaluation unit, configured to calculate the backdoor attack success rate based on the category label output by the machine learning model, evaluate the robustness of the machine learning model's safe operation based on the backdoor attack success rate, and perform image recognition based on the machine learning model that passes the robustness evaluation.

[0014] Furthermore, the robustness evaluation device for the secure operation of the machine learning model further includes: a first establishment module for establishing a loss function of the machine learning model on the normal dataset; a second establishment module for establishing a loss function of the machine learning model on the triggered dataset; a first construction module for constructing a loss function for the number of triggered forgotten instances based on the arctangent function approximating the number of triggered forgotten instances, wherein the triggered forgotten instances represent forgotten instances selected by the attacker to trigger a backdoor attack; a second construction module for constructing a loss function for the trigger detection probability based on the gradient difference generated by the triggered data input and normal data input after forward propagation of the machine learning model; and a third construction module for constructing a loss function of the robustness evaluation framework based on the loss function of the machine learning model on the normal dataset, the loss function of the machine learning model on the triggered dataset, the loss function for the number of triggered forgotten instances, and the loss function for the trigger detection probability, thereby obtaining the robustness evaluation framework.

[0015] Furthermore, the loss function of the machine learning model on the normal dataset is expressed as: ,in, For hyperparameters, For machine learning models after forgetting, and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the loss function of the machine learning model. These are the model parameters for the machine learning model after forgetting.

[0016] Furthermore, the loss function of the machine learning model on the trigger dataset is expressed as: ,in, For hyperparameters, It is a trigger.

[0017] Furthermore, the loss function for triggering the number of forgotten instances is expressed as: , in, For hyperparameters, The probability of an instance being forgotten is selected for the attacker.

[0018] Furthermore, the loss function for the trigger detection probability is expressed as: , in, For hyperparameters, Represents the parameter gradient matrix, Indicates the gradient anomaly threshold. For the sample dataset, For the detection function, output trigger The probability of being identified as a backdoor attack.

[0019] Further, the screening unit includes: a first splitting module, used to split the sample dataset into K mutually exclusive data subsets, and construct multiple rounds of cross-validation based on the data subsets, wherein in each round of cross-validation, one subset is used as the test set, and the remaining K-1 sub-machines are used as the training set; a first training module, used to iteratively train the machine learning model using the training set for each round of cross-validation, and calculate the parameter gradient matrix and Hessian matrix during the training process, wherein the parameter gradient matrix is ​​the first derivative matrix corresponding to the loss function of the robustness evaluation framework, and the Hessian matrix is ​​the second derivative matrix corresponding to the loss function of the robustness evaluation framework; a first updating module, used to update the model parameters of the forgetting model based on the Hessian matrix, the parameter gradient matrix, and the learning rate of the machine learning model; and a first determining module, used to determine the sample data containing the trigger and the forgotten instance that minimizes the loss function of the robustness evaluation framework based on the updated model parameters, thereby obtaining the target trigger and the target forgotten instance.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described robustness evaluation methods for the safe operation of machine learning models.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors enable the one or more processors to implement any of the above-described robust evaluation methods for the safe operation of machine learning models.

[0022] In this application, a sample dataset is obtained from an authorized user. The sample dataset contains two data types: a normal dataset and a triggered dataset. The normal dataset includes datasets that will not trigger backdoor attacks, while the triggered dataset includes datasets used to trigger backdoor attacks. Each training sample in the sample dataset includes a forgotten instance and a class label. The forgotten instance is a training sample that the user requests the machine learning model to forget. Then, the sample dataset is cross-validated based on a pre-built robustness evaluation framework. Based on the cross-validation, the target trigger and the target forgotten instance that minimize the loss function of the robustness evaluation framework are selected. The machine learning model is then subjected to a forgetting operation based on the target forgotten instance, and the target trigger is input into the machine learning model that performs the forgetting operation to obtain the class label output by the machine learning model. Finally, the backdoor attack success rate is calculated based on the class label output by the machine learning model, and the robustness of the machine learning model's operation is evaluated based on the backdoor attack success rate. Image recognition is then performed based on the machine learning model that passes the robustness evaluation.

[0023] This application constructs a robustness evaluation framework to effectively assess the robustness and security of a machine learning model against backdoor attacks during the data forgetting process. By addressing an optimization problem, the robustness evaluation framework automatically selects the optimal forgetting instance and backdoor trigger, minimizing the impact on model performance and the probability of trigger detection. This evaluation method fills a gap in existing technologies regarding the assessment of model operational security during data forgetting. Only machine learning models that pass the robustness evaluation can perform relevant tasks, thus enhancing the machine learning model's ability to defend against backdoor attacks and improving its ability to balance data privacy protection and secure operation. This solves the technical problem in related technologies of lacking an assessment mechanism for the operational security of machine learning models against backdoor attacks in machine forgetting scenarios. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a robust evaluation method to ensure the safe operation of machine learning models is shown.

[0026] Figure 2 This is a flowchart of an optional robustness evaluation method for running a machine learning model according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an optional robustness evaluation process for running a machine learning model according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of an optional robustness evaluation device for running a machine learning model securely, according to an embodiment of the present invention.

[0029] Figure 5 This is a hardware structure block diagram of an electronic device (or mobile device) that performs an optional robustness evaluation method for running a machine learning model securely, according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] It should be noted that the robustness assessment method and apparatus for the operational safety of machine learning models in this application can be used in the fields of artificial intelligence and big data to assess the robustness of machine learning models based on a robustness assessment framework, and can also be used in any field other than artificial intelligence and big data to assess the robustness of machine learning models based on a robustness assessment framework. This application does not limit the application areas of the robustness assessment method and apparatus for the operational safety of machine learning models.

[0033] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0034] The following embodiments of the present invention can be applied to robustness evaluation systems / applications / devices for the safe operation of various machine learning models. The present invention proposes a robust evaluation framework that achieves the evaluation objective without contaminating any training data of the model being evaluated. Based on an optimization problem, optimal forgotten instances and backdoor triggers are derived, with the goal of maximizing evaluation utility while minimizing the number of instances in the forgotten subset and the probability of trigger detection. It can evaluate attack frameworks that balance the effectiveness and stealth of attacks. It fills the current gap in the evaluation of backdoor attacks.

[0035] The present invention will now be described in detail with reference to various embodiments.

[0036] Example 1

[0037] According to an embodiment of the present invention, an embodiment of a robustness evaluation method for the safe operation of a machine learning model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1A hardware block diagram of a computer terminal (or mobile device) for implementing a robust evaluation method to ensure the safe operation of machine learning models is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0039] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the robustness evaluation method for the safe operation of the machine learning model in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned robustness evaluation method for the safe operation of the machine learning model. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0042] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0043] Under the aforementioned operating environment, this application provides the following: Figure 2 The method shown is a robustness evaluation method for ensuring the safe operation of machine learning models. The main body implementing this method is a robustness evaluation system for ensuring the safe operation of machine learning models.

[0044] Figure 2 This is a flowchart of an optional robustness evaluation method for machine learning model execution security according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0045] To ensure user information security during machine learning model training, a data forgetting mechanism has been proposed. This mechanism removes the influence of specific data on the model's training results, allowing the model to "forget" the data from specific users while retaining its ability to learn from other data. However, during this forgetting process, some users who contribute training data may act maliciously, submitting deceptive forgetting requests to induce malicious behavior in the machine learning model, thereby launching an attack. One common form of attack is a backdoor attack. Attackers aim to inject a hidden backdoor into the machine learning model using methods such as polluting its training set. This backdoor can then alter the model's predictions when specific triggers occur during the inference phase, thus attacking the machine learning model.

[0046] Step S201: Obtain the sample dataset from the authorized user.

[0047] In step S201 above, a sample dataset is obtained from an authorized user. The sample dataset refers to a model containing forgotten instances of the machine learning model. Forgotten instances are divided into normal forgotten instances and triggered forgotten instances used by attackers to carry out backdoor attacks. The sample data types in the sample dataset include normal datasets and triggered datasets. Normal datasets include datasets that will not trigger backdoor attacks, and triggered datasets include datasets used to trigger backdoor attacks. Each training sample in the sample dataset includes a forgotten instance and a class label. The class label is divided into positive samples and negative samples. Positive samples are normal forgotten instances that will not trigger backdoor attacks, and negative samples are triggered forgotten instances that will trigger backdoor attacks. Forgotten instances are training samples that the user requests the machine learning model to forget. By forgetting a portion of the training data, user privacy can be protected without affecting the performance of the machine learning model.

[0048] Step S202: Cross-validate the sample dataset based on the pre-built robustness evaluation framework, and select the target triggers and target forgetting instances that minimize the loss function of the robustness evaluation framework based on the cross-validation.

[0049] In step S202 above, the robustness evaluation framework is constructed based on a composite loss function, aiming to quantify and balance the safety and performance of the machine learning model during the data forgetting process. The robustness evaluation framework covers key elements such as the loss of the machine learning model on normal and triggered datasets, the number of forgotten instances, and the detection probability of the trigger.

[0050] In the robustness evaluation process, the target triggers and target forgetting instances that minimize the loss function of the robustness evaluation framework are first selected based on cross-validation. Cross-validation maximizes the use of sample data, and through iterative cross-validation, the framework continuously tries different combinations of triggers and forgetting instances. Ultimately, those combinations that significantly reduce the loss function value are identified as target triggers and target forgetting instances. This selection process ensures that the selected instances not only effectively promote data forgetting but also minimize the chance of successful backdoor attacks. This allows for a more realistic and effective simulation of backdoor attacks to evaluate the security and resistance of machine learning models to backdoor attacks during the data forgetting process.

[0051] Furthermore, the robustness evaluation framework is pre-built. The steps for building the robustness evaluation framework include: establishing a loss function for the machine learning model on a normal dataset; establishing a loss function for the machine learning model on a triggered dataset; constructing a loss function for the number of triggered forgotten instances based on the arctangent function approximating the number of triggered forgotten instances, where the triggered forgotten instances represent forgotten instances selected by the attacker to trigger a backdoor attack; constructing a loss function for the trigger detection probability based on the gradient difference generated by the forward propagation of the triggered data input and the normal data input in the machine learning model; and constructing the loss function of the robustness evaluation framework based on the loss function of the machine learning model on the normal dataset, the loss function of the machine learning model on the triggered dataset, the loss function for the number of triggered forgotten instances, and the loss function for the trigger detection probability, thus obtaining the robustness evaluation framework.

[0052] Specifically, the robustness evaluation framework is essentially a composite loss function that integrates multiple objectives and considerations. It aims to comprehensively evaluate the security and robustness of machine learning models in specific scenarios, especially in the context of data forgetting and resistance to backdoor attacks. The loss function includes the loss function of the forgetting model on the normal dataset, the loss function of the forgetting model on the triggered dataset, the number of instances chosen by the attacker for erasure, and the detection probability of the trigger.

[0053] First, we define the loss function for the machine learning model on a normal dataset, which refers to forgotten instances without any triggers. The loss function is constructed to measure the model's predictive accuracy when not subjected to backdoor attacks. Minimizing this loss ensures the model's robustness and accuracy during normal operation. Next, we establish the loss function for the machine learning model on a triggered dataset, which evaluates the model's performance on this dataset—its reaction to triggers. The triggered dataset contains specific input samples designed to trigger backdoor attacks. By maximizing or selectively tuning this loss, the framework can detect whether the model reacts anomalously to triggers, thereby assessing the model's backdoor attack risk.

[0054] Considering that attackers might select specific instances as triggers for forgetting during the data forgetting process to indirectly launch backdoor attacks, a loss function for the number of trigger forgetting instances is constructed to assess and control this risk. This loss function is approximated by using the arctangent function. The choice of function ensures the continuity and differentiability of the loss function, facilitating the application of optimization algorithms. The objective is to minimize the number of trigger forgetting instances while ensuring that the model's performance and security are not significantly affected.

[0055] From a defensive perspective, the framework needs to consider the detection probability of triggers. By analyzing the gradient difference between trigger data input and normal data input after forward propagation in the model, a loss function can be constructed to calculate the trigger detection probability. The larger the gradient difference, the higher the probability of the trigger being detected, and vice versa. The loss function aims to minimize the probability of trigger detection, thereby improving the stealth of attacks.

[0056] Finally, the four loss functions mentioned above are combined to construct the overall loss function of the robustness evaluation framework. By reasonably setting hyperparameters, the relationship between model performance, attack success rate, and stealth is balanced. The goal of constructing the overall loss function is to find a set of forgotten instances and triggers. The triggers and forgotten instances selected by the robustness evaluation framework are more stealthy, reducing the probability of detection and thus improving the attack success rate. This allows for more effective simulation of backdoor attacks to test the performance of machine learning models against backdoor attacks.

[0057] Furthermore, the loss function of the machine learning model on a normal dataset is expressed as: ,in, For hyperparameters, For machine learning models after forgetting, and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the loss function of the machine learning model. These are the model parameters for the machine learning model after forgetting.

[0058] Specifically, the loss function of a machine learning model on a normal dataset is defined as follows: ,in The hyperparameter is used to balance the aggressiveness and stealth of the framework. This loss function measures the prediction loss for each forgotten instance under the class label, thereby quantifying the prediction accuracy and generalization ability of the machine learning model on normal datasets after performing data forgetting operations. This ensures that while forgetting sensitive information, the machine learning model can still effectively process normal data, maintaining its basic functionality and accuracy.

[0059] Furthermore, the loss function of the machine learning model on the trigger dataset is expressed as: ,in, For hyperparameters, It is a trigger.

[0060] Specifically, the loss function of the machine learning model on the trigger dataset is defined as follows: This study calculates the loss between the model's output and the attacker's expected label when the model encounters a triggered instance. This quantifies and evaluates the behavioral changes of the machine learning model when encountering a specific backdoor trigger. It examines and evaluates the model's sensitivity and responsiveness to backdoor attacks. This helps identify potential weaknesses in machine learning models, particularly security vulnerabilities that may be exposed after data forgetting operations.

[0061] Furthermore, the loss function for triggering the number of forgotten instances is expressed as: , in, For hyperparameters, The probability of an instance being forgotten is selected for the attacker.

[0062] The number of triggered forgotten instances helps evaluators quantify a model's vulnerability to such attacks. A smaller number of instances successfully implanting a backdoor indicates the machine learning model is easily affected by this type of attack, while a larger number suggests the model has strong resistance. Furthermore, the number of triggered forgotten instances can also be used to assess the stealth of an attack. Typically, attackers aim to achieve their goals with minimal data manipulation to reduce the risk of detection. Therefore, models that can successfully launch attacks with a small number of triggered forgotten instances may be vulnerable to stealthy backdoor attacks.

[0063] In machine learning and optimization, continuous and differentiable functions are crucial for efficient optimization. The number of triggered forgotten instances is essentially a discrete value, representing the number of instances selected for forgetting in the dataset. However, most optimization algorithms, such as gradient descent or its variants, are designed for continuous spaces and are often inefficient or even unapplicable for optimizing discrete variables. Therefore, it is necessary to approximate the discrete quantity of the number of triggered forgotten instances into a continuous variable. This invention uses the arctangent function for approximation, ensuring that the optimization problem involved in the evaluation framework is both differentiable and continuous. This allows gradient-based optimization methods to be applied to this problem, effectively finding the optimal combination of the number of forgotten instances and the triggers to achieve the robustness and stealth objectives of the attack framework.

[0064] Specifically, the loss function that triggers the number of forgotten instances is defined as follows: , The primary purpose of constructing this loss function is to optimize the number of forgotten instances selected by the attacker, ensuring that this number effectively triggers the backdoor attack while minimizing the probability of detection. By minimizing the loss function that triggers the number of forgotten instances, the evaluation framework can simulate how an attacker optimally selects which instances to forget in order to achieve their attack objective.

[0065] Furthermore, the loss function for the trigger detection probability is expressed as: , in, For hyperparameters, Represents the parameter gradient matrix, Indicates the gradient anomaly threshold. For the sample dataset, For the detection function, output trigger The probability of being identified as a backdoor attack.

[0066] Trigger detection probability reflects the model's sensitivity and detection accuracy to backdoor triggers. A higher detection probability means the model can effectively identify the existence of backdoor attacks, indicating a better defense mechanism; conversely, a low detection probability suggests the model is vulnerable to undetected backdoor attacks, indicating vulnerabilities in security protection. For attackers, trigger detection probability is a key indicator for evaluating the success rate of attack strategies. When designing attack frameworks, attackers try to minimize the probability of their triggers being detected to improve the stealth and success rate of the attack. Therefore, trigger detection probability can, to some extent, guide the optimization of attack frameworks, enabling them to effectively implant backdoors while circumventing existing detection mechanisms. Incorporating trigger detection probability into a robustness evaluation framework can more effectively filter out triggers with strong stealth, thereby more effectively simulating backdoor attacks.

[0067] Specifically, the loss function for trigger detection probability is defined as follows: , The goal is to minimize the probability that the machine learning model will detect the trigger. This function quantifies the model's resistance to backdoor attack triggers by introducing hyperparameters and using the output of the detection function. During optimization, reducing the loss function value that lowers the trigger detection probability means improving the trigger's stealth, making it harder for the machine learning model to identify the backdoor attack. This allows the model's robustness to be more accurately reflected in the security assessment framework, reflecting its true defense level.

[0068] Therefore, based on the loss function of the machine learning model on the normal dataset, the loss function of the machine learning model on the triggered dataset, the loss function for the number of forgotten triggered instances, and the loss function for the trigger detection probability, a loss function for the robustness evaluation framework can be constructed, which is expressed as: , , ,in, , , , These are hyperparameters used to balance the aggressiveness and stealth of the framework. For the forgotten task model, These are the parameters of the original model. These are the model parameters after forgetting. and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the model's loss function. For triggers, Forgotten datasets of attackers, The sample dataset provided by the attacker Choose the probability of instance forgetting for the attacker. Let A be the number of forgotten instances in the attacker's sample dataset. To adjust the function's parameters, The constraints for the trigger (where) For triggers constraint values, The infinity norm of the trigger is represented by the maximum absolute value of all elements in the trigger. = , (element in the trigger) The sample dataset contains detected samples of known backdoors. For the detection function, output trigger The probability of being identified as a backdoor attack.

[0069] Furthermore, the process of cross-validating the sample dataset based on a pre-built robustness evaluation framework, and selecting the target trigger and target forgotten instance that minimizes the loss function of the robustness evaluation framework based on cross-validation, includes: splitting the sample dataset into K mutually exclusive data subsets, and constructing multiple rounds of cross-validation based on the data subsets, wherein in each round of cross-validation, one subset is used as the test set, and the remaining K-1 subsets are used as the training set; for each round of cross-validation, iteratively training the machine learning model using the training set, and calculating the parameter gradient matrix and Hessian matrix during the training process, wherein the parameter gradient matrix is ​​the first derivative matrix corresponding to the loss function of the robustness evaluation framework, and the Hessian matrix is ​​the second derivative matrix corresponding to the loss function of the robustness evaluation framework; updating the model parameters of the forgetting model based on the Hessian matrix, parameter gradient matrix, and the learning rate of the machine learning model; and determining the sample data containing the trigger and forgotten instance that minimizes the loss function of the robustness evaluation framework based on the updated model parameters, thus obtaining the target trigger and target forgotten instance.

[0070] Cross-validation is performed based on a robust evaluation framework. Trigger instance combinations that meet preset requirements are selected. Specifically, the sample dataset is first uniformly partitioned into K independent and mutually exclusive subsets. This ensures that each subset has the opportunity to be used as a test set in multiple rounds of cross-validation, resulting in a comprehensive and impartial evaluation of model performance. This approach avoids bias caused by a single test set, ensuring the reliability and general applicability of the evaluation results. In the specific cross-validation process, each round follows the same pattern: one subset serves as the test set, while the remaining K-1 subsets are combined as the training set for updating and evaluating the machine learning model. This cyclical mechanism of partitioning and recombination ensures that the evaluation covers all parts of the sample dataset, enabling a comprehensive examination of the model's performance under data forgetting conditions and its ability to withstand backdoor attacks.

[0071] For each round of cross-validation, the model is iteratively trained based on the training set to continuously optimize model parameters to adapt to the environment after forgetting data. During training, particular attention is paid to the calculation of the parameter gradient matrix and the Hessian matrix. The parameter gradient matrix reflects the array of first-order partial derivatives of the loss function with respect to the model parameters, while the Hessian matrix is ​​the array of second-order derivatives of the loss function with respect to the model parameters. These two matrices are crucial for understanding how the model responds to external changes and how it adjusts parameters based on these changes. By integrating the Hessian matrix, parameter gradient matrix, and model learning rate, the parameters of the machine learning model are updated to reflect the impact of the data forgetting operation. This process utilizes the concept of a second-order erasure algorithm, the core of which is how to minimize the damage to model security caused by malicious forgetting requests while preserving model performance. The model parameter update strategy is key to implementing the forgetting operation in the evaluation framework and directly affects the model's performance in subsequent tests.

[0072] Based on the updated model parameters, triggers and forgotten instances that minimize the loss function of the robustness evaluation framework are identified. Through multiple rounds of iterative training, a dataset containing multiple target triggers and multiple target forgotten instances can be obtained. This selection process is the core of the entire implementation, aiming to find the smallest but sufficiently effective portion of the dataset (forgotten instances) and triggers that an attacker might exploit to induce the model to exhibit backdoor behavior without significantly reducing the model's accuracy on normal datasets. By precisely calculating and comparing the loss function values ​​under different combinations of triggers and forgotten instances, the most dangerous attack patterns can be identified.

[0073] Step S203: Perform a forgetting operation on the machine learning model based on the target forgotten instance, and input the target trigger into the machine learning model that performs the forgetting operation to obtain the category label output by the machine learning model.

[0074] In step S203 above, the goal of performing the forgetting operation is to simulate the scenario where data is legitimately requested to be forgotten, observe the model's response to the target trigger after parameter updates, and evaluate the framework's effectiveness and the model's robustness. First, based on the optimization objective defined in the framework, a set of target forgetting instances is selected; these instances are considered to have a critical impact on the model's security. Next, the forgetting operation is performed on the machine learning model, i.e., removing or reducing the impact of the target forgetting instances from the model's training records. Subsequently, the target trigger is input into the updated machine learning model; the trigger is designed to elicit potential backdoor behavior in the model. Through the model's inference process, the model's classification label for the input containing the trigger can be obtained. The purpose of this step is to test whether the model after the forgetting operation can resist attacks from the target trigger, or to what extent it exhibits backdoor behavior.

[0075] Step S204: Calculate the backdoor attack success rate based on the category labels output by the machine learning model, evaluate the robustness of the machine learning model's operation security based on the backdoor attack success rate, and perform image recognition based on the machine learning model that has passed the robustness evaluation.

[0076] In step S204 above, after performing the forgetting operation and inputting the target trigger, the backdoor attack success rate is calculated based on the category labels output by the machine learning model. The backdoor attack success rate represents the frequency with which a forgotten instance containing a specific trigger is identified by the machine learning model as the content desired by the attacker. A higher backdoor attack success rate indicates that the model is easily manipulated by triggers and that the security defense mechanism is weak. Subsequently, based on the calculated backdoor attack success rate, the robustness of the machine learning model in terms of operational security is evaluated. The robustness evaluation framework plays a central role here, comprehensively considering the model's accuracy, the stealth of the backdoor attack, and the success rate of the attack, thereby deriving a comprehensive overview of the model's security status. The evaluation process aims to determine whether the model can maintain stable performance and a high level of security protection when facing data forgetting and backdoor attacks.

[0077] Only machine learning models that pass robustness evaluations—meaning their security and performance meet established standards—can be used for image recognition tasks. Security-verified models can more confidently process and parse image data in practical applications, are less susceptible to backdoor attacks, and ensure the accuracy of image recognition results and the overall security of the system.

[0078] Through the above steps, a sample dataset is obtained from authorized users. The sample dataset contains two data types: a normal dataset and a trigger dataset. The normal dataset includes datasets that will not trigger backdoor attacks, while the trigger dataset includes datasets used to trigger backdoor attacks. Each training sample in the sample dataset includes a forgotten instance and a class label. The forgotten instance is a training sample that the user requests the machine learning model to forget. Then, the sample dataset is cross-validated based on a pre-built robustness evaluation framework. Based on the cross-validation, the target trigger and the target forgotten instance that minimize the loss function of the robustness evaluation framework are selected. The machine learning model is then subjected to a forgetting operation based on the target forgotten instance, and the target trigger is input into the machine learning model that performs the forgetting operation to obtain the class label output by the machine learning model. Finally, the backdoor attack success rate is calculated based on the class label output by the machine learning model, and the robustness of the machine learning model's operation is evaluated based on the backdoor attack success rate. Image recognition is then performed based on the machine learning model that passes the robustness evaluation.

[0079] In this embodiment, by constructing a robustness evaluation framework, the robustness and security of a machine learning model against backdoor attacks can be effectively evaluated during the data forgetting process. Through an optimization problem, the robustness evaluation framework can automatically select the optimal forgetting instance and backdoor trigger, while minimizing the impact on model performance and the probability of trigger detection. This evaluation method fills a gap in existing technologies, namely, the evaluation of model operational security during data forgetting. Only machine learning models that pass the robustness evaluation can perform relevant tasks, improving the machine learning model's ability to defend against backdoor attacks and enhancing its ability to balance data privacy protection and secure operation. This solves the technical problem in related technologies where there is a lack of an evaluation mechanism for the operational security of machine learning models against backdoor attacks in machine forgetting scenarios.

[0080] The following describes in detail another optional implementation method.

[0081] Figure 3 This is a schematic diagram illustrating an optional robustness evaluation process for running a machine learning model securely, according to an embodiment of the present invention. Figure 3 As shown, the robustness evaluation process for running this machine learning model includes:

[0082] In today's data-driven era, it has become commonplace for organizers to collect user data to train machine learning models through legal and compliant means. With user consent, organizers collect various types of information to build and optimize models; this data is crucial for ensuring accurate predictions and good performance. However, malicious attackers lurk within the user base, their goal not being privacy protection, but rather attempting to manipulate the model through backdoor attacks. To avoid detection, attackers choose a covert method of sending malicious forgetting requests. During the attack, attackers carefully disguise themselves. Quantitatively, they strictly control the number of data instances requested for forgetting to avoid triggering alerts with large-scale deletions at once, maintaining a request frequency and quantity similar to normal users. Regarding impact, they fully consider the effect of forgetting requests on model performance, carefully calculating and planning to select specific data instances that seem insignificant but can build a backdoor for the request. These data instances trigger the backdoor when certain conditions are met. Once the malicious forgetting request succeeds, the backdoor will be activated as the model is subsequently trained and applied, affecting the model's operational security.

[0083] First, obtain the sample dataset from authorized users. ,in, y is the number of instances in the dataset, and y is the class label of instance x. The data used by the attacker is... The attacker's goal is to persuade the model owner to forget a subset of their training data. (Right now This is so that the forgotten model can exhibit backdoor behavior.

[0084] Step 1: Problem Modeling;

[0085] To ensure the operational safety of the machine learning model, we construct loss functions for the machine learning model on normal datasets, loss functions for the machine learning model on triggered datasets, loss functions for the number of forgotten instances triggered, and loss functions for the detection probability of triggers.

[0086] Furthermore, the loss function of the machine learning model on the normal dataset is expressed as: ,in, For hyperparameters, For machine learning models after forgetting, and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the loss function of the machine learning model. These are the model parameters for the machine learning model after forgetting.

[0087] Furthermore, the loss function of the machine learning model on the trigger dataset is expressed as: ,in, For hyperparameters, It is a trigger.

[0088] Furthermore, the loss function for triggering the number of forgotten instances is expressed as: , in, For hyperparameters, The probability of an instance being forgotten is selected for the attacker.

[0089] Furthermore, the loss function for the trigger detection probability is expressed as: , in, For hyperparameters, Represents the parameter gradient matrix, Indicates the gradient anomaly threshold. For the sample dataset, For the detection function, output trigger The probability of being identified as a backdoor attack.

[0090] Step 2: Construct a robustness assessment framework;

[0091] We construct a robust evaluation framework for machine learning forgetting, which is essentially an optimization framework incorporating a composite loss function.

[0092] Therefore, based on the loss function of the machine learning model on the normal dataset, the loss function of the machine learning model on the triggered dataset, the loss function for the number of forgotten triggered instances, and the loss function for the trigger detection probability, a loss function for the robustness evaluation framework can be constructed, which is expressed as: , , ,in, , , , These are hyperparameters used to balance the aggressiveness and stealth of the framework. For the forgotten task model, These are the parameters of the original model. These are the model parameters after forgetting. and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the model's loss function. For triggers, Forgotten datasets of attackers, The sample dataset provided by the attacker Choose the probability of instance forgetting for the attacker. Let A be the number of forgotten instances in the attacker's sample dataset. To adjust the function's parameters, The constraints for the trigger (where) For triggers constraint values, The infinity norm of the trigger is represented by the maximum absolute value of all elements in the trigger. = , (element in the trigger) The sample dataset contains detected samples of known backdoors. For the detection function, output trigger The probability of being identified as a backdoor attack.

[0093] Step 3: Filter the target forgotten instances and target triggers for testing;

[0094] This invention employs a second-order erasure algorithm to simulate the process of erasing sensitive information, verifying the effectiveness and stealth of the attack framework. The second-order erasure algorithm is expressed as follows: , in, It is the Hessian inverse matrix. For the forgotten dataset, For learning rate, The trigger dataset provided by the attacker For the attacker to choose the probability of instance forgetting, θ * θ represents the parameters of the original machine learning model. u These are the model parameters for the machine learning model after forgetting.

[0095] The specific implementation of the second-order erasure algorithm involves calculating the gradient based on the overall evaluation framework defined above, using L as the loss function. The gradient matrix of the model parameters is obtained, and then the Hessian matrix H is solved, where , and These are model parameters, Multiply by the gradient matrix of the parameters, then multiply by the learning rate. Then, using the original model parameters Update the parameters of the model after forgetting Target forgotten instances selected through cross-validation and target trigger .

[0096] Step four: Erase based on the target forgotten instance and the target trigger, and evaluate the model's operational safety and robustness.

[0097] The machine learning model is erased using instances of target forgetting, while the selected triggers are... Injecting backdoors into the dataset to launch backdoor attacks and calculating the success rate of backdoor attacks are used to evaluate the robustness of machine learning models.

[0098] This invention proposes a robust evaluation framework that achieves the evaluation objective without contaminating any training data of the model being evaluated. Based on an optimization problem, optimal forgotten instances and backdoor triggers are derived, aiming to maximize evaluation utility while minimizing the number of instances in the forgotten subset and the probability of trigger detection. This framework can evaluate attack frameworks that balance the effectiveness and stealth of attacks, filling a gap in current backdoor attack evaluation methods.

[0099] The following is a detailed description with reference to another embodiment.

[0100] Example 2

[0101] The robustness evaluation device for the safe operation of a machine learning model provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above. The specific implementation method and its beneficial effects can be referred to the aforementioned method embodiment, and will not be repeated here.

[0102] Figure 4 This is a schematic diagram of an optional robustness evaluation device for running a machine learning model securely, according to an embodiment of the present invention. Figure 4 As shown, the robustness evaluation device for the safe operation of this machine learning model may include: an acquisition unit 41, a filtering unit 42, an execution unit 43, and an evaluation unit 44, wherein,

[0103] The acquisition unit 41 is used to acquire a sample dataset from an authorized user. The sample data types in the sample dataset are normal dataset and trigger dataset. The normal dataset includes datasets that will not trigger backdoor attacks, and the trigger dataset includes datasets used to trigger backdoor attacks. Each training sample in the sample dataset includes a forgotten instance and a category label. The forgotten instance is a training sample that the user requests the machine learning model to forget.

[0104] The filtering unit 42 is used to perform cross-validation on the sample dataset based on a pre-built robustness evaluation framework, and to filter out the target triggers and target forgetting instances that minimize the loss function of the robustness evaluation framework based on the cross-validation.

[0105] Execution unit 43 is used to perform a forgetting operation on the machine learning model based on the target forgotten instance, and input the target trigger to the machine learning model performing the forgetting operation to obtain the category label output by the machine learning model;

[0106] Evaluation unit 44 is used to calculate the backdoor attack success rate based on the category labels output by the machine learning model, evaluate the robustness of the machine learning model's security based on the backdoor attack success rate, and perform image recognition based on the machine learning model that has passed the robustness evaluation.

[0107] The robustness evaluation device for the aforementioned machine learning model's secure operation obtains a sample dataset from an authorized user through acquisition unit 41. The sample dataset contains two data types: a normal dataset and a trigger dataset. The normal dataset includes datasets that will not trigger backdoor attacks, while the trigger dataset includes datasets used to trigger backdoor attacks. Each training sample in the sample dataset includes a forgotten instance and a category label. The forgotten instance is a training sample that the user requests the machine learning model to forget. A filtering unit 42 performs cross-validation on the sample dataset based on a pre-built robustness evaluation framework, selecting the target trigger and the target forgotten instance that minimizes the loss function of the robustness evaluation framework. An execution unit 43 performs a forgetting operation on the machine learning model based on the target forgotten instance, inputting the target trigger into the machine learning model to obtain the category label output by the machine learning model. An evaluation unit 44 calculates the backdoor attack success rate based on the category label output by the machine learning model and evaluates the robustness of the machine learning model's secure operation based on the backdoor attack success rate. Image recognition is then performed based on the machine learning model that passes the robustness evaluation.

[0108] In this embodiment, by constructing a robustness evaluation framework, the robustness and security of a machine learning model against backdoor attacks can be effectively evaluated during the data forgetting process. Through an optimization problem, the robustness evaluation framework can automatically select the optimal forgetting instance and backdoor trigger, while minimizing the impact on model performance and the probability of trigger detection. This evaluation method fills a gap in existing technologies, namely, the evaluation of model operational security during data forgetting. Only machine learning models that pass the robustness evaluation can perform relevant tasks, improving the machine learning model's ability to defend against backdoor attacks and enhancing its ability to balance data privacy protection and secure operation. This solves the technical problem in related technologies where there is a lack of an evaluation mechanism for the operational security of machine learning models against backdoor attacks in machine forgetting scenarios.

[0109] Furthermore, the robustness evaluation device for the secure operation of the machine learning model also includes: a first establishment module for establishing the loss function of the machine learning model on a normal dataset; a second establishment module for establishing the loss function of the machine learning model on a triggered dataset; a first construction module for constructing a loss function for the number of triggered forgotten instances based on the arctangent function approximating the number of triggered forgotten instances, where the triggered forgotten instances represent forgotten instances selected by the attacker to trigger a backdoor attack; a second construction module for constructing a loss function for the trigger detection probability based on the gradient difference generated by the forward propagation of the triggered data input and the normal data input in the machine learning model; and a third construction module for constructing a loss function for the robustness evaluation framework based on the loss function of the machine learning model on the normal dataset, the loss function of the machine learning model on the triggered dataset, the loss function for the number of triggered forgotten instances, and the loss function for the trigger detection probability, thus obtaining the robustness evaluation framework.

[0110] Furthermore, the loss function of the machine learning model on a normal dataset is expressed as: ,in, For hyperparameters, For machine learning models after forgetting, and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the loss function of the machine learning model. These are the model parameters for the machine learning model after forgetting.

[0111] Furthermore, the loss function of the machine learning model on the trigger dataset is expressed as: ,in, For hyperparameters, It is a trigger.

[0112] Furthermore, the loss function for triggering the number of forgotten instances is expressed as: , in, For hyperparameters, The probability of an instance being forgotten is selected for the attacker.

[0113] Furthermore, the loss function for the trigger detection probability is expressed as: , in, For hyperparameters, Represents the parameter gradient matrix, Indicates the gradient anomaly threshold. For the sample dataset, For the detection function, output trigger The probability of being identified as a backdoor attack.

[0114] Further, the screening unit includes: a first splitting module, used to split the sample dataset into K mutually exclusive data subsets and construct multiple rounds of cross-validation based on the data subsets, wherein in each round of cross-validation, one subset is used as the test set and the remaining K-1 sub-machines are used as the training set; a first training module, used to iteratively train the machine learning model using the training set for each round of cross-validation, and calculate the parameter gradient matrix and Hessian matrix during the training process, wherein the parameter gradient matrix is ​​the first derivative matrix corresponding to the loss function of the robustness evaluation framework, and the Hessian matrix is ​​the second derivative matrix corresponding to the loss function of the robustness evaluation framework; a first updating module, used to update the model parameters of the forgetting model based on the Hessian matrix, the parameter gradient matrix, and the learning rate of the machine learning model; and a first determining module, used to determine the sample data containing the trigger and the forgotten instance that minimizes the loss function of the robustness evaluation framework based on the updated model parameters, thereby obtaining the target trigger and the target forgotten instance.

[0115] It should be noted that the acquisition unit 41, filtering unit 42, execution unit 43, and evaluation unit 44 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules or units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0116] The invention will now be described in conjunction with another alternative embodiment.

[0117] Example 3

[0118] The present invention can also provide an electronic device. Figure 5 This is a hardware structure block diagram of an electronic device (or mobile device) that performs an optional robustness evaluation method for executing machine learning models according to an embodiment of the present invention, such as... Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (Only one is shown) processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0119] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] The processor can access information and applications stored in memory via a transmission device to perform the following steps: Obtain a sample dataset from an authorized user, wherein the sample data types in the sample dataset are normal datasets and trigger datasets. The normal dataset includes datasets that will not trigger backdoor attacks, and the trigger dataset includes datasets used to trigger backdoor attacks. Each training sample in the sample dataset includes a forgotten instance and a class label. The forgotten instance is a training sample that the user requests the machine learning model to forget. Perform cross-validation on the sample dataset based on a pre-built robustness evaluation framework. Based on the cross-validation, select the target trigger and the target forgotten instance that minimizes the loss function of the robustness evaluation framework. Perform a forgetting operation on the machine learning model based on the target forgotten instance, and input the target trigger into the machine learning model performing the forgetting operation to obtain the class label output by the machine learning model. Calculate the backdoor attack success rate based on the class label output by the machine learning model, and evaluate the robustness of the machine learning model's operation based on the backdoor attack success rate. Perform image recognition based on the machine learning model that passes the robustness evaluation.

[0121] The processor can access information and applications stored in memory via a transmission device to perform the following steps: establishing a loss function for the machine learning model on a normal dataset; establishing a loss function for the machine learning model on a triggered dataset; constructing a loss function for the number of triggered forgotten instances based on the arctangent function approximating the number of triggered forgotten instances, where triggered forgotten instances represent forgotten instances selected by the attacker to trigger a backdoor attack; constructing a loss function for the trigger detection probability based on the gradient difference generated by the forward propagation of the triggered data input and normal data input in the machine learning model; and constructing a loss function for the robustness evaluation framework based on the loss function of the machine learning model on the normal dataset, the loss function of the machine learning model on the triggered dataset, the loss function for the number of triggered forgotten instances, and the loss function for the trigger detection probability, thus obtaining the robustness evaluation framework.

[0122] The processor can access information and applications stored in memory via a transmission device to execute the following steps: The loss function of the machine learning model on a normal dataset is expressed as: ,in, For hyperparameters, For machine learning models after forgetting, and Let represent the forgotten instances and class labels in the sample dataset, respectively, and L be the loss function of the machine learning model. These are the model parameters for the machine learning model after forgetting.

[0123] The processor can access information and applications stored in memory via a transmission device to execute the following steps: The loss function of the machine learning model on the trigger dataset is expressed as: ,in, For hyperparameters, It is a trigger.

[0124] The processor can access information and applications stored in memory via a transmission device to perform the following steps: The loss function for triggering the number of forgotten instances is expressed as: , in, For hyperparameters, The probability of an instance being forgotten is selected for the attacker.

[0125] The processor can access information and applications stored in memory via a transmission device to execute the following steps: The loss function for the trigger detection probability is expressed as: , in, For hyperparameters, Represents the parameter gradient matrix, Indicates the gradient anomaly threshold. For the sample dataset, For the detection function, output trigger The probability of being identified as a backdoor attack.

[0126] The processor can access information and applications stored in memory via a transmission device to execute the following steps: Split the sample dataset into K mutually exclusive subsets and construct multiple rounds of cross-validation based on these subsets. In each round of cross-validation, one subset is used as the test set, and the remaining K-1 subsets are used as the training set. For each round of cross-validation, iteratively train the machine learning model using the training set, calculating the parameter gradient matrix and Hessian matrix during training. The parameter gradient matrix is ​​the first derivative matrix of the loss function of the robustness evaluation framework, and the Hessian matrix is ​​the second derivative matrix of the loss function of the robustness evaluation framework. Update the model parameters of the forgetting model based on the Hessian matrix, the parameter gradient matrix, and the learning rate of the machine learning model. Based on the updated model parameters, determine the sample data containing the trigger and the forgotten instance that minimizes the loss function of the robustness evaluation framework, thus obtaining the target trigger and the target forgotten instance.

[0127] This invention provides a robustness evaluation scheme for the operational security of machine learning models. By constructing a robustness evaluation framework, the robustness and security of a machine learning model against backdoor attacks can be effectively evaluated during the data forgetting process. Through an optimization problem, the robustness evaluation framework can automatically select the optimal forgetting instance and backdoor trigger, while minimizing the impact on model performance and the probability of trigger detection. This evaluation method fills a gap in the prior art, namely, the evaluation of model operational security during the data forgetting process. Only machine learning models that pass the robustness evaluation can perform relevant tasks, improving the machine learning model's ability to prevent backdoor attacks and enhancing its ability to balance data privacy protection and secure operation. This solves the technical problem in related technologies where there is a lack of an evaluation mechanism for the operational security of machine learning models against backdoor attacks in machine forgetting scenarios.

[0128] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0129] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0130] The invention will now be described in conjunction with another alternative embodiment.

[0131] Example 4

[0132] This invention also provides a computer-readable storage medium. Optionally, in this invention, the computer-readable storage medium can be used to store the program code executed by the robustness evaluation method for the safe operation of the machine learning model provided in Embodiment 1.

[0133] Optionally, in this embodiment of the invention, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0134] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for performing the steps of a robustness evaluation method for the safe operation of a machine learning model: obtaining a sample dataset from an authorized user, wherein the sample data types in the sample dataset are normal datasets and trigger datasets, the normal datasets include datasets that will not trigger backdoor attacks, the trigger datasets include datasets used to trigger backdoor attacks, each training sample in the sample datasets includes a forgotten instance and a class label, the forgotten instance being a training sample that the user requests the machine learning model to forget; performing cross-validation on the sample datasets based on a pre-built robustness evaluation framework, and selecting the target trigger and target forgotten instance that minimize the loss function of the robustness evaluation framework based on the cross-validation; performing a forgetting operation on the machine learning model based on the target forgotten instance, and inputting the target trigger into the machine learning model performing the forgetting operation to obtain the class label output by the machine learning model; calculating the backdoor attack success rate based on the class label output by the machine learning model, and evaluating the robustness of the machine learning model's operation based on the backdoor attack success rate; and performing image recognition based on the machine learning model that has passed the robustness evaluation.

[0135] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0136] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0141] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for robustness evaluation of machine learning model execution security, characterized in that, The method comprises the following steps: obtaining a sample data set from an authorized user, wherein the sample data set comprises normal data sets and trigger data sets, the normal data sets comprise data sets that will not trigger a backdoor attack, and the trigger data sets comprise data sets for triggering a backdoor attack, each training sample in the sample data set comprises a forgetting instance and a class label, and the forgetting instance is a training sample requested by the user to forget; cross-validating the sample data set based on a pre-constructed robustness evaluation framework, and screening a target trigger and a target forgetting instance based on cross-validation to minimize the loss function of the robustness evaluation framework; performing a forgetting operation on the machine learning model based on the target forgetting instance, and inputting the target trigger into the machine learning model to obtain a class label output by the machine learning model; calculating a backdoor attack success rate based on the class label output by the machine learning model, and evaluating the robustness of the machine learning model based on the backdoor attack success rate to assess the safety of the machine learning model in running, and performing image recognition based on the machine learning model that passes the robustness evaluation.

2. The method of claim 1, wherein, The robustness evaluation framework is pre-constructed, and the steps of constructing the robustness evaluation framework comprise: establishing a loss function of the machine learning model on the normal data sets; establishing a loss function of the machine learning model on the trigger data sets; approximating the number of trigger forgetting instances based on an inverse tangent function, and constructing a loss function of the number of trigger forgetting instances, wherein the trigger forgetting instance represents a forgetting instance selected by an attacker to trigger a backdoor attack; constructing a loss function of trigger detection probability based on the gradient difference generated after the trigger data input and the normal data input are forward propagated in the machine learning model; constructing a loss function of the robustness evaluation framework based on the loss function of the machine learning model on the normal data sets, the loss function of the machine learning model on the trigger data sets, the loss function of the number of trigger forgetting instances, and the loss function of the trigger detection probability, to obtain the robustness evaluation framework.

3. The method of claim 2, wherein, The loss function of the machine learning model on the normal data set is represented as: wherein, is a hyperparameter, is the machine learning model after forgetting, and are the forgetting instance and the category label in the sample data set respectively, and L is the loss function of the machine learning model, is the model parameter of the machine learning model after forgetting.

4. The method of claim 3, wherein, The loss function of the machine learning model on the trigger dataset is represented as: wherein, is a hyperparameter, is a trigger.

5. The method of claim 4, wherein, The loss function of the number of trigger forgetting instances is represented as: ， where, is a hyperparameter, is the probability that the attacker chooses an instance to forget.

6. The method of claim 5, wherein, The loss function of the trigger detection probability is represented as: ， wherein, is a hyperparameter, denotes a parameter gradient matrix, denotes a gradient anomaly threshold, is a sample data set, is a detection function, output trigger the probability of being identified as a backdoor attack.

7. The method of claim 6, wherein, The steps of cross-validating the sample data set based on the pre-constructed robustness evaluation framework and screening a target trigger and a target forgetting instance based on cross-validation to minimize the loss function of the robustness evaluation framework comprise: splitting the sample data set into K mutually exclusive data subsets, and constructing multiple rounds of cross-validation based on the data subsets, wherein in each round of cross-validation, one subset is used as a test set, and the remaining K-1 subsets are used as training sets; for each round of cross-validation, iteratively training the machine learning model based on the training sets, and calculating a parameter gradient matrix and a Hessian matrix during the training process, wherein the parameter gradient matrix is a first derivative matrix corresponding to the loss function of the robustness evaluation framework, and the Hessian matrix is a second derivative matrix corresponding to the loss function of the robustness evaluation framework; updating model parameters of the forgetting model based on the Hessian matrix, the parameter gradient matrix, and a learning rate of the machine learning model; determining sample data containing the target trigger and the target forgetting instance that minimize a loss function of the robustness evaluation framework based on the updated model parameters. 8.A device for robustness evaluation of machine learning model execution security, characterized in that, The method comprises the following steps: an acquisition unit is configured to acquire a sample data set from an authorized user, wherein the sample data in the sample data set is of a normal data set and a trigger data set, the normal data set includes a data set that does not trigger a backdoor attack, and the trigger data set includes a data set for triggering a backdoor attack, each training sample in the sample data set includes a forgetting instance and a class label, and the forgetting instance is a training sample for which the user requests the machine learning model to forget; a screening unit is configured to perform cross-validation on the sample data set based on a pre-constructed robustness evaluation framework, and screen out a target trigger and a target forgetting instance that minimize a loss function of the robustness evaluation framework based on the cross-validation; an execution unit is configured to perform a forgetting operation on the machine learning model based on the target forgetting instance, and input the target trigger into the machine learning model that performs the forgetting operation, to obtain a class label output by the machine learning model; an evaluation unit is configured to calculate a backdoor attack success rate based on the class label output by the machine learning model, and evaluate the robustness of the machine learning model in terms of running safety based on the backdoor attack success rate, and perform image recognition based on the machine learning model that passes the robustness evaluation.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored computer program, wherein the computer program controls a device in which the computer-readable storage medium is located to perform the robustness evaluation method of the machine learning model running safety according to any one of claims 1 to 7 when the computer program is running.

10. An electronic device, comprising: The device comprises one or more processors and a memory, and the memory is configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the robustness evaluation method of the machine learning model running safety according to any one of claims 1 to 7.

Images