Training data poisoning detection

Abstract

Training data poisoning detection is disclosed. In some examples, a system receives a plurality of training samples of a training dataset for a machine learning model, where each training sample of the plurality of training samples includes a plurality of features. The system determines a number of changes made to respective ones of the plurality of features, calculates a score representing the integrity of the training dataset based on the number, and detects poisoning of the training dataset based on the score.

Description

Background Technology

[0001] Artificial intelligence is increasingly being used in computing environments to drive efficiency and innovation in service and / or product delivery. The artificial intelligence exhibited by machines relies on machine learning models that can learn from a knowledge base, which can be based on any one or a combination of the following: data from past activities, training data, or data from other sources. Attached Figure Description

[0002] Some embodiments of this disclosure are described with reference to the following figures.

[0003] Figure 1 It is a block diagram of a computer system, including a training data poisoning detection engine, based on some examples.

[0004] Figure 2 This is a block diagram illustrating, based on some examples, the input training sample set and the counters used to track feature changes in the training samples.

[0005] Figure 3 It is a block diagram of a storage medium with machine-readable instructions based on some examples of storage.

[0006] Figure 4 It is a block diagram of a system based on some examples.

[0007] Figure 5 It is a flowchart based on some examples.

[0008] In all the accompanying drawings, the same reference numerals refer to similar but not necessarily identical elements. The drawings are not necessarily drawn to scale, and the size of some parts may be enlarged to illustrate the examples more clearly. Furthermore, the drawings provide examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments provided in the drawings. Detailed Implementation

[0009] Training datasets can be used to train machine learning models for prediction. A training dataset includes training samples, where a "sample" can refer to a data record with corresponding attribute values. These attributes are also called features of the sample. Training samples in a training dataset can include labeled training samples, where each set of features for a given sample is assigned a label (selected from multiple labels) corresponding to that set of features. For example, a first set of features in a first training sample might include values ​​that collectively indicate an attack is occurring against the computing environment. Therefore, a first label (e.g., "attack detected" label) can be assigned to the first set of features of the first sample. On the other hand, a second set of features in a second training sample might include values ​​that collectively indicate no attack is occurring against the computing environment. In the latter case, a second label (e.g., "attack not detected" label) can be assigned to the second set of features of the second sample. In other cases, other labels can be assigned to training samples.

[0010] Attackers (e.g., humans, malware, or machines) may attempt to influence the predictions made by a machine learning model by modifying the training dataset used to train it. Such modification to the training dataset results in training dataset poisoning. A machine learning model trained on a poisoned training dataset will produce inaccurate outputs; for example, the model might predict an attack that is not detected in the computing environment when it is actually occurring. The erroneous outputs produced by the machine learning model can lead to security vulnerabilities, potentially compromising the integrity of the computing environment, damaging data stored in the computing environment, or resulting in data theft. Training datasets can also be poisoned for other reasons, such as due to data errors or malfunctions in the machine or program that generated the training dataset. As used herein, training data "poisoning" refers to any modification (whether intentional or unintentional) made to the training data that causes the machine learning model to produce inaccurate outputs based on the input dataset.

[0011] Based on some examples of this disclosure, techniques or mechanisms are provided for detecting poisoning of training datasets used to train machine learning models based on one or more of the following: the number of changes to corresponding features in the training samples of the training dataset, and the number of outliers in the values ​​of the corresponding features in the training samples. Based on the aforementioned quantities, a score representing the integrity of the training dataset is calculated. This score is used to detect poisoning of the training dataset.

[0012] Figure 1This is a block diagram of an example arrangement of a computer system 102, which includes a training data generator 104 that creates or updates training data for a machine learning model 106. This computer system can be implemented using one or more computers. The machine learning model 106 can be executed in a separate computer system from computer system 102, or alternatively, the machine learning model 106 can be executed within computer system 102.

[0013] The training data generator 104 can be implemented using machine-readable instructions executed by processing resources in the computer system 102. Alternatively or additionally, the training data generator 104 can be implemented using hardware processing circuitry. Furthermore, the training data source for the machine learning model 106 can include external sources outside the computer system 102, such as people, programs, or machines. External sources can create or update training data for the machine learning model 106.

[0014] Training data (generated by training data generator 104 and / or an external source) can be written to main storage system 108, which includes one or more storage devices. Training data generator 104 and / or the external source can issue write requests to write the training data. Write requests are processed by driver 110 in computer system 102. In response to the write request, driver 110 issues a write transaction to write the training data to training dataset 109 in main storage system 108.

[0015] The training dataset 109 can be in a specified format, such as any of the following: columnar data file format (e.g., relational table format in a Structured Query Language (SQL) database), tabular data file format (e.g., Excel format, comma-separated values ​​(CSV) format, etc.), nested file format (e.g., Extensible Markup Language (XML) format, JavaScript Object Notation (JSON) format, etc.) or any other format.

[0016] Training dataset 109 is used to train machine learning model 106. Once trained, machine learning model 106 can recognize patterns in the input data. Machine learning model 106 produces an output representing the predictions made by machine learning model 106 based on the input data. The output of machine learning model 106 is used by one or more consumers to perform various actions. Consumers of the output of machine learning model 106 may include users, programs, or machines.

[0017] Driver 110 is a program that manages access to the main storage system 108. In some examples, driver 110 is part of the operating system (OS). In other examples where a virtual computing environment is implemented in computer system 102, driver 110 may be part of a virtualization hypervisor, such as a hypervisor or a container engine. The hypervisor creates and manages virtual machines (VMs) in computer system 102. The container engine creates and manages containers in computer system 102.

[0018] In some examples, computer system 102 further includes a data replication manager 112 and a training data poisoning detection engine 114. Each of the data replication manager 112 and the training data poisoning detection engine 114 can be implemented using machine-readable instructions executable on the processing resources of computer system 102. In other examples, the data replication manager 112 and the training data poisoning detection engine 114 can be implemented using one or more hardware processing circuits.

[0019] Data replication manager 112 writes and replicates data to backup storage system 116, which includes one or more storage devices. In some examples, backup storage system 116 is located outside of computer system 102. In other examples, backup storage system 116 may be located inside computer system 102. Primary storage system 108 is used to store training data (e.g., training dataset 109) for application to machine learning model 106. Backup storage system 116 is used to store copies of the training data for use in restoring the training data.

[0020] Backup storage system 116 may be physically separate from primary storage system 108. Alternatively, backup storage system 116 may be part of the same physical storage infrastructure but logically separate from primary storage system 108.

[0021] Input / output (I / O) operations between driver 110 and primary storage system 108 may include read I / O operations and write I / O operations. Data replication manager 112 is capable of detecting write I / O operations and copying them to backup storage system 116. "Copying" of data writes may refer to storing a representation of the write I / O operation in backup storage system 116. The representation of the write I / O operation may include changed data (e.g., new data, modified data, or deleted data). The representation of the write I / O operation may also include information about the type of write operation (e.g., insert operation adding new data, update operation modifying data, or delete operation deleting data).

[0022] The copying of the training data results in a copied training dataset 118 being stored in backup storage system 116. The copied training dataset 118 includes checkpoints 120A to 120B corresponding to different time points. Each checkpoint in the copied training dataset 118 represents a version of the training data at the corresponding time point. Different checkpoints in the copied training dataset 118 can be created at different time points. If any part of the training dataset 109 is lost or corrupted, the training data in the training dataset 109 can be recreated using the checkpoints in the copied training dataset 118.

[0023] When the copied training data is written to the backup storage system 116, the training data poisoning detection engine 114 can be applied to the copied training data. The training data poisoning detection engine 114 applies its analysis to the set of input training samples written to the copied training dataset 118. The set of input training samples to which the training data poisoning detection engine 114 applies its analysis can include training samples within a specified time interval, such as a time window of a specified length. The time window has a range from T1 to T2, where T2 can be the current time, and T1 and T2 define the specified length. In such an example, the training data poisoning detection engine 114 applies its analysis to the most recent time window, which is a shifted time window that moves with the current time. In other examples, the set of input training samples can be selected in different ways. For example, training samples can be randomly selected when the copied training data is written to the backup storage system 116.

[0024] Each training sample includes a set of features and a label assigned to that set of features. The label can be assigned by a human, a program, or a machine. The features of the training samples can include numerical features and / or categorical features. Numerical features are assigned values ​​within a range of values, while categorical features are assigned categorical values ​​from a set of discrete categorical values.

[0025] The training data poisoning detection engine 114 can perform "real-time" detection of training data poisoning. Real-time detection of training data poisoning is based on the training data poisoning detection engine 114 analyzing training samples in the copied training data when the data copy manager 112 writes the copied training data to the backup storage system 116.

[0026] If the training data poisoning detection engine 114 detects potential poisoning of training data in the input training sample set, the training data poisoning detection engine 114 issues a poisoning alarm 130, which can be in the form of a message, information element, or any other type of indicator. The poisoning alarm 130 indicates that the training dataset 109 used by the machine learning model 106 has been poisoned. The poisoning alarm 130 may include a timestamp indicating the time when potential poisoning of the training data was detected.

[0027] The training data poisoning detection engine 114 sends a poisoning alert 130 to a remediation engine 132, which can take remedial action in response to the poisoning alert 130. Examples of remedial actions may include any of the following: issuing an alert to the target entity (e.g., a human administrator, program, or machine), disabling the machine learning model 106, disabling the computer system in which the machine learning model 106 is executed (e.g., by shutting down the computer system), disabling the network connection of the computer system in which the machine learning model 106 is executed, or any other remedial action.

[0028] Additionally, the remediation engine 132 can retrieve checkpoints (e.g., 120A or 120B) from the copied training dataset 118 to restore the poisoned training dataset 109 to its previous state. The retrieved checkpoints were created before the timestamp included in the poisoning alert 130. Since the poisoning alert 130 is generated by the training data poisoning detection engine 114 based on real-time detection of training data poisoning, the timestamp in the poisoning alert 130 likely represents the approximate time when the training dataset 109 was poisoned. Therefore, checkpoints created before the timestamp of the poisoning alert 130 (or some threshold time interval prior to that timestamp) likely include unpoisoned training data.

[0029] The training data poisoning detection engine 114 can issue a poisoning alert 130 based on one or more criteria. The first criterion concerns whether the training samples (which may be new or modified training samples) conform to a pattern of training data (referred to as a "training data pattern"). The training data pattern can define the format of the training data. For example, the training data pattern specifies which features are included in each training sample, and the possible values ​​for each feature (e.g., a range of values ​​or a set of classification values).

[0030] If the training data poisoning detection engine 114 detects that the training samples (or a specified number of training samples) in the input training sample set do not conform to the training data pattern, then the training sample is likely to have been tampered with, and therefore the training data poisoning detection engine 114 issues a poisoning alarm 130.

[0031] The second criterion concerns whether the training samples have been completely deleted. In most cases, completely deleting training samples is likely a legitimate action. For example, training samples might be deleted as part of data cleaning. Therefore, deleting training samples will not cause the training data poisoning detection engine 114 to issue a poisoning alarm 130.

[0032] The third criterion concerns whether features of a subset (less than all) of the training samples in the input training sample set have been modified. Changing individual features of training samples can be associated with an attack on the training data, especially when these changes target features of some training samples but not others. It should be noted that changing the value of a single feature (or subset of features) across all training samples in the input training sample set can be considered a legitimate action. For example, the value of a single feature (or subset of features) across all training samples in the input training sample set might be changed as part of a data scaling or data transformation operation. Therefore, changing the value of a single feature (or subset of features) across all training samples in the input training sample set will not cause the training data poisoning detection engine 114 to issue a poisoning alarm 130. However, changes to feature values ​​in some training samples while features in others remain unchanged will be considered an indication of training data poisoning by the training data poisoning detection engine 114.

[0033] More generally, a change in the value of a single feature (or subset of features) of more than a threshold number of training samples in the input training sample set will not cause the training data poisoning detection engine 114 to issue a poisoning alarm 130. However, a change in the feature values ​​of some training samples (less than the threshold number) in the input training sample set while the feature values ​​of other training samples remain unchanged will be considered an indication of training data poisoning by the training data poisoning detection engine 114. The threshold number may be based on a certain relative percentage of the total number of training samples in the input training sample set (e.g., 99%, 95%, 90%, 80%, etc.). In other examples, a third criterion involves determining whether the change to the training samples is consistent with a target change pattern due to common or expected data transformations that may be applied to the training samples. If the change is inconsistent with the target pattern, it may indicate training data poisoning.

[0034] Another issue to consider is that some features of the training samples might be categorical features, or even text features, and these features may be transformed. An example transformation of categorical features is from {-1, 1} to {0, 1} to prepare for use in some machine learning models, or for label encoding (e.g., "red" → 0, "green" → 1, "blue" → 2). Therefore, non-numerical features can be ignored, or it can be checked whether there has been any change after the data preparation phase (where non-numerical features may be transformed).

[0035] The fourth criterion concerns whether there are outliers in the features of the training samples in the input training sample set, and whether the presence of these outliers satisfies one or more specified conditions. In some examples, outliers are feature values ​​that fall outside the expected set of values ​​based on the distribution of feature values, where the distribution of values ​​can be based on the observed values ​​of the feature. An “observed” value of a feature can refer to a feature value within the input training sample set, or alternatively, to a feature value within a larger training sample set (such as a historical training sample set). In other examples, outliers can be identified statistically. For example, a feature value is considered an outlier if the difference between the value of a feature and the mean of the observed values ​​of that feature exceeds a specified number of standard deviations. The presence of outliers may indicate the injection of synthetic or manipulated data if the following conditions are met: The first condition concerns whether the number of outliers increases after the training data has been edited. If this first condition is met, it may indicate that the training data has been poisoned. The second condition concerns whether the variance of the outliers deviates from the previous standard deviation. If this second condition is met, it may indicate that the training data has been poisoned.

[0036] Figure 2 This is a block diagram of the input training sample set 200, where each training sample has features A, B, C, D, and E. The input training sample set 200 includes five training samples: 202, 204, 206, 208, and 210. The shaded cells in the input training sample set 200 represent changes to feature values, such as due to the addition of new values ​​or the writing of modifications to existing feature values. Although Figure 2 An example is shown where the input training sample set 200 has 5 training samples; however, in other examples, the input training sample set 200 can have a different number of training samples. Furthermore, in other examples, the training samples can have more than... Figure 2 The number of features shown is less or more.

[0037] Each of features A through E is associated with a corresponding counter. For example, feature A is associated with counter 222A, feature B with counter 222B, feature C with counter 222C, feature D with counter 222D, and feature E with counter 222E. Each counter tracks the number of changes made to the corresponding feature in the training samples of the input training sample set 200. Therefore, counter 222A tracks the number of changes made to feature A in training samples 202, 204, 206, 208, and 210. Figure 2 In the example, the value of feature A has been changed in training samples 202 and 204. Therefore, counter 222A is incremented to 2 to represent these two changes.

[0038] Counter 222B tracks the number of changes made to feature B in training samples 202, 204, 206, 208, and 210. In this example, the value of feature B has been changed in all training samples 202, 204, 206, 208, and 210. Therefore, counter 222B increments to 5 to represent these five changes.

[0039] Counter 222C tracks the number of changes made to feature C in training samples 202, 204, 206, 208, and 210. In this example, the value of feature C has been changed in training samples 202 and 204. Therefore, counter 222C increments to 2 to represent these two changes.

[0040] Counter 222D tracks the number of changes made to feature D in training samples 202, 204, 206, 208, and 210. In this example, the value of feature D has been changed in training sample 204. Therefore, counter 222D increments to 1 to indicate that this change has been made.

[0041] Counter 222E tracks the number of changes made to feature E in training samples 202, 204, 206, 208, and 210. In this example, the value of feature E has been changed in training sample 204. Therefore, counter 222E increments to 1 to indicate that this change has been made.

[0042] As stated above, according to the third criterion, changing a single feature of all training samples in the input training sample set can be considered a legal action. Figure 2 In the example, since counter 222B has increased to 5 for an input training sample set with 5 training samples, the training data poisoning detection engine 114 can ignore changes to feature B. In other words, the training data poisoning detection engine 114 can ignore the value of counter 222B when calculating a score related to whether training data poisoning has occurred.

[0043] The following describes an example of how the poison detection engine 114 calculates scores in training data. The following parameters are defined.

[0044] The parameter N_Samples represents the total number of training samples in the input training sample set. Figure 2 N_Samples = 5.

[0045] Parameter C i This represents the count of changes made to feature i. For example, for Figure 2 For features A and C A =2, for feature C, C C =2, for feature D, C D =1, and for feature E, CE =1. Note that the count C of feature B... B =5 has been ignored because the value of feature B has changed across all training samples in the input training sample set of 200. The count C of feature B is ignored. R This can be achieved in one of two ways. First, the count C can be eliminated from Equation 1 below. B Secondly, count C B It can be set to 0 and included in Equation 1.

[0046] Parameter L i L represents the number of outliers for feature i. A L represents the number of outliers in feature A. C L represents the number of outliers in feature C. D L represents the number of outliers in feature D, and L E This represents the number of outliers in feature E. In some examples, L i ≤C i .

[0047] In the example, the score representing the completeness of training dataset 109 can be calculated as follows:

[0048]

[0049] In Equation 1, This represents the ratio of the number of changes made to feature i to the total number of training samples (N_Samples) in the input training sample set. It is called the "change rate".

[0050] per change rate Multiply by according to The value of the factor f (as calculated according to Table 1 below) i In practice, these factors are used to scale the number of changes made to the corresponding feature (C). i This generates scaling values. These scaling values ​​are then used in the first expression of Equation 1. Add them together. Factor f i This is used to detect the small number of changes to feature i in the input training sample set, to prevent large counts (i.e., C). i The larger value of the score dominates the calculation of the score.

[0051] Table 1

[0052] start Finish factor 0 0.04 10 0.04 0.1 5 0.1 0.2 2 0.2 0.3 1 0.3 1 0.5

[0053] According to Table 1, if If the value falls within the range from a value greater than 0 to 0.04, then the factor f i Set to 10. If If the value falls within the range from greater than 0.04 to 0.1, then the factor f i Set to 5. If If the value falls within the range from greater than 0.1 to 0.2, then the factor f i Set to 2. If If the value falls within the range from greater than 0.2 to 0.3, then the factor f i Set to 1. If If the value falls within the range from greater than 0.3 to 1, then the factor f i Set it to 0.5. Generally speaking, The smaller the value of the factor f i The larger the value, the better.

[0054] While Table 1 provides example ranges and corresponding factor values, in other examples, it is possible to... Different ranges assign other factor values.

[0055] In Equation 1, the first expression calculate with f i The sum of the products. In the first expression, the sum is determined by coefficients W, which are assigned a specified constant value. Change Weighting. Typically, the value produced by the first expression of Equation 1 represents the contribution of the number of changes made to the corresponding features of the input training sample set to the score.

[0056] In Equation 1, the second expression calculate (Number of outliers for feature i (L) i ) and feature change count (C i The sum of the ratios of ( ). In the second expression, the sum is determined by the coefficient W, which has been assigned a specified constant value. Outliers Weighted, this specified constant value can be compared with W Change Same or different. Typically, the value produced by the second expression of Equation 1 represents the contribution of the number of outliers of the corresponding feature in the input training sample set to the score.

[0057] coefficient W Change sum coefficient W Outliers The relative value of determines which of the first or second expressions has a greater weight in the calculation of the Score.

[0058] In other examples, the first and / or second expressions used to calculate the Score can use aggregations other than summation. More generally, the first expression can be based on the number of changes made to the corresponding feature (C). i The first aggregation is used to calculate the first aggregation value, and the second expression can be based on the number of outliers (L). i The second aggregation is used to calculate the second aggregation value. The "aggregation" of the quantity can refer to summation, average, mean, or any other type of mathematical aggregation.

[0059] The training data poisoning detection engine 114 compares the score with a specified threshold. Typically, in some examples, a higher score indicates a greater likelihood that the training dataset 109 is poisoned. If the score exceeds the specified threshold, it indicates that the training dataset 109 has been potentially poisoned. As a result, the training data poisoning detection engine 114 can issue a poisoning alert 130.

[0060] In other examples, depending on the formula used to calculate the Score, a lower Score indicates a higher probability that the training dataset 109 is poisoned. In this latter example, the training data poisoning detection engine 114 can issue a poisoning alarm 130 when the Score falls below a specified threshold.

[0061] Figure 3 This is a block diagram of a non-transitory machine-readable or computer-readable storage medium 300 that stores machine-readable instructions that, when executed, cause the system to perform various tasks. The system includes one or more computers.

[0062] The machine-readable instructions include training sample receiving instructions 302 for receiving a plurality of training samples from a training dataset for a machine learning model, wherein each of the plurality of training samples includes a plurality of features. The plurality of training samples may be provided by a data replication manager (e.g., Figure 1 The training samples copied in 112) are copied.

[0063] The machine-readable instructions include a change quantity determination instruction 304, used to determine the number of changes made to a corresponding feature among a plurality of features. For example, the change quantity is determined by... Figure 2 The counts are represented by counters 222A to 222E.

[0064] The machine-readable instructions include poisoning score calculation instruction 306, which calculates a score representing the integrity of the training dataset based on the number of changes. This score can be calculated according to Equation 1 or any other formula.

[0065] Machine-readable instructions include training data poisoning detection instructions 308, used to detect poisoning of the training dataset based on scores. For example, the machine-readable instructions can determine whether the scores have a specified relationship with a threshold (e.g., exceeding or falling below the threshold). If the scores have a specified relationship with the threshold, the machine-readable instructions can issue a poisoning alert (e.g., ...). Figure 1 (130 in the middle).

[0066] In some examples, machine-readable instructions can further determine the number of outliers in the values ​​of the corresponding feature. This score is then based on the number of outliers. Outliers include feature values ​​that are outside the specified distribution of feature values.

[0067] In some examples, the calculation of the score involves calculating a first aggregate value based on the number of changes made to the corresponding feature, and a second aggregate value based on the number of outliers. Examples of the first aggregate are provided by the first expression of Equation 1, and examples of the second aggregate are provided by the second expression of Equation 1.

[0068] In some examples, a first coefficient is used to weight the first aggregated value, and a second coefficient is used to weight the second aggregated value.

[0069] In some examples, the first aggregation of the number of changes made to the corresponding feature includes scaling the number of changes made to the corresponding feature to produce a scaled value, and aggregating the scaled value (e.g., according to the first expression of Equation 1).

[0070] In some examples, scaling the number of changes made to a particular feature involves dividing the number of changes made to the particular feature by the total number of training samples (e.g., N_Samples).

[0071] In some examples, scaling the number of changes made to the corresponding feature includes assigning a factor (e.g., f) to the corresponding feature. i These factors are then combined with the number of changes made to the corresponding feature. The first factor is associated with which of a range of values ​​based on the first number of changes made to the first feature. Examples of these ranges include the change rate ranges included in Table 1 above.

[0072] In some examples, machine-readable instructions may calculate the change rate of a first feature based on dividing the first change number by the total number of training samples. A first factor is determined based on which range of values ​​the change rate of the first feature falls into.

[0073] In some examples, machine-readable instructions may assign a higher value to the first factor than the value of the second factor of the second feature, based on the fact that the number of first changes made to the first feature is less than the number of second changes made to the second feature.

[0074] In some examples, machine-readable instructions can identify a given feature among multiple features, where the number of changes made to that given feature exceeds a threshold. This threshold can be a value equal to the total number of training samples. Alternatively, the threshold can be a value corresponding to a certain percentage of the total number of training samples. Based on the identification that the number of changes made to the first feature exceeds the threshold, the number of changes made to the first feature is excluded when calculating the score.

[0075] In some examples, multiple training samples are included in a data replication manager that writes and replicates data to the storage system (e.g., Figure 1 The training data provided in section 112) is copied into a backup storage system (e.g., ...). Figure 1 116 in the middle).

[0076] In some examples, machine-readable instructions can identify the time point at which poisoning of the training dataset is detected, and generate an undamaged version of the training dataset from the copied training data based on the identified time point.

[0077] In some examples, generating an undamaged version of the training dataset involves selecting checkpoints from multiple checkpoints (e.g., 120A to 120B) in the copied training data. These multiple checkpoints represent different versions of the training dataset at corresponding different time points.

[0078] Figure 4 This is a block diagram of system 400 based on some examples. System 400 includes hardware processor 402 (or multiple hardware processors). The hardware processor may include a microprocessor, the core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit.

[0079] System 400 includes a storage medium 404 that stores machine-readable instructions that can be executed on a hardware processor 402 to perform certain tasks. The machine-readable instructions that can be executed on a hardware processor can refer to instructions that can be executed on a single hardware processor or instructions that can be executed on multiple hardware processors.

[0080] The machine-readable instructions in storage medium 404 include training sample collection and reception instructions 406 for receiving a set of input training samples for training a machine learning model. Each training sample in the set of input training samples includes multiple features.

[0081] The machine-readable instructions in storage medium 404 include outlier count determination instructions 408, used to determine the number of outliers in the values ​​of a corresponding feature among multiple features.

[0082] The machine-readable instructions in storage medium 404 include poisoning score calculation instructions 410, used to calculate a score representing the integrity of the training dataset based on these quantities. An example of the score is calculated according to Equation 1.

[0083] The machine-readable instructions in storage medium 404 include training data poisoning detection instructions 412, used to detect poisoning of the training dataset based on scores. If poisoning of the training dataset is detected, the machine-readable instructions can issue a poisoning alarm.

[0084] In some examples, machine-readable instructions further determine the number of changes made to a corresponding feature among multiple features, where the score is further based on the number of changes.

[0085] Figure 5 This is a flowchart of process 500 based on some examples. For example, process 500 can be... Figure 1 The training data was used to execute the poison detection engine 114.

[0086] Process 500 includes receiving (at 502) a set of input training samples for training a machine learning model, wherein each training sample in the input training sample set includes multiple features. The input training sample set may include copied training samples provided by a data replication manager. The input training sample set may include training samples within a moving time window ending at the current time.

[0087] Process 500 includes determining (at 504) the number of changes made to a corresponding feature among a plurality of features. The number of changes can be provided by a corresponding counter that counts how many changes were made to the corresponding feature.

[0088] Procedure 500 includes determining (at 506) the number of outliers among the values ​​of the corresponding feature. Procedure 500 includes calculating (at 508) a score representing the integrity of the training dataset based on the number of changes and the number of outliers.

[0089] Process 500 includes detecting poisoning of the training dataset (at 510) based on scores (e.g., by comparing scores with a threshold).

[0090] Using techniques or mechanisms exemplified by some examples of this disclosure, it is possible to detect poisoned training datasets for machine learning models and take remedial actions. Therefore, the integrity of AI systems using machine learning models can be protected. In some examples, by using copied training data provided by a data replication manager, training data poisoning detection can be performed in real time, and alerts can be issued promptly.

[0091] "Storage device" can refer to disk-based storage devices, solid-state drives, or any other type of storage device.

[0092] Storage media (e.g., Figure 3 300 or Figure 4 The 404 clause can include any one or a combination of the following: semiconductor memory devices such as DRAM or SRAM, EPROM, EEPROM, and flash memory; disks such as fixed disks, floppy disks, and removable disks; another magnetic medium, including magnetic tape; optical media such as optical discs (CDs) or digital video discs (DVDs); or another type of storage device. Note that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed across a large system having potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or article of manufacture). An article or article of manufacture can refer to any single or multiple manufactured components. One or more storage media may be located in a machine that executes the machine-readable instructions, or at a remote site from which the machine-readable instructions can be downloaded via a network for execution.

[0093] In this disclosure, unless the context clearly indicates otherwise, the terms “a,” “an,” or “the” are intended to include the plural form as well. Similarly, when used in this disclosure, the terms “includes / including / comprises / comprising” or “have / having” indicate the presence of the said element but do not preclude the presence or addition of other elements.

[0094] In the foregoing description, numerous details have been set forth to facilitate understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A non-transitory machine-readable storage medium comprising instructions that, when executed, cause a system to: receiving a plurality of training samples of a training dataset for a machine learning model, wherein, each of the plurality of training samples comprises a plurality of features; determine a number of changes made to a respective feature of the plurality of features; compute a score representing completeness of the training dataset based on the number; and detect poisoning of the training dataset based on the score.

2. The non-transitory machine-readable storage medium of claim 1, wherein, the instructions, when executed, cause the system to: determine a number of outliers in values of a respective feature, wherein the score is further based on the number of outliers.

3. The non-transitory machine-readable storage medium of claim 2, wherein, an outlier comprises a feature value that is outside of a specified distribution of values of the feature.

4. The non-transitory machine-readable storage medium of claim 2, wherein, the computation of the score comprises: computing a first aggregate value based on a first aggregation of the number of changes made to a respective feature, and computing a second aggregate value based on a second aggregation of the number of outliers.

5. The non-transitory machine-readable storage medium of claim 4, wherein, the first aggregation of the number of changes made to a respective feature comprises scaling the number of changes made to a respective feature to produce a scaled value, and aggregating the scaled values.

6. The non-transitory machine-readable storage medium of claim 5, wherein, the scaling of the number of changes made to a respective feature comprises dividing the number of changes made to a respective feature by a total number of the plurality of training samples.

7. The non-transitory machine-readable storage medium of claim 5, wherein, the scaling of the number of changes made to a respective feature comprises assigning a factor to a respective feature, and combining the factor with the number of changes made to the respective feature, wherein a first factor of the factors is based on which range of a plurality of value ranges a first number of changes made to a first feature is associated with.

8. The non-transitory machine-readable storage medium of claim 7, wherein, the instructions, when executed, cause the system to: compute a change rate of the first feature based on dividing the first number of changes by a total number of the plurality of training samples, wherein the first factor is based on which range of the plurality of value ranges the change rate of the first feature falls into.

9. The non-transitory machine-readable storage medium of claim 7, wherein, the instructions, when executed, cause the system to: assign a higher value to the first factor than a value of a second factor of a second feature based on a first number of changes made to the first feature being less than a second number of changes made to the second feature.

10. The non-transitory machine-readable storage medium of claim 1, wherein, the instructions, when executed, cause the system to: identify a first feature of the plurality of features, wherein a number of changes made to the first feature exceeds a threshold, wherein the number of changes made to the first feature is excluded from the computation of the score based on identifying that the number of changes made to the first feature exceeds the threshold.

11. The non-transitory machine-readable storage medium of claim 1, wherein, the plurality of training samples are included in replicated training data provided by a data replication manager that writes data to a storage system, wherein the data writes are replicated to persistent storage.

12. The non-transitory machine-readable storage medium of claim 11, wherein, the instructions, when executed, cause the system to: identify a point in time at which the poisoning of the training dataset is detected; and produce an uncorrupted version of the training dataset from the replicated training data based on the identified point in time.

13. The non-transitory machine-readable storage medium of claim 12, wherein, producing the uncorrupted version of the training dataset comprises: selecting a checkpoint from a plurality of checkpoints in the replicated training data, the plurality of checkpoints comprising different versions of the training dataset at respective different points in time.

14. A system comprising: a processor; and a non-transitory storage medium storing instructions executable on the processor to: receive a set of input training samples of a training dataset, the training dataset used to train a machine learning model, wherein each training sample of the set of input training samples comprises a plurality of features; determine a number of outliers in values of a respective feature of the plurality of features; compute a score representing integrity of the training dataset based on the number; and detect poisoning of the training dataset based on the score. the instructions executable on the processor to:

15. The system of claim 14, wherein, determine a number of changes made to a respective feature of the plurality of features, wherein the score is further based on the number of changes. the computation of the score comprises:

16. The system of claim 14, wherein, computing a first aggregate value based on a first aggregation of the number of changes made to a respective feature, and computing a second aggregate value based on a second aggregation of the number of outliers. the computation of the score comprises:

17. The system of claim 16, wherein, weighting the first aggregate value using a first coefficient, and weighting the second aggregate value using a second coefficient. the detection of poisoning of the training dataset comprises comparing the score to a specified threshold.

18. The system of claim 14, wherein, 19. A method comprising: receiving a set of input training samples of a training dataset, the training dataset used to train a machine learning model, wherein each training sample of the set of input training samples comprises a plurality of features; determining, by a system comprising a hardware processor, a number of changes made to a respective feature of the plurality of features; determining, by the system, a number of outliers in values of the respective feature; computing, by the system, a score representing integrity of the training dataset based on the number of changes and the number of outliers; and detecting, by the system, poisoning of the training dataset based on the score.

20. The method of claim 19, further comprising: identifying a first feature of the plurality of features, wherein the number of changes made to the first feature exceeds a threshold, wherein the number of changes made to the first feature is excluded from the computation of the score based on identifying that the number of changes made to the first feature exceeds the threshold. ​

Images