DETECTION OF TRAINING DATA POISONING

Abstract

In some examples, a system receives a multitude of training samples of a training dataset for a machine learning model, where each training sample within the multitude comprises a multitude of features. The system determines the quantities of changes made to each feature within the multitude, calculates a score based on these quantities that represents the integrity of the training dataset, and detects any poisoning of the training dataset based on this score.

Description

background

[0001] Artificial intelligence is increasingly used in computing environments to promote efficiency and innovation in the delivery of services and / or products. Artificial intelligence demonstrated by machines relies on the use of machine learning models that can learn from a knowledge base based on any or a combination of the following elements: data from past activities, training data, or data from other sources. Brief description of the drawings

[0002] Some embodiments of the present disclosure are described with reference to the following figures. Fig. Figure 1 is a block diagram of a computer system that includes a training data poisoning detection machine, according to some examples. Fig. Figure 2 is a block diagram showing an input collection of training samples and counters for tracking changes to features in the training samples, according to some examples. Fig. Figure 3 is a block diagram of a storage medium that stores machine-readable instructions according to some examples. Fig. Figure 4 is a block diagram of a system according to some examples. Fig. Figure 5 is a flowchart of a procedure according to some examples.

[0003] In the drawings, identical reference numbers denote similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated to make the example shown clearer. Furthermore, the drawings contain examples and / or embodiments that correspond to the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description

[0004] A training dataset can be used to train a machine learning model and make predictions. A training dataset comprises training patterns, where a "pattern" can refer to a dataset containing values ​​of the respective attributes. The attributes are also referred to as features of a sample. The training patterns of a training dataset can include labeled training patterns, where a collection of features of each given pattern is assigned a label (selected from several labels) that corresponds to the collection of features. For example, an initial collection of features in an initial training sample might contain values ​​that collectively indicate an attack on a computer environment. Consequently, the initial collection of features in the initial sample can be assigned an initial label (e.g., a label "Attack Detected").On the other hand, a second set of features in a second training sample might contain values ​​that, taken together, indicate that no attack is taking place with respect to the computer environment. In the latter case, a second label (e.g., the label "attack not detected") can be assigned to the second set of features in the second sample. In other contexts, training patterns can be labeled with other types of labels.

[0005] An attacker (e.g., a human, malware, or a machine) may attempt to influence the predictions of a machine learning model by altering a training dataset used to train the model. Such a modification of the training dataset results in its poisoning. The machine learning model trained on the poisoned training dataset will produce inaccurate results. For example, the machine learning model might predict that an attack will go undetected in a computer environment, even though an attack is actually occurring. Incorrect results from the machine learning model can lead to security breaches that compromise the integrity of the computer environment, endanger data stored in the computer environment, or enable data theft. Training datasets can also be poisoned by other causes, such as...This can occur due to data errors or malfunctions in the operation of machines or programs that generate the training datasets. The term "poisoning" of training data, as used here, refers to any (intentional or unintentional) alteration of the training data that causes a machine learning model to produce inaccurate results based on an input dataset.

[0006] In accordance with some examples in this disclosure, techniques or mechanisms are provided to detect corruption of a training dataset used to train a machine learning model, based on one or more of the following quantities: the amount of changes made to respective features in training samples of the training dataset, and the amount of outliers in the values ​​of the respective features in the training samples. Based on the aforementioned quantities, a score is calculated that represents the integrity of the training dataset based on these quantities. This score is used to detect corruption of the training dataset.

[0007] Fig. Figure 1 is a block diagram of an exemplary arrangement comprising a computer system 102, which includes a training data generator 104 that generates or updates training data for a machine learning model 106. The computer system can be implemented with one or more computers. The machine learning model 106 can be executed in a computer system separate from the computer system 102, or alternatively, the machine learning model 106 can be executed in the computer system 102.

[0008] The training data generator 104 can be implemented with machine-readable instructions that are executed by a processing resource in the computer system 102. Alternatively or additionally, the training data generator 104 can be implemented with hardware processing circuits. A source of training data for the machine learning model 106 can also be an external source outside the computer system 102, such as a human, a program, or a machine. The external source can create or update training data for the machine learning model 106.

[0009] Training data (generated by the training data generator 104 and / or an external source) can be written to a primary storage system 108 containing one or more storage devices. The training data generator 104 and / or the external source can issue write requests to write the training data. These write requests are processed by a driver 110 in the computer system 102. In response to the write requests, the driver 110 issues write transactions to write the training data to a training data set 109 in the primary storage system 108.

[0010] The training dataset 109 can conform to a specific format, e.g., a columnar file format (e.g., a relational table format in Structured Query Language (SQL) databases), a tabular file format (e.g., an Excel format, a CSV (Comma Separated Value) format, etc.), a nested file format (e.g., an XML (Extensible Markup Language) format, a JSON (JavaScript Object Notation) format, etc.), or another format.

[0011] The training dataset 109 is used to train the machine learning model 106. After training, the machine learning model 106 can recognize patterns in the input data. The machine learning model 106 generates outputs that represent predictions made by the machine learning model 106 based on the input data. The outputs of the machine learning model 106 are used by one or more consumers to perform various actions. A consumer of the outputs of the machine learning model 106 can be a user, a program, or a machine.

[0012] Driver 110 is a program that manages access to the primary storage system 108. In some examples, driver 110 is part of an operating system (OS). In other examples, where a virtual computing environment is implemented in computer system 102, driver 110 may be part of a virtualization management program, such as a hypervisor or a container engine. The hypervisor creates and manages virtual machines (VMs) in computer system 102. The container engine creates and manages containers in computer system 102.

[0013] In some examples, the computer system 102 also includes a data replication manager 112 and a training data poisoning detection machine 114. Both the data replication manager 112 and the training data poisoning detection machine 114 can be implemented using machine-readable instructions that can be executed on the processing resource of the computer system 102. In other examples, the data replication manager 112 and the training data poisoning detection module 114 can be implemented using one or more hardware processing circuits.

[0014] The data replication manager 112 replicates data written to a backup storage system 116 with one or more storage devices. In some examples, the backup storage system 116 is located outside the computer system 102. In other examples, the backup storage system 116 may be located inside the computer system 102. The primary storage system 108 stores training data (for example, the training dataset 109) for application to the machine learning model 106. The backup storage system 116 stores a replication of the training data for use in restoring the training data.

[0015] Backup storage system 116 can be physically separate from primary storage system 108. Alternatively, backup storage system 116 can be part of the same physical storage infrastructure but logically separate from primary storage system 108.

[0016] Input / output (I / O) operations between the driver 110 and the primary storage system 108 can include read I / O operations and write I / O operations. The data replication manager 112 is able to detect the I / O write operations and replicate them to the backup storage system 116. A "replication" of a data write operation can refer to storing a representation of a write I / O operation in the backup storage system 116. The representation of the I / O write operation can include modified data (e.g., new data, modified data, or deleted data). The representation of the I / O write operation can also include information about the type of write operation, such as an insert operation to add new data, an update operation to modify data, or a delete operation to erase data.

[0017] Data replication results in a replicated training dataset 118 being stored in the backup storage system 116. The replicated training dataset 118 contains checkpoints 120A to 120B, corresponding to different points in time. A "checkpoint" in the replicated training dataset 118 comprises a version of the training data at a specific point in time. Different checkpoints in the replicated training dataset 118 can be created at different points in time. A checkpoint in the replicated training dataset 118 can be used to restore training data in the training dataset 109 if part of the training dataset 109 is lost or corrupted.

[0018] The training data corruption detection engine 114 can be applied to replicated training data while the replicated training data is being written to the backup storage system 116. The training data corruption detection engine 114 applies its analysis to an input collection of training samples written to the replicated training dataset 118. The input collection of training samples to which the training data corruption detection engine 114 applies its analysis can contain training samples within a specific time interval, such as a time window of a certain length. The time window has a range that begins at T1 and ends at T2, where T2 can be the current time and T1 and T2 define the specified length.In such examples, the training data poisoning detection program 114 applies its analysis to a recent time window, which is a moving time window that shifts with the current time. In other examples, the input collection of training samples can be selected differently. For example, the training patterns can be randomly selected when the replicated training data is written to the backup storage system 116.

[0019] Each training sample contains a collection of features and a label assigned to that collection of features. The label can be assigned by a human, a program, or a machine. The features of a training sample can include numeric features and / or categorical features. A numeric feature is assigned numeric values ​​from a range of numeric values, while a categorical feature is assigned a categorical value from a discrete set of categorical values.

[0020] Engine 114 for training data poisoning detection can perform real-time training data poisoning detection. This real-time detection is based on the analysis of training samples in replicated training data by the training data poisoning detection engine 114, while the data replication manager 112 writes the replicated training data to the backup storage system 116.

[0021] When the training data poisoning detection engine 114 detects potential poisoning of training data in the input collection of training samples, it issues a poisoning alert 130, which may be in the form of a message, an information element, or another type of indicator. The poisoning alert 130 indicates that the training dataset 109 used by the machine learning model 106 is poisoned. The poisoning alert 130 may include a timestamp indicating when the potential poisoning of the training data was detected.

[0022] The poison alert 130 is sent by the training poison detection machine 114 to a remediation machine 132, which can take a remedial action in response to the poison alert 130. Examples of remedial actions may include: issuing a warning to a target entity (e.g., a human administrator, a program, or a machine), disabling the machine learning model 106, disabling a computer system in which the machine learning model 106 is running (e.g., by shutting down the computer system), disabling a network connection of the computer system in which the machine learning model 106 is running, or any other remedial action.

[0023] Furthermore, the Remediation Engine 132 can retrieve a checkpoint (e.g., 120A or 120B) from the replicated training dataset 118 to restore the poisoned training dataset 109 to a previous state. The retrieved checkpoint was created before the timestamp contained in the poisoning alert 130. Since the poisoning alert 130 was generated by the Poisoning Detection Engine 114 based on real-time poisoning detection in the training data, the timestamp in the poisoning alert 130 likely represents the approximate time at which the poisoning of the training dataset 109 occurred. Therefore, a checkpoint created before the timestamp (or a certain threshold time interval before the timestamp) of the poisoning alert 130 is likely to contain uncontaminated training data.

[0024] The Poisons Detection Engine 114, which uses training data, can trigger Poisons Alert 130 based on one or more criteria. A primary criterion is whether a training pattern (which can be a new or modified training pattern) matches a training data schema (referred to as the "training data schema"). The format of the training data can be defined by the training data schema. For example, the training data schema specifies which features are included in each training sample and which values ​​(e.g., a range of values ​​or a set of categorical values) are possible for each feature.

[0025] If the training data poisoning detection module 114 determines that a training sample (or a certain set of training samples) in the input collection of training samples does not conform to the training data schema, then it is likely that the training sample has been tampered with, and therefore the training data poisoning detection module 114 triggers the poisoning alarm 130.

[0026] A second criterion concerns whether the training samples have been completely deleted. In most cases, completely deleting a training sample can be a legitimate measure. For example, training patterns might be deleted as part of data cleansing. Consequently, deleting training patterns would not cause the training data poisoning detection system 114 to trigger the poisoning alarm 130.

[0027] A third criterion relates to whether the characteristics of a subset (fewer than all) of the training samples in the input collection of training samples have been modified. Modifying individual characteristics of a training sample can potentially be associated with an attack on the training data, especially if the modifications are made to characteristics of some training samples but not others. It should be noted that modifying the values ​​of a single characteristic (or a subset of characteristics) of all training samples in the input collection of training samples can be considered a legitimate action. For example, modifying the values ​​of a single characteristic (or a subset of characteristics) of all training samples in the input collection of training samples might be part of a data scaling or data transformation operation.Therefore, changes to the values ​​of a single feature (or a subset of features) of all training samples in the input collection of training samples would not cause the training data poisoning detection system 114 to trigger the poisoning alarm 130. However, changes to the values ​​of features in some training samples, but not in other training samples of the input collection of training samples, would be considered by the training data poisoning detection machine 114 as an indication of training data poisoning.

[0028] More generally, changes in the values ​​of a single feature (or a subset of features) that exceed a threshold set of training samples in the input pool would not trigger the training data poisoning detection engine 114, which then raises the poisoning alarm 130. However, changes in feature values ​​in some training samples (less than the threshold set), but not in other training samples in the input pool, would be considered by the training data poisoning detection engine 114 as an indication of training data poisoning. The threshold set can be based on a relative percentage (e.g., 99%, 95%, 90%, 80%, etc.) of the total number of training samples in the input pool.In other examples, the third criterion relates to determining whether changes to the training samples match a target pattern of changes resulting from common or expected data transformations that can be applied to the training samples. If the changes do not match the target pattern, this may indicate training data poisoning.

[0029] Another point to consider is that some features of the training patterns may be categorical or even textual, and these features can also be transformed. An example of transforming a categorical feature is converting {-1, 1} to {0, 1} to prepare some machine learning models or encoding labels (e.g., "Red" → 0, "Green" → 1, "Blue" → 2). Thus, non-numeric features can either be ignored or, after a data preparation phase in which the non-numeric features can be transformed, not checked for changes.

[0030] A fourth criterion relates to whether features of training samples in the input set of training samples contain outlier values, and whether the presence of these outliers satisfies one or more specific conditions. In some examples, an outlier refers to a value of a feature that lies outside an expected set of values ​​based on a distribution of values ​​of the feature, where the distribution of values ​​may be based on observed values ​​of the feature. The "observed" values ​​of the feature may refer to the values ​​of the feature within the input set of training samples, or alternatively, to values ​​of the feature within a larger set of training samples (e.g., a historical set of training samples). In other examples, an outlier may be statistically determined.For example, a value of a characteristic is considered an outlier if it deviates from the mean of the observed values ​​of that characteristic by more than a certain number of standard deviations. The presence of outliers that meet the following conditions may indicate the input of synthetic or manipulated data. The first condition relates to whether the number of outliers increases after the training data has been processed. If this first condition is met, it may indicate training data poisoning. The second condition relates to whether the variance of the outlier values ​​deviates from a previous standard deviation. If this second condition is met, it may also indicate training data poisoning.

[0031] Fig. Figure 2 is a block diagram of an input collection of training patterns 200, where each training pattern has the features A, B, C, D, and E. The input collection of training patterns 200 comprises 5 training patterns: 202, 204, 206, 208, and 210. A shaded cell in the input collection of training patterns 200 indicates a change in the value of a feature, for example, due to a write operation that adds a new value or modifies an existing value of the feature. Although Fig. Example 2 shows that the initial set of training patterns contains 200 training patterns. However, in other examples, the initial set of training patterns may contain a different number of training patterns. Furthermore, in other examples, a training sample may contain a smaller or larger number of features than in the example above. Fig. 2 shown.

[0032] Each of the features A through E is associated with a corresponding counter. For example, feature A is associated with counter 222A, feature B with counter 222B, feature C with counter 222C, feature D with counter 222D, and feature E with counter 222E. Each counter tracks how many changes were made to the respective feature across the training samples in the input collection of training samples 200. Thus, counter 222A tracks the number of changes made to feature A in training samples 202, 204, 206, 208, and 210. In the example of Fig. In step 2, the values ​​of feature A in training samples 202 and 204 were changed. Consequently, the counter 222A was incremented to 2 to represent the two changes made.

[0033] Counter 222B tracks the number of changes made to feature B in training samples 202, 204, 206, 208, and 210. In this example, the values ​​of feature B were changed in all training samples 202, 204, 206, 208, and 210. Consequently, counter 222B was incremented to 5 to indicate the five changes made.

[0034] Counter 222C tracks the number of changes made to feature C in training samples 202, 204, 206, 208, and 210. In this example, the values ​​of feature C were changed in training samples 202 and 204. Consequently, counter 222C was incremented to 2 to represent the two changes.

[0035] Counter 222D tracks the number of changes made to feature D in practice examples 202, 204, 206, 208, and 210. In this example, the value of feature D was changed in practice example 204. Consequently, counter 222D was incremented to 1 to represent the single change.

[0036] Counter 222E tracks the number of changes made to feature E in training samples 202, 204, 206, 208, and 210. In this example, the value of feature E was changed in training sample 204. Consequently, counter 222E was incremented to 1 to represent the single change.

[0037] As mentioned above, according to the third criterion, changing a single characteristic of all training samples in the initial collection of training samples can be considered a legitimate action. In the example of Fig. 2. Since the counter 222B for the input collection of training samples, which contains 5 training samples, has been incremented to 5, the changes to feature B can be ignored by the training data poisoning detection machine 114. In other words, the value of counter 222B can be ignored by the training data poisoning detection machine 114 when calculating a score related to whether training data poisoning has occurred.

[0038] The following is an example of how the score is calculated by the training data recognition engine 114. The following parameters are defined.

[0039] The parameter N_Samples represents the total number of training samples in an input collection of training samples. Fig. 2, .N_Samples = 5

[0040] A parameter C i represents the number of changes made to feature i. For example, for Fig. 2: CA = 2 for feature A, C C = 2 for feature C, C D = 1 for feature D and C E = 1 for feature E. Note that the count C B The value of 5 for characteristic B is disregarded because the values ​​of characteristic B were changed in all training samples of the initial collection of 200 training samples. The number C is not taken into account. B Feature B can be achieved in two ways. First, the number C can be increased. B from equation 1. Secondly, the number C can be B set to 0 and included in equation 1.

[0041] A parameter L i represents a set of outliers for the feature i. L A represents a set of outliers for feature A, L C represents a set of outliers for feature C, L D represents a large number of outliers for feature D, and L Erepresents a set of outliers for the feature E. In some examples, L i ≤ C i .

[0042] In one example, a value (score) representing the integrity of the training dataset 109 can be calculated as follows: Score=WChange∑iCiN_Samples⋅fi+WOutliers∑iLiCi.

[0043] Equation 1 states CiN_Samples for the ratio between the number of changes made to feature i and the total number of training patterns (N_Samples) in the input collection of training patterns. This ratio (CiN_Samples) is referred to as the "change rate".

[0044] Each change ratio (CiN_Samples) is multiplied by a factor f i multiplied according to the value of CiN_Samples is calculated, e.g. according to Table 1 below. The factors are used to scale the changes made to the respective characteristics (C i ) is used to obtain scaled values. These scaled values ​​are then used in the first expression. (WChange∑iCiN_Samples⋅fi) summed from equation 1. The factor f i is used to detect smaller amounts of changes in feature i in the input collection of training patterns by preventing large counts (i.e., large values ​​of C) from being detected. i ) dominate the calculation of the score. TABLE 1 Start Ende Faktor 0 0.04 10 0.04 0.1 5 0.1 0.2 2 0.2 0.3 1 0.3 1 0.5

[0045] According to Table 1, the factor f i set to 10 if CiN_Samples falls within the range that begins with a value greater than 0 and ends at 0.04. CiN_Samples If the value falls within the range that begins with a value greater than 0.04 and ends at 0.1, the factor f is used. i set to 5. If CiN_Samples If the value falls within the range that begins with a value greater than 0.1 and ends at 0.2, the factor f is used. i set to 2. If CiN_Samples C i If the value falls within the range that begins at a value greater than 0.2 and ends at 0.3, the factor f is used. i set to 1. If CiN_Samples If the range falls within the period that begins at , starts at a value greater than 0.3 and ends at 1, the factor f is used. i set to 0.5. Generally speaking: The smaller the value of CiN_Samples The larger the value, the greater the value of the factor f. i .

[0046] Although Table 1 provides example ranges and corresponding factor values, other examples may use different factor values ​​for various ranges. CiN_Samples be assigned.

[0047] In Eq. 1, the first expression calculates (WChange∑iCiN_Samples⋅fi) a sum of the product of CiN_Samples and f i In the first expression, the sum is divided by a coefficient W. Change weighted, to which a specific constant value is assigned. In general, the value generated by the first expression of Equation 1 represents the contribution of the changes made to the respective characteristics of the input collection of training samples to the result (score).

[0048] In Eq. 1, the second expression calculates (WOutliers∑iLiCi) a sum of LiCi (the ratio of a set of outliers (L i ) of feature i for the number of changes (C i ) of feature i). In this second expression, the sum is given by a coefficient W. Outliersweighted, to which a fixed constant value is assigned that is equal to or different from W Change It can be. In general, the value generated by the second expression of equation 1 represents the contribution of the sets of outliers of the respective features of the input collection of training samples to the evaluation (score).

[0049] The relative values ​​of the coefficient W Change and the coefficient W Outliers determine which of the two expressions receives more weight in the calculation of the score.

[0050] In other examples, the first expression and / or the second expression can use other types of aggregation besides a sum to calculate the score. More generally, the first expression can provide an initial aggregation value based on an initial aggregation of the change quantities applied to the respective characteristics (C). i) calculate, and the second expression can calculate a second aggregation value based on a second aggregation of the outlier sets (L i ) calculate. An "aggregation" of quantities can refer to a sum, an average, a mean, or any other type of mathematical aggregation.

[0051] Engine 114, used for detecting training data poisoning, compares the score to a specified threshold. In some examples, a higher score generally indicates a greater probability of poisoning of the training dataset 109. If the score exceeds the specified threshold, it means that potential poisoning of the training dataset 109 has occurred. Consequently, the training data poisoning detection engine 114 can issue the poisoning warning 130.

[0052] In other examples, a lower Score value indicates a greater probability of poisoning of the training dataset 109, depending on the formula used to calculate the Score. In such latter examples, the poisoning detection engine 114 can issue the poisoning warning 130 if the Score falls below a certain threshold.

[0053] Fig. Figure 3 is a block diagram of a non-transitory machine-readable or computer-readable storage medium 300 that stores machine-readable instructions which, when executed, cause a system to perform various tasks. The system comprises one or more computers.

[0054] The machine-readable instructions include instructions for receiving training patterns 302 to receive a multitude of training patterns from a training dataset for a machine learning model, where each training pattern in the multitude contains a multitude of features. The multiple training patterns may include replicated training patterns that have been replicated by a data replication manager (e.g., 112 in Fig. 1).

[0055] The machine-readable instructions contain instructions for determining the change quantity 304, in order to determine the quantity of changes made to the respective characteristics of the multitude of characteristics. The change quantities are determined, for example, by the counter readings of counters 222A to 222E. Fig. 2 shown.

[0056] The machine-readable instructions include instruction 306 for calculating the poison score, in order to calculate a score that represents the integrity of the training dataset based on the change quantities. The value can be calculated according to Equation 1 or another formula.

[0057] The machine-readable instructions contain instructions for training data poisoning detection (308) to detect poisoning of the training dataset based on the result. For example, the machine-readable instructions can determine whether the score is in a certain ratio to a threshold (e.g., exceeds or falls below the threshold). If the score is in a certain ratio to the threshold, the machine-readable instructions can issue a poison warning (e.g., 130 in Fig. 1).

[0058] In some examples, the machine-readable instructions can also determine the set of outliers in the values ​​of the respective characteristics. The evaluation is further based on the set of outliers. An outlier is a value of a characteristic that lies outside a specific distribution of values ​​for that characteristic.

[0059] In some examples, the score calculation involves calculating a first total score based on a first aggregation of the sets of changes made to the respective features, and calculating a second total score based on a second aggregation of the sets of outliers. An example of the first aggregation is given by the first expression of Equation 1, and an example of the second aggregation is given by the second expression of Equation 1.

[0060] In some examples, the first total value is weighted with a first coefficient and the second total value with a second coefficient.

[0061] In some examples, the first aggregation of the change quantities made to the respective characteristics includes scaling the change quantities made to the respective characteristics to produce scaled values, and aggregating the scaled values ​​(e.g., according to the first expression of Eq. 1).

[0062] In some examples, scaling the change quantities made to the respective characteristics involves dividing the change quantities made to the respective characteristics by a total set of the multitude of training samples (e.g., BN_Samples).

[0063] In some examples, scaling the amounts of changes made to the respective characteristics involves assigning factors (e.g., f). i) to the respective characteristics and the combination of the factors with the quantities of changes made to the respective characteristics. A first factor is based on which range of a multitude of value ranges a first set of changes made to a first characteristic is assigned to. An example of the multitude of ranges are the ranges of change ratios listed in Table 1.

[0064] In some examples, the machine-readable instructions can calculate a change ratio for the first feature, based on dividing the initial set of changes by the total set of the multitude of training samples. The first factor is based on the range of the multitude of value ranges into which the change ratio for the first feature falls.

[0065] In some examples, the machine-readable instructions can assign a higher value to the first factor than to a value of a second factor for a second feature, based on the first set of changes made to the first feature being smaller than a second set of changes made to the second feature.

[0066] In some examples, the machine-readable instructions can identify a specific feature from a set of features for which the number of changes made to that feature exceeds a threshold. The threshold can be a value equal to the total number of training samples. Alternatively, this threshold can be a value equal to a percentage of the total number of training samples. The number of changes made to the first feature is excluded from use in calculating the score if it is determined that the number of changes made to the first feature exceeds the threshold.

[0067] In some examples, the multitude of training samples is contained in replicated training data managed by a data replication manager (e.g., 112 in Fig. 1) be provided, which replicates data write operations to a storage system. The written data is replicated to a backup storage system (e.g., 116 in Fig. 1).

[0068] In some examples, the machine-readable instructions can identify a point in time at which the poisoning of the training dataset is detected and generate an uncorrupted version of the training dataset from the replicated training data based on the identified poisoning.

[0069] In some examples, generating the uncorrupted version of the training dataset involves selecting a control point from a multitude of control points (e.g., 120A to 120B) in the replicated training data. This multitude of control points comprises different versions of the training dataset at different time points in time.

[0070] Fig. Figure 4 is a block diagram of a System 400 according to some examples. The System 400 comprises a hardware processor 402 (or multiple hardware processors). A hardware processor can include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or other hardware processing circuitry.

[0071] The System 400 includes a storage medium 404 on which machine-readable instructions are stored that can be executed on the hardware processor 402 to perform specific tasks. Machine-readable instructions executable on a hardware processor can refer to instructions that can be executed on a single hardware processor or on multiple hardware processors.

[0072] The machine-readable instructions in storage medium 404 contain instructions for receiving training samples 406 to receive an input collection of training samples for a training dataset, the training dataset being used to train a machine learning model. Each training sample in the input collection of training samples comprises a variety of features.

[0073] The machine-readable instructions in the storage medium 404 contain instructions for determining the outlier set 408, in order to determine the quantity of outliers in the values ​​of the respective features of the multitude of features.

[0074] The machine-readable instructions in storage medium 404 contain instructions for calculating the toxicity value 410, in order to calculate a value that represents the integrity of the training dataset based on the quantities. An example of the value is calculated according to Equation 1.

[0075] The machine-readable instructions in storage medium 404 contain instructions for training data poisoning detection 412, to detect poisoning of the training dataset based on the result. If poisoning of the training dataset is detected, the machine-readable instructions can issue a poisoning warning.

[0076] In some examples, the machine-readable instructions further determine the quantities of changes made to the respective features of the multitude of features, with the evaluation further based on the quantities of changes.

[0077] Fig. Figure 5 is a flowchart of a process 500 according to some examples. Process 500 can, for example, originate from the training data poisoning detection machine 114. Fig. 1 will be carried out.

[0078] Process 500 involves receiving (at 502) an input collection of training samples for a training dataset, where the training dataset is used to train a machine learning model. Each training sample in the input collection comprises a variety of features. The input collection of training samples may include replicated training samples provided by a data replication manager. The input collection of training samples may contain training samples within a sliding time window that ends at the current time.

[0079] Process 500 involves determining (in 504) the number of changes made to each of the many features. The number of changes can be provided by counters that count how many changes were made to each feature.

[0080] Process 500 involves determining (at 506) the number of outliers in the values ​​of the respective features. Process 500 also involves calculating (at 508) a score that represents the integrity of the training dataset based on the number of changes and the number of outliers.

[0081] Process 500 includes the detection (at 510) of poisoning of the training data set based on the result, for example by comparing the result with a threshold.

[0082] Using techniques or mechanisms as described in some examples in this disclosure, poisoned training datasets for machine learning models can be detected and remediation measures can be taken. As a result, the integrity of an AI system using a machine learning model can be protected. In some examples, the use of replicated training data provided by a data replication manager enables real-time detection of poisoned training data and the issuance of a timely warning.

[0083] A “storage device” can refer to a disk-based storage device, a solid-state drive, or any other type of storage device.

[0084] A storage medium (e.g., 300 in Fig. 3 or Fig. 404 in Fig.4) may include one or a combination of the following: a semiconductor storage device such as DRAM or SRAM, EPROM, EEPROM, and flash memory; a magnetic disk such as a hard disk, floppy disk, or removable disk; another magnetic medium, including tape; an optical medium such as a compact disc (CD) or a digital video disc (DVD); or another type of storage device. It should be noted that the instructions discussed above may be provided on a single computer- or machine-readable storage medium, or alternatively, on multiple computer- or machine-readable storage media distributed throughout a large system, possibly with multiple nodes. Such computer- or machine-readable storage medium or media are considered part of an article (or article to be manufactured).An article or manufactured item can refer to each individual component produced or to multiple components. The storage medium or media can be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions can be downloaded for execution over a network.

[0085] In this disclosure, the use of the term "a," "an," or "the" includes the plural forms unless the context clearly indicates otherwise. Similarly, the terms "includes," "including," "comprises," "have," or "have," when used in this disclosure, specify the presence of the elements indicated but do not exclude the presence or addition of other elements.

[0086] The foregoing description includes numerous details to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and deviations from the details described above. It is intended that the accompanying claims cover such modifications and variations.

Claims

[1] A non-transitory, machine-readable storage medium containing instructions which, when executed, cause a system to: to receive a multitude of training samples of a training dataset for a machine learning model, where each training sample of the multitude of training samples comprises a multitude of features; To determine the quantities of changes made to the respective characteristics of the multitude of characteristics; to calculate a score that represents the integrity of the training dataset based on the sets; and to detect the poisoning of the training dataset based on the score. [2] Non-transitory, machine-readable storage medium according to claim 1, wherein the instructions, upon execution, cause the system to: to determine outlier quantities in the values ​​of the respective characteristics, whereby the score is also based on the quantities of outliers. [3] Non-transient, machine-readable storage medium according to claim 2, wherein an outlier comprises a value of a feature that lies outside a specified distribution of values ​​of the feature. [4] Non-transitory, machine-readable storage medium according to claim 2, wherein the calculation of the score comprises: Calculating an initial aggregation value based on an initial aggregation of the change quantities made to the respective characteristics, and Calculating a second aggregation value based on a second aggregation of the outlier sets. [5] Non-transitory, machine-readable storage medium according to claim 4, wherein the first aggregation of the change quantities made to the respective features comprises scaling the change quantities made to the respective features to generate scaled values, and aggregating the scaled values. [6] Non-transitory, machine-readable storage medium according to claim 5, wherein scaling the change quantities made to the respective features comprises dividing the change quantities made to the respective features by a total set of the plurality of training samples. [7] Non-transitory, machine-readable storage medium according to claim 5, wherein scaling the change quantities made to the respective features comprises assigning factors to the respective features and combining the factors with the change quantities made to the respective features, wherein a first factor of the factors is based on which range from a plurality of value ranges a first change quantity made to a first feature is assigned. [8] Non-transitory, machine-readable storage medium according to claim 7, wherein the instructions, upon execution, cause the system to: to calculate a change ratio for the first characteristic based on dividing the first change quantity by a total quantity of the multitude of training samples, where the first factor is based on which range of the multitude of value ranges the change ratio for the first characteristic falls into. [9] Non-transitory, machine-readable storage medium according to claim 7, wherein the instructions, upon execution, cause the system to: to assign a higher value to the first factor than to a value of a second factor for a second characteristic, based on the first change made to the first characteristic being smaller than a second change made to the second characteristic. [10] Non-transitory, machine-readable storage medium according to claim 1, wherein the instructions, upon execution, cause the system to: to identify a first characteristic from the multitude of characteristics for which a change quantity made to the first characteristic exceeds a threshold, where the amount of change made to the first characteristic is excluded from use in calculating the score based on the identification that the amount of change made to the first characteristic exceeds the threshold. [11] Non-transitory, machine-readable storage medium according to claim 1, wherein the plurality of training samples is contained in replicated training data provided by a data replication manager which replicates data writes to a storage system, wherein the data writes are replicated to a permanent storage. [12] Non-transitory, machine-readable storage medium according to claim 11, wherein the instructions, upon execution, cause the system to: to identify a point in time at which the poisoning of the training dataset is detected; and to generate an uncorrupted version of the training dataset from the replicated training data, based on the identified time point. [13] Non-transient, machine-readable storage medium according to claim 12, wherein generating the uncorrupted version of the training data set comprises: Selecting a control point from a multitude of control points in the replicated training data, where the multitude of control points comprises different versions of the training dataset at different times. [14] A system that includes the following: a processor; and a non-transitory storage medium that stores instructions that can be executed on the processor to: to receive an input collection of training samples for a training dataset, wherein the training dataset is used to train a machine learning model, with each training sample of the input collection of training samples comprising a variety of features; To determine outlier sets in the values ​​of the respective characteristics from the multitude of characteristics; to calculate a score that represents the integrity of the training dataset based on the sets; and to detect the poisoning of the training dataset based on the score. [15] System according to claim 14, wherein the instructions are executable on the processor to: To determine the quantities of changes made to the respective characteristics of the multitude of characteristics, the score is also based on the amount of change. [16] System according to claim 14, wherein the calculation of the score comprises: Calculating an initial aggregation value based on an initial aggregation of the change quantities made to the respective characteristics, and Calculating a second aggregation value based on a second aggregation of the outlier sets. [17] System according to claim 16, wherein the calculation of the score comprises: Weights of the first aggregation value with a first coefficient, and Weighting the second aggregation value with a second coefficient. [18] System according to claim 14, wherein the detection of poisoning of the training data set comprises comparing the score with a certain threshold. [19] A procedure that includes the following: Receiving an input collection of training samples for a training dataset, wherein the training dataset is used to train a machine learning model, with each training sample of the input collection of training samples comprising a variety of features; Determine, by a system comprising a hardware processor, the quantities of changes made to each feature of the multitude of features; Determining outlier sets in the values ​​of the respective characteristics using the system; The system calculates a score that represents the integrity of the training dataset based on the sets of changes and outliers; and Detecting poisoning of the training dataset by the system based on the score. [20] The method of claim 19, which further comprises: Identifying a first characteristic from the multitude of characteristics for which a change quantity made to the first characteristic exceeds a threshold, where the amount of change made to the first characteristic is excluded from use in calculating the score based on the identification that the amount of change made to the first characteristic exceeds the threshold.

Images