A method for identifying whether a classification system is configured to use a specific machine learning classification model.
By generating test sets associated with specific machine learning models and utilizing classification result matching checks, the problem of verifying the identity and integrity of machine learning models is solved, ensuring the legitimate use and security of the models.
Patent Information
- Application Number
- CN202211464954.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-11-22
- Filing Date
- 2022-11-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-11-22
AI Technical Summary
Existing technologies struggle to effectively verify the identity and integrity of machine learning models, preventing unauthorized use and modification, especially in scenarios where deployment could lead to serious consequences.
By generating test sets associated with a specific machine learning classification model, and using multiple test samples to check the matching of classification results, it is possible to identify whether the classification system uses a specific model. This includes generating and updating test samples to detect changes in classification results caused by minor modifications.
It implements authentication and integrity verification of machine learning models, ensuring the legitimate use of models and preventing unauthorized modifications, thereby improving the security and reliability of model deployment.
Smart Images

Figure CN116150648B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to: a method for identifying whether a classification system is configured to use a particular machine learning classification model; a method for generating a test set for use in such a method for identifying whether a classification system is using a particular machine learning classification model; and an apparatus and computer program for performing such a method. Background Technology
[0002] Artificial intelligence (AI) or machine learning (ML) technologies are being used in an increasing number of deployment scenarios, especially as the processing power of devices / systems continues to improve. These technologies involve creating and training models—trained models that can then be used for any purpose intended by AI / ML. Training typically uses large amounts of training data (or training samples), which can be costly and time-consuming to acquire and maintain, and may be commercially sensitive. The training process itself can take a significant amount of time to complete. It is well known that many different types of ML algorithms exist for creating / building models—these different types of ML algorithms yield models with different structures, which are trained and operate in different ways, and have different representations. For example, ML algorithms such as artificial neural networks, decision trees, support vector machines, etc., are all well-known and can be implemented in many different ways—trained models implemented / generated through this technique can be used to generate outputs (or predictions or decisions) based on inputs, and the model has "learned" how to do this from the training data / sample set. Since ML algorithms, their models, representations, implementations, and training are well-known, they will not be described in more detail in this paper except where necessary for understanding. More information about ML and ML models can be found at https: / / en.wikipedia.org / wiki / Machine_learning#Models, the full content of which is incorporated herein by reference.
[0003] Once deployed, a trained ML model can be vulnerable to unauthorized use or modification. For example, an entity may have licensed a trained ML model for certain specific uses, but that entity might then use it in an unauthorized manner (e.g., outside the boundaries of the permitted uses specified in the license). Similarly, an entity might acquire a trained ML model through unauthorized channels and thus potentially benefit from its use without incurring the time and cost of: acquiring training data; selecting / configuring / designing the architecture of the ML algorithm and model; and performing training and testing to obtain the resulting model. Therefore, it would be desirable to be able to verify the identity of the ML model, for example, enabling the initiator / generator of the trained ML model to check whether an entity might be using its trained ML model in an unauthorized manner, i.e., whether the “questionable” trained ML model being used by the entity is actually the trained ML model developed by the initiator / generator. In fact, it would be desirable to do this in a way that restricts access to “suspect” ML models in the sense that the internal weights / values / results of a “suspect” ML model may be inaccessible, but only the outputs of that “suspect” ML model are accessible, for example if the deployed “suspect” ML model operates behind an interface such as a webpage, and the only interaction with that “suspect” ML model is via that interface (e.g., by providing input and receiving the final result from that “suspect” ML model, without being able to access intermediate weights / values / results).
[0004] Similarly, once deployed, it is possible to modify the ML model. This modification might be unintentional, such as due to errors / failures in software, hardware, memory, etc. Alternatively, it could be malicious (e.g., an attacker intentionally attempting to alter the ML model to induce erroneous results). Such modifications can have serious consequences—for example, an ML model might have been trained to identify road signs for the purpose of guiding autonomous vehicles, and modifications to such a trained ML model could lead to incorrect vehicle handling, potentially causing an accident. Therefore, it will be desirable to be able to verify that the ML model used in a particular deployment is indeed the expected / desired ML model, or in other words, to be able to verify the integrity of the deployed ML model. Again, depending on the deployment scenario, this verification may or may not have access to the intermediate weights / values / results of the deployed ML model. Summary of the Invention
[0005] According to a first aspect of the invention, a method is provided for identifying whether a classification system is configured to use a specific machine learning classification model, the method comprising: generating a corresponding classification result for each test sample in the predetermined test set comprising a plurality of test samples using the classification system; and identifying any of the following: (i) if for each test sample in the test set, the corresponding classification result matches a classification result generated using the specific machine learning classification model for that test sample, then the classification system is using the specific machine learning classification model; or (ii) if there are test samples in the test set for which the corresponding classification result does not match a classification result generated using the specific machine learning classification model for that test sample, then the classification system is not using the specific machine learning classification model; wherein the test set is associated with the specific machine learning classification model, and for each test sample in the test set, there is a corresponding minor modification for that test sample, the minor modification resulting in a change in the classification result generated using the specific machine learning classification model for that test sample.
[0006] According to a second aspect of the invention, a method for generating a test set is provided for use in a method according to the first aspect for identifying whether a classification system is using a specific machine learning classification model, the test set being associated with the specific machine learning classification model, wherein the test set includes a plurality of test samples, and for each test sample in the test set, there exists a corresponding minor modification for that test sample, the minor modification resulting in a change in the classification result generated for that test sample using the specific machine learning classification model, wherein the method includes: obtaining a first set including a plurality of candidate samples applicable to the specific machine learning classification model; and updating the first set, the update including, for each candidate sample, performing one or more further modifications. The corresponding sequence of new steps, wherein each update step includes: generating a second candidate sample based on the candidate sample; generating a corresponding classification metric using the specific machine learning classification model for each of the candidate sample and the second candidate sample; and evaluating the generated classification metric, wherein the evaluation includes: performing one or more of the following steps based on a comparison of the generated classification metric: (a) terminating the sequence of one or more update steps; (b) setting the candidate sample as the second candidate sample if the comparison indicates that the second candidate sample is more likely than the candidate sample to have a corresponding minor modification that would lead to a change in the classification result generated using the specific machine learning classification model; wherein the test set includes some or all of the updated first set.
[0007] In some embodiments of the second aspect, generating a second candidate sample based on the candidate sample includes generating a second candidate sample by performing random changes on the candidate sample.
[0008] In some embodiments of the second aspect, for each candidate sample, for each update step of a first subsequence of a sequence corresponding to one or more update steps: for each of the candidate sample and the second candidate sample, the corresponding classification metric is a score generated by the following steps: generating a plurality of corresponding values using the specific machine learning classification model, each value indicating that the sample belongs to a corresponding category; and generating a score for the sample based on the plurality of corresponding values using a scoring function, the score indicating the possibility of a small modification for the sample, the small modification causing a change in the classification result generated for the sample using the specific machine learning classification model; and evaluating the generated classification metric includes: setting the candidate sample as the second candidate sample if the classification metric of the second candidate sample indicates a higher probability than the classification metric of the candidate sample. In some such embodiments, for each candidate sample, for each update step of a second subsequence of a corresponding sequence following one or more update steps after the first subsequence: for each of the candidate sample and the second candidate sample, the corresponding classification metric is an identifier of the class of the sample generated using the specific machine learning classification model; and evaluating the generated classification metric includes: if the classification metric of the second candidate sample is the same as the classification metric of the candidate sample, then terminating the second subsequence; if the classification metric of the second candidate sample is different from the classification metric of the candidate sample: for each of the candidate sample and the second candidate sample: using the specific machine learning classification model to generate a plurality of corresponding values, each value indicating that the sample belongs to a corresponding class; and using a scoring function to generate a score for the sample based on the plurality of corresponding values, the score indicating the possibility of a small modification for the sample, the small modification causing a change in the classification result generated for the sample using the specific machine learning classification model; and if the score of the second candidate sample indicates a higher probability than the score of the candidate sample, then setting the candidate sample as the second candidate sample.
[0009] In some embodiments of the second aspect, each value represents the probability that the corresponding sample belongs to the corresponding category. In some such embodiments, for each of the first and second candidate samples, the corresponding plurality of values are normalized to have a predetermined sum.
[0010] In some embodiments of the second aspect, for each candidate sample, for each update step of a first subsequence of a sequence corresponding to one or more update steps: for each of the candidate sample and the second candidate sample, the corresponding classification metric is an identifier of the class of the sample generated using the particular machine learning classification model; evaluating the generated classification metric includes: if the classification metric of the second candidate sample is the same as the classification metric of the candidate sample, then terminating the first subsequence if a termination condition is met; if the classification metric of the second candidate sample is not the same as the classification metric of the candidate sample: setting the candidate sample as the second candidate sample; and when the second candidate sample is generated at the next update step, reducing the size of the random changes to be applied to the candidate sample.
[0011] In some embodiments of the first or second aspect, one or more test samples from the test set are generated as adversarial examples for the particular machine learning classification.
[0012] According to a third aspect of the invention, an apparatus is provided that is arranged to perform the method according to the first or second aspect or any embodiment thereof.
[0013] According to a fourth aspect of the invention, a computer program is provided that, when executed by one or more processors, causes the one or more processors to perform the method according to the first or second aspect or any embodiment thereof. The computer program may be stored on a computer-readable medium. Attached Figure Description
[0014] Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings, in which: Figure 1 An example of a computer system is illustrated schematically; Figure 2a An example classification system is illustrated schematically; Figure 2b Examples of categories that can be performed are illustrated; Figure 3 A system according to some embodiments of the present invention is illustrated schematically; Figure 4 This is a flowchart illustrating a method for identifying whether a classification system is configured to use a specific ML classification model according to some embodiments of the present invention; Figure 5a The illustration shows the generation process according to some embodiments of the present invention. Figure 4 The flowchart shows the steps of the test set method in the flowchart; Figure 5b schematically illustrated Figure 5a The operation of the method shown; Figure 6 This is a flowchart illustrating a method for performing one or more update steps on a candidate sample according to some embodiments of the present invention; Figure 7 An example of a double-helix dataset is illustrated; and Figure 8 , 9 Figures 10 and 10 illustrate example experimental results using embodiments of the present invention. Detailed Implementation
[0015] Certain embodiments of the invention are described in the following description and accompanying drawings. However, it will be understood that the invention is not limited to the described embodiments, and some embodiments may not include all the features described below. It will be apparent, however, that various modifications and changes may be made herein without departing from the broader spirit and scope of the invention as set forth in the appended claims.
[0016] (a) System Overview Figure 1 An example of a computer system 100 is schematically illustrated. System 100 includes a computer 102. Computer 102 includes: storage medium 104, memory 106, processor 108, interface 110, user output interface 112, user input interface 114, and network interface 116, which can be linked together via one or more communication buses 118.
[0017] Storage medium 104 can be any form of non-volatile data storage device, such as one or more of hard disk drives, magnetic disks, solid-state storage devices, optical disks, ROMs, etc. Storage medium 104 can store an operating system for the processor 108 to execute in order to enable the computer 102 to function. Storage medium 104 can also store one or more computer programs (or software or instructions or code).
[0018] The memory 106 may be any random access memory (memory cell or volatile storage medium) suitable for storing data and / or computer programs (or software or instructions or code).
[0019] Processor 108 may be any data processing unit adapted to execute one or more computer programs (such as those stored on storage medium 104 and / or memory 106), some of which may be computer programs according to embodiments of the invention, or computer programs that, when executed by processor 108, cause processor 108 to perform methods according to embodiments of the invention and configure system 100 as a system according to embodiments of the invention. Processor 108 may include a single data processing unit, or multiple data processing units that operate in parallel, individually, or cooperatively with each other. In performing the data processing operations of embodiments of the invention, processor 108 may store data to storage medium 104 and / or memory 106, and / or read data from storage medium 104 and / or memory 106.
[0020] Interface 110 can be any unit that provides an interface to device 122, which is external to or removable from computer 102. Device 122 can be one or more data storage devices, such as optical discs, hard disks, solid-state storage devices, etc. Device 122 can have processing capabilities—for example, device 122 can be a smart card. Therefore, interface 110 can access data from device 122, provide data to device 122, or interface with device 122 based on one or more commands it receives from processor 108.
[0021] User input interface 114 is configured to receive input from a user or operator of system 100. The user provides this input via one or more input devices (such as a mouse (or other pointing device) 126 and / or keyboard 124) connected to or communicating with user input interface 114. However, it will be appreciated that the user may provide input to computer 102 via one or more additional or alternative input devices (such as a touchscreen). Computer 102 may store the input received from the input device via user input interface 114 in memory 106 for subsequent access and processing by processor 108, or may pass it directly to processor 108 so that processor 108 can respond accordingly to the user input.
[0022] User output interface 112 is arranged to provide graphical / visual and / or auditory output to the user or operator of system 100. Thus, processor 108 may be arranged to instruct user output interface 112 to generate an image / video signal representing the desired graphical output and to provide such signal to a monitor (or screen or display unit) 120 of system 100 connected to user output interface 112. Additionally or alternatively, processor 108 may be arranged to instruct user output interface 112 to generate an audio signal representing the desired audio output and to provide such signal to one or more speakers 121 of system 100 connected to user output interface 112.
[0023] Finally, network interface 116 provides functionality for enabling computer 102 to download data from one or more data communication networks and / or upload data to one or more data communication networks.
[0024] What will be understood is that, Figure 1 The architecture of the system 100 illustrated and described above is merely exemplary, and different architectures (e.g., having a higher degree of integration) can be used in embodiments of the invention. Figure 1 The components shown are fewer, or with Figure 1 The components shown are compared to other computer systems 100 that have additional and / or alternative components. As an example, computer system 100 may include one or more of the following: personal computer; server computer; mobile phone; tablet device; laptop; television; set-top box; game console; other mobile device or consumer electronics device; and so on. Additionally, some components of computer system 100 may not be located in computer 102, but instead are part of a computer network connected to computer 102 via network interface 116. Additionally or alternatively, computer system 100 may include multiple computers 102, for example, in a network of computers such as a cloud system containing computing resources.
[0025] Figure 2a An example classification system 200 is schematically illustrated. Classification system 200 can, for example, use a system like the one described in the reference. Figure 1 The discussion is implemented by one or more computer systems 100.
[0026] The classification system 200 includes an input (or interface) 202 for receiving (or obtaining) input samples 208 (or a certain amount of input data). The classification system 200 uses or executes an ML classification model (or a trained ML classification algorithm) 204 to perform classification on the input samples 208. For example, the ML classification model 204 may be arranged to identify whether the input samples 208 correspond to multiple (or a set of) data. N C A known category or subcategory {C k : k= 1,…, N C A category of} C j Alternatively, the ML classification model 204 can be arranged for multiple (or a group) classification models. N C Known categories { C k : k= 1,…, N C Each category in} C j The corresponding weight or value indicates that input sample 208 belongs to that category. C j The probability or likelihood of classification. The ML classification system 200 includes an output (or interface) 206 for providing results (or outputs) 210 for classification performed on input sample 208 using the ML classification model 204. For example, results 210 may include the category to which input sample 208 belongs or corresponds according to the ML classification model 204. C j This indication can accompany the confirmation that input sample 208 indeed belongs to that category. C j An indication of confidence level or probability.
[0027] Of course, classification system 200 can be a component (or subsystem) of a larger system—the larger system can be configured to provide input samples 208 to classification system 200 and use the results 210 generated by classification system 200. For example, the larger system could be an engine; input samples 208 could include measurements from sensors around / inside the engine; classification system 200 could be configured to use its ML classification model 204 to classify input samples 208 as either a normal state or a fault state of the engine; if the results 210 provided to the engine by classification 200 indicate that input sample 208 corresponds to a fault state, the engine could be configured to shut down or issue an alarm.
[0028] Input interface 202 may be the same as or different from output interface 206. Input interface 202 and / or output interface 206 may, for example, include one or more web pages or user interfaces. Additionally or alternatively, input interface 202 and / or output interface 206 may be simply arranged to access input sample 208 from the storage device or medium of classification system 200 or to store result 210 to the storage device or medium of classification system 200. It will be understood that, depending on the specific deployment scenario of classification system 200, input interface 202 and output interface 206 may take other forms.
[0029] Of course, the properties and categories of input sample 208 { C k : k= 1,…, N C The classification system 200 can vary widely depending on its application / use. For example, the classification system 200 can be part of an autonomous vehicle—input samples 208 can include one or more images captured by one or more cameras of the vehicle, and an ML classification model 204 can be arranged to classify the input samples 208 to obtain results 210 used to help control the driving of the vehicle. Then, the categories { C k : k= 1,…, N C} can correspond to the following: an image containing or depicting a road sign; an image containing or depicting an obstacle (e.g., another vehicle or pedestrian); and so on. Similarly, classification system 200 can be arranged to identify the language being spoken in input audio sample 208 (e.g., audio captured by a microphone), with the aim of obtaining a result 210 identifying one or more possible languages being spoken within that input audio sample 208. Then, the category { C k : k= 1,…, N C} can correspond to different languages being spoken. As another example, classification system 200 can be arranged to identify whether an email input sample 208 is spam. Then, the categories { C k : k= 1,…, N C} can correspond to spam and non-spam emails.
[0030] Figure 2b An example of the classifications that can be performed is illustrated. In this example, N C=4, so that there are four categories: C 1. C 2. C 3. C 4. Of course, what will be understood is that... N C Other values can be used. Additionally, in this example, the input space or sample space 250 (i.e., the set of all possible / valid input samples 208 suitable for input to the classification model 204) is represented as a two-dimensional sample space 250, where the input samples 208 are points in this two-dimensional sample space 250. Again, it will be understood that this is merely an example for illustrative purposes, and the sample space 250 and input samples 208 can take many different representations. The ML classification model 204 results in, corresponds to, or implements the partitioning of the sample space 250 into categories corresponding to various classes { C k : k= 1,…, N C Different regions of} – for a given category C j The regions can be connected, or alternatively, can be formed by multiple unconnected smaller regions. Therefore, for a given input sample 208, the classification result 210 performed by the ML classification model 204 can simply be the category corresponding to the region / partition of the sample space 250 where the input sample 208 resides. C j The indication. As mentioned, result 210 may also include whether input sample 208 belongs to that category. C j An indication of confidence level or probability.
[0031] Since classification systems are well-known, they will not be described in more detail here. Further information on ML classification can be found at, for example, https: / / en.wikipedia.org / wiki / Statistical_classification, the full content of which is incorporated herein by reference.
[0032] Modifications to ML classification model 204 will change the categories { C k : k= 1,…, N C At least some boundaries between samples 208 in the sample space 250. In other words, modifications to the ML classification model 204 will change the result 210 of at least some samples 208 in the sample space 250. Similarly, different ML classification models, even if they use the same category { C k :k= 1,…, N C This will also result in different boundaries between those categories in the sample space 250, and therefore, different results 210 will be produced for at least some samples 208 in the sample space 250. In fact, two ML classification models trained on the same training data using the same underlying ML algorithm may end up being different from each other, thus producing different results 210 for one or more input samples 208.
[0033] As can be seen, some input samples 208 (e.g., by...) Figure 2b Point 252 (represented by points 254a, 254b, 254c) is relatively far from the boundary of the region / partition corresponding to its class. Therefore, modifications to the ML classification model 204 (or different implementations of the ML classification model 204) are unlikely to change the classification result 210 of those samples 252. However, some input samples 208 (e.g., represented by points 254a, 254b, 254c) are relatively close to the boundary of the region / partition corresponding to their class. Therefore, modifications to the ML classification model 204 (or different implementations of the ML classification model 204) are likely to change the classification result 210 of one or more of these input samples 254, or at least are significantly more likely to change the classification result 210 of the input samples 254 compared to the case for input samples 252. For example, a small modification to the ML classification model 204 (or different implementations of the ML classification model 204) can: (a) move C 1 and C The boundary between 4 and 5 ensures that input sample 254a is classified into category 4. C 1, not C 4; (b) move C 2 and C 1 and / or C The boundary between 4 ensures that input sample 254b is classified into category C 1 or C 4, not C 2; (C) move C 2 and C 3 and / or C The boundary between 4 and 5 ensures that the input sample 254c is classified into category 4. C 3 or C 4, not C2. Therefore, as will be understood, some input samples 208 available from sample space 250 may be more “sensitive” to changes (even small changes) to the ML classification model 204 than other input samples 208, because some input samples 208 are more likely to produce changes in their corresponding classification results 210 due to changes (even small changes) to the ML classification model 204 compared to other input samples 208. A “sensitive” sample can be considered as one for which, according to the ML classification model 204, the evidence that the sample belongs to only one class is low. In other words, a “sensitive” sample can be considered as one for which a small modification or perturbation will result in a change in the classification result of the sample—that is, they are close to at least one of the class boundaries. In other words, given a set of different ML classification models for sample space 250, “sensitive” sample 254 is statistically more likely than sample 252 to produce different classification results 210 for at least two models.
[0034] One way to identify whether a “suspect” ML classification model corresponds to a specific (or target) ML classification model is to obtain a first classification result 210 using the “suspect” ML classification model and a second classification result 210 using the specific ML classification model for each of the plurality of test samples 208. If the corresponding first and second classification results are different for any of the test samples 208, then the “suspect” ML classification model does not match the specific ML classification model. On the other hand, if the corresponding first and second classification results are the same for each of the plurality of test samples 208, then it can be concluded / inferred that the “suspect” ML classification model matches the specific ML classification model—however, this conclusion cannot be guaranteed without testing all possible input samples 208. As discussed above, and as from Figure 2bAs can be seen, if a “suspicious” ML classification model does not match a specific ML classification model, some input samples 204 (e.g., point 254) are more likely to provide different classification results than other input samples 204 (e.g., point 252). Therefore, by using such “sensitive” input samples 254 (i.e., input samples 254 close to the classification boundary) instead of other “less sensitive” input samples 252 (i.e., input samples 252 less close to the classification boundary), it is more likely to detect when a “suspicious” ML classification model does not match a specific ML classification model, and similarly, if all test samples have matching classification results, it is more certain that the “suspicious” ML classification model does indeed match the specific ML classification model. This is true even if the two ML classification models differ only to a minor degree. Therefore, embodiments of the invention operate based on this principle by identifying and using a test set created for a specific ML classification model, wherein the test samples in the test set are specifically selected to be sensitive to changes in the ML classification model (or at least more sensitive than most other possible input samples 208).
[0035] Figure 3 A system 300 according to some embodiments of the present invention is schematically illustrated. System 300 includes a test set creation system 320, a testing system 330, and a classification system 310. The testing system 330 can interact with the classification system 310 in various ways, such as through communication over a network 350.
[0036] Classification system 310 can be referenced above. Figure 2a and 2b The classification system 200 under discussion. As shown, classification system 310 utilizes ML classification model 312 (corresponding to the above reference). Figure 2a and 2b The ML classification model discussed (204).
[0037] In summary, the testing system 330 is configured to attempt to identify whether the ML classification model 312 being used by the classification system 310 is a specific ML classification model 322. As discussed above, this can be achieved by the testing system 330 supplying multiple or a set of test samples to the classification system 310 and obtaining corresponding first classification results from the classification system 310 based on the classification model 312 used by the classification system 310. The testing system 330 can, for example, provide test samples to the classification system 310 and receive corresponding first classification results back from the classification system 310 via network 350. The testing system 330 can compare these first classification results with second classification results for the set of test samples, wherein the second classification results are generated using or have been generated using the specific ML classification model 322. Based on this comparison, the testing system 330 can determine whether the ML classification model 312 being used by the classification system 310 is a specific ML classification model 322. The purpose of the test set creation system 320 is based on the above references. Figure 2b The principle of "sensitive samples" discussed is used to generate a test sample set corresponding to a specific ML classification model 322.
[0038] Test suite creation system 320 and test system 330 can use the reference Figure 1 The discussion is implemented using one or more computer systems 100. In fact, the test set creation system 320 and the test system 330 can be implemented on the same computer system 100 and / or operated by the same entity, and therefore can be considered as a composite system, such as... Figure 3 The test set creation system 320 and the test system 330 are illustrated by the dashed line 340. However, it will be understood that the test set creation system 320 and the test system 330 may be implemented on different computer systems 100 and / or may be operated by different entities, and therefore may be considered as separate systems.
[0039] The test set creation system 320 includes a test set creation module 326. The test set creation module 326 can be implemented as software executing on a processor 108 of a computer system 100 that is, for example, part of (or forming) the test set creation system 320, or as a hardware module of the test set creation system 320, or as a combination of both. In operation, the test set creation module 326 is arranged to generate a test set 324 comprising multiple test samples using a specific ML classification model 322—therefore, the test set 324 is associated with, corresponds to, or has been generated for the specific ML classification model 322. Of course, the test samples of the test set 324 are samples used as valid input to the specific ML classification model 322 (or, using...). Figure 2bThe following analogy is from samples 250 in the sample space.
[0040] The test set creation system 320 may already possess a specific ML classification model 322—for example, the test set creation system 320 may have already generated (or created) a specific ML classification model 322, or may have previously received / obtained a specific ML classification model 322. Alternatively, the test set creation system 330 may receive / obtain a specific ML classification model 322 from a third party (e.g., via network 302).
[0041] Test system 330 includes test module 336. Test module 336 may be implemented as software executing on processor 108 of computer system 100, which is part of (or forms part of) test system 330, or as a hardware module of test system 330, or as a combination of both. In operation, test module 336 is arranged to use test set 324 (generated by test set creation system 320) associated with a specific ML classification model 322 to determine whether ML classification model 312 used by classification system 310 is the specific ML classification model 322 associated with test set 324.
[0042] In some embodiments, a specific ML classification model 322 is provided in advance to the test set creation system 320 (e.g., as part of a registration process), allowing the test set creation system 320 to pre-generate a test set 324, thus preparing it for when needed. Therefore, the test set 324 can be stored in a storage device ( Figure 3 (Not shown in the image) thus preparing it for when the testing system 330 needs it. In other embodiments, while the classification system 310 and its ML classification model 312 are being investigated, a specific ML classification model 322 is provided to the test set creation system 320. In this embodiment, the test set creation system 320 may generate a test set 324 for subsequent use by the testing system 330—in which case the test set 324 may or may not be stored in a storage device.
[0043] As discussed above, the test set creation system 320 uses a specific ML classification model 322 to generate a test set 324. The test set creation system 320 can also use the specific ML classification model 322 to generate classification results (the second classification results described above) for the test samples in the test set 324. These classification results can then be stored in association with their corresponding test samples (e.g., stored in a storage device). Figure 3(not shown in the image) (e.g., as part of a test set or as metadata associated with a test set or with test samples). In this embodiment, the test system 330 does not need to utilize a specific ML classification model 322 because the test system 330 has both the test samples of the test set 324 and the classification results already generated by the test set creation system 320 for those test samples using the specific ML classification model 322. However, in other embodiments, the test set creation system 320 does not use the specific ML classification model 322 to generate classification results—instead, the test system 330 has access to and utilizes the specific ML classification model 322 to generate classification results for the test samples of the test set 324.
[0044] In some embodiments, the test set creation system 320 and / or the testing system 330 may have access to the "raw" data output / generated by a specific ML classification model 322, such as intermediate results / values generated by the specific ML classification model 322, or weights used / generated by the specific ML classification model 322. In other embodiments, the test set creation system 320 and / or the testing system 330 do not have access to such "raw" data output / generated by the specific ML classification model 322; that is, the test set creation system 320 and / or the testing system 330 may only have access to the final classification result 210 generated using the specific ML classification model 322 as output. For example, a first entity that created / generated a specific ML classification model 322 can contact a second entity that is operating on system 340 to determine whether the ML classification model 312 being used by classification system 310 is their specific ML classification model 322. However, the first entity may not want to provide the second entity with full access to / details of their specific ML classification model 322. Instead, it may provide the second entity with only access to the input interface 202 and output interface 206 of classification system 210 that uses their specific ML classification model 322, so that the second entity can use the specific ML classification model 322 to generate classification results 210 for test sample 208.
[0045] Network 350 can be any data communication network through which test system 330 can communicate with classification system 310. For example, network 350 may include one or more of the following: local area network, wide area network, metropolitan area network, Internet, wireless communication network, wired or cable communication network, etc.
[0046] Figure 4The flowchart illustrates a method 400 according to some embodiments of the present invention, which is used to identify whether a classification system 310 is configured to use a specific ML classification model 322, i.e., to identify whether an ML classification model 312 is or corresponds to a specific ML classification model 322 (this may be for purposes such as checking the integrity of the ML classification model 312, or to determine whether the classification system 310 is using the specific ML classification model 322 without authorization).
[0047] At step 402, the test set creation system 320 (or its test set creation module 326) generates a test set 324. As discussed above, the test set 324 is associated with or corresponds to a specific ML classification model 322—therefore, once generated, the test set 324 is intended for use with the specific ML classification model 322. The test set 324 includes multiple test samples. The test samples are "sensitive" samples because for each test sample in the test set 324, there exists a corresponding small modification that results in a change in the classification result produced using the specific ML classification model 322 for that test sample. The method for generating the test set 324 will be briefly described. Let the test set 324 contain... N S test samples (where a certain integer) N S >1), where the test sample is T j ,,in j =1…, N S .
[0048] At step 404, the testing system 330 (or its testing module 336) uses the classification system 310 to generate a corresponding first classification result for each test sample in the test set 324. Therefore, the first classification result is the classification result generated using the ML classification model 312 for each test sample in the test set 324. Thus, for each test sample... T j (in j =1,…, N S The first classification result is generated based on the ML classification model 312. R 1,j ,in j =1,…, N S .
[0049] At step 406, for each test sample in test set 324, a corresponding second classification result is generated or produced using a specific ML classification model 322 for that test sample. As discussed above, step 404 can be performed by test set creation system 320 (e.g., its test set creation module 326) or by testing system 330 (e.g., its testing module 336). Therefore, for each test sample... T j (in j =1,…, N S ), Generate the corresponding second classification result based on the specific ML classification model 322 R 2,j (in j =1,…, N S ).
[0050] At step 408, the test system 330 (or its test module 336) determines whether the first and second classification results match the test sample.
[0051] If at step 408 it is determined that the first classification result matches the second classification result for each test sample in test set 324 (i.e., ... R 1,j = R 2,j ,in j= 1,…, N S If the test system 330 identifies and thus generates a conclusion / judgment 410: the classification system 310 is using a specific ML classification model 322 (i.e., the ML classification model 312 matches or is equal to the specific ML classification model 322). Appropriate measures can then be taken as needed, for example: (a) if the purpose of the inspection / investigation of the classification system 310 is to check the integrity of the ML classification model 312, it can be concluded that the integrity of the ML classification model 312 has not been compromised and its logs can be made accordingly; (b) if the purpose of the inspection / investigation of the classification system 310 is to check whether the classification system 310 is using the specific ML classification model 322 in an unauthorized manner, measures can be taken to attempt to stop the operation of the classification system 310 or at least stop the use of the specific ML classification model 322.
[0052] On the other hand, if at step 408 it is determined that there exists a test sample in test set 324 for which the first classification result does not match the second classification result (i.e., there exists a certain...) k ∊{1,…, N S}, for it, R 1,k ≠R 2,k If the test system 330 identifies and thus generates conclusion / judgment 412: the classification system 310 is not using the specific ML classification model 322 (i.e., the ML classification model 312 does not match or is not equal to the specific ML classification model 322). Appropriate measures can then be taken as needed, for example: (a) if the purpose of the inspection / investigation of the classification system 310 is to check the integrity of the ML classification model 312, it can be concluded that the integrity of the ML classification model 312 has been compromised, and appropriate actions can be taken, such as preventing the classification system 310 from using the compromised model, issuing an alert, etc.; (b) if the purpose of the inspection / investigation of the classification system 310 is to check whether the classification system 310 is using the specific ML classification model 322 in an unauthorized manner, then no measures are required at this stage.
[0053] It will be understood that the test set creation system 320 can generate a test set 324 with more test samples than the test set 330 actually needs to use in steps 406 and 408. In this case, the test set 330 can use a subset of the test set 324 (although this can still be considered as the test set creation system 320 having generated the subset).
[0054] Method 400 can be used without a specially generated test set 324, instead using a separate test set with “regular” test samples that have not yet been generated to be “sensitive” (e.g., simply random samples from sample space 250). However, by using a specially generated test set 324 with “sensitive” samples, embodiments of the invention enable more accurate testing, providing more confident test results with fewer test samples. Example experimental results for this will be described later.
[0055] (b) Test set generation Figure 5a The illustration shows the generation process according to some embodiments of the present invention. Figure 4 The flowchart of method 500 for test set 324 at step 402. Figure 5b schematically illustrated Figure 5a The operation of method 500 shown here—therefore, these two diagrams will be described together.
[0056] Method 500 begins with step 502, where the test set creation system 320 obtains a first set 510 comprising multiple candidate samples. Figure 5b As shown, let the first set 510 contain N Bcandidate samples (where some integer) N B >1), where the candidate sample is B j ,in j =1,…, N B Each candidate sample B j ( j =1,…, N B This applies to a specific ML classification model 322 (or is a valid input sample for a specific ML classification model 322), that is, each candidate sample is a sample that the specific ML classification model 322 intends to operate / process, or, through... Figure 2b The following analogy is from the sample space 250 used for a specific ML classification model 322.
[0057] N B The value can be predetermined. For example, N B It could be on the order of approximately 10 to 20, although of course, N B The higher the value, the more accurate the analysis and conclusions of Method 400 will be. Example experimental results for this will be presented later to illustrate how... N B Improvements that can be achieved through changes.
[0058] Initial candidate samples B j ( j =1,…, N B ) can be randomly generated samples from the sample space 250 used for a specific ML classification model 322—for example, reference Figure 2b Example, candidate samples B j ( j =1,…, N B () can be a randomly identified point in the sample space of 250. Alternatively, candidate samples B j ( j =1,…, N B The candidate samples can be selected to be uniformly distributed within the sample space 250 used for a specific ML classification model 322. Alternatively, the candidate samples... B j ( j =1,…, N BThe candidate samples can be samples from the training or testing sample set used to train or test a specific ML classification model 322 (which may be randomly selected / picked). It will be understood that candidate samples can be selected using... B j ( j =1,…, N B Other methods of the first set 510.
[0059] At step 504, the test set creation system 320 updates the first set 510. This involves: for each candidate sample B j ( j =1,…, N B ), the corresponding sequence 520 for performing one or more update steps. j The process of implementing a sequence of one or more update steps for each candidate sample. B j ( j =1,…, N B The sequence of one or more update steps can be the same, but the actual sequence of one or more update steps may vary from one candidate sample to another (e.g., if for candidate samples...). B j ( j =1,…, N B Sequence 520 of one or more update steps j Depends on candidate samples B j The assumed (one or more) values). However, in some embodiments, the process of implementing a sequence of one or more update steps for candidate samples. B j ( j =1,…, N B Two or more of them can be different. (Refer to...) Figure 6 Briefly describe the sequence of one or more update steps 520 j The properties of.
[0060] like Figure 5b As shown, for candidate samples B j ( j =1,…, N B Sequence 520, which executes one or more update steps. j One or more corresponding updated candidate samples were obtained. B *j Therefore, step 504 generates updated candidate samples from the first set 510. B * j ( j =1,…, N * B The second set of 530 contains integers. N * B ≥ N B Therefore, the second set 530 is an updated version of the first set 510. In some embodiments, for candidate samples... B j ( j =1,…, N B Sequence 520, which executes one or more update steps. j Only one corresponding updated candidate sample is obtained. B * j (make) N * B = N B However, this does not always need to be the case. Test set 324 includes some or all of the updated first set 530. In some embodiments, test set 324 includes all updated candidate samples. B * j ( j =1,…, N * B In some embodiments, test set 324 includes updated candidate samples. B * j ( j =1,…, N * B A subset of ) (selected based on one or more selection criteria).
[0061] For candidate samples B j ( j =1,…, N B Sequence 520 of one or more update steps j The purpose is to refine, or attempt to optimize or improve, the candidate samples. B j In order to obtain a ratio of 324 candidate samples for the test set.B j More "sensitive" (or at least based on available evidence, appearing more sensitive than candidate samples) B j One or more test samples that are more "sensitive".
[0062] For candidate samples B j ( j =1,…, N B Sequence 520 of one or more update steps j This can be implemented in various ways, examples of which are illustrated below. As mentioned above, in some embodiments, the test set creation system 320 and / or the testing system 330 may have access to the "raw" data output / generated by a specific ML classification model 322, such as intermediate results / values generated by the specific ML classification model 322, or weights used / generated by the specific ML classification model 322. This "raw" data can be used as a sequence 520 of one or more update steps. j This is part of the process. In this embodiment, any statistical method capable of optimizing the objective function (such as particle swarm optimization, genetic algorithms, simulated annealing, stochastic gradient descent, etc.) can be used. However, in other embodiments, the test set creation system 320 and / or the test system 330 do not have access to this "raw" data output / generated by the specific ML classification model 322; that is, the test set creation system 320 and / or the test system 330 only have access to the final classification result 210 generated using the specific ML classification model 322 as output. In this embodiment, any search / refinement technique (such as binary search) can be used. Specific examples will be described later.
[0063] Figure 6 The illustration shows candidate samples according to some embodiments of the present invention. B j ( j =1,…, N B Sequence 502 executes one or more update steps. j The flowchart illustrates the general method. Each update step includes steps 602, 604b, 606, and optionally step 604a. Each update step processes candidate samples. P j And may (or may not) eventually update the candidate sample. P j For candidate samples B j ( j =1,…, N BSequence 502 of one or more update steps j The first update step in the process is to process candidate samples. P j Set as a candidate sample B j Sequence 502 consists of one or more update steps. j The obtained (one or more) updated candidate samples B * j Including (potentially updated) candidate samples from the last update step of the sequence. P j (And possibly, additional samples generated as part of the process).
[0064] At step 602, the test set creation system 320 is based on the current candidate samples. P j To generate a second candidate. Second candidate sample Q j It is a sample that a specific ML classification model 322 intends to operate on / process (i.e., a valid sample for a specific ML classification model 322), or, through Figure 2b The following analogy is from the sample space 250 used for a specific ML classification model 322.
[0065] In some embodiments, the test set creation system 320 performs tests on current candidate samples. P j Perform random changes to generate a second candidate sample Q j This random change can, for example, be made by adjusting the current candidate sample. P j This is achieved by randomly modifying or perturbing one or more components—for example, if the current candidate sample P j If a sample is a vector or set of values, then one or more of these values can be randomly perturbed. Of course, it will be understood that such random perturbation can be implemented in a variety of ways, depending on the nature of the samples themselves. In some embodiments, the magnitude of the random perturbation can be at most a predetermined threshold; for example, one or more thresholds can be placed with respect to this random perturbation to affect the current candidate samples in the sample space 250. P j Compared with the second candidate sample Q j The distance between them is subject to restrictions.
[0066] However, as will be discussed later, in some embodiments, the test set creation system 320 is based on the current candidate samples. Pj Together with some other information, a second candidate sample is generated. Q j (For example, another point in the sample space 250, for instance, as the current candidate sample) P j (The midpoint between this and the other point). Therefore, based on the current candidate sample P j To generate a second candidate sample Q j It may be more certain.
[0067] At step 604b, the test set creation system 320 uses a specific ML classification model 322 for the second candidate sample. Q j Generate the corresponding classification metric. In some embodiments, the classification metric can simply be derived from multiple... N C Known categories { C k : k= 1,…, N C Specific categories in} C j As instructed by the specific ML classification model 322, the second candidate sample Q j Belongs to this specific category C j Alternatively, in some embodiments, the test set creation system 320 may be arranged to: target the plurality of N C Known categories { C k : k= 1,…, N C Each category in} C j The corresponding weights or values are obtained from the ML classification model 322, and these weights or values represent the second candidate samples. Q j Belongs to this category C j The likelihood (or probability or confidence level) – a classification metric – can then be a score generated based on a function (or measure) of those weights or values.
[0068] In addition to step 604a, which includes the test set creation system 320 using a specific ML classification model 322 to target the current candidate samples P j Apart from generating the corresponding classification metric, steps 604a and 604b are the same. This is in accordance with the method used for the second candidate sample. Qj It is executed in the same way. Now, as mentioned above, some update steps may result in updating the candidate sample. P j Therefore, in the subsequent update step, it is necessary to generate the candidate sample that is now being updated. P j The classification metric is used (and therefore, step 604a is performed in the subsequent update step). However, some update steps may not result in updating the candidate sample. P j Therefore, in subsequent update steps, candidate samples P j The classification metric is known and does not need to be regenerated (and therefore, step 604a can be omitted from the subsequent update step).
[0069] At step 606, the test set creation system 320 evaluates the current candidate samples. P j The generated classification metric and the second candidate sample Q j The generated classification metric. This evaluation includes: • At substep 608, a comparison is made for the current candidate sample. P j The generated classification metric and the second candidate sample Q j The generated classification metric; Based on this comparison, perform one or more of the following steps: (a) at sub-step 610, terminate the sequence of one or more update steps; and (b) at sub-step 612, if the comparison indicates a second candidate sample Q j Compared to the current candidate samples P j It is more likely to have a corresponding small modification that leads to a change in the classification results produced using a specific ML classification model 322 (i.e., if the second candidate sample Q j Compared to the current candidate samples P j Potentially more "sensitive"), then the current candidate sample P j Set as the second candidate sample Q j .
[0070] If at least one termination condition (or criterion) is met, step 610 terminates the process for the candidate sample. B j ( j =1,…,N B Sequence 502 of one or more update steps j In some embodiments, at least one termination condition may be based on the current candidate sample. P j The generated classification metric and the second candidate sample Q j Comparison of generated classification metrics, for example, if for the current candidate sample. P j The generated classification metric and the second candidate sample Q j The difference between the generated classification metrics is less than a predetermined threshold (or has already been less than the predetermined threshold within a predetermined number of update steps) – that is, if the test set creation system 320 evaluation is unlikely to have a difference compared to the current candidate samples. P j Further substantial improvements could terminate sequence 502 of one or more update steps. j Additionally or alternatively, in some embodiments, at least one of the termination conditions is: if the current update step is for some non-negative integer. V The first in the sequence V If an update step is performed, the process for updating candidate samples will terminate. B j ( j =1,…, N B Sequence 502 of one or more update steps j —That is, for candidate samples B j ( j =1,…, N B Sequence 502 of one or more update steps j It can be limited to a maximum V An update step is required to ensure that the process of generating test set 324 is terminated (or at least terminated for a certain period of time).
[0071] If the termination condition is not met at step 610, processing continues to the next update step, where the candidate sample to be processed will either be the current (non-updated) candidate sample from the current update step. P j (That is, if step 612 does not...) P j Set to Yes Q j ), or will be the second candidate sample Q j (That is, if step 612 does indeed...) Pj Set to Yes Q j ).
[0072] Below is how to perform the process for candidate samples. B j ( j =1,…, N B Sequence 502 of one or more update steps j Some examples. However, it will be understood that other implementations / methods are also possible.
[0073] Example 1 In this example, at step 602, by analyzing the current candidate samples P j Implement (possibly random) changes to generate a second candidate sample. Q j .
[0074] In this example, a specific ML classification model 322 targets multiple N C Known categories { C k : k= 1,…, N C Each category in} C j Generate the corresponding values (or weights). p j This value p j This indicates that the given input sample 208 belongs to this category. C j The probability (or likelihood or confidence level) of the value. p j This indicates that the sample belongs to the corresponding category. C j In this example, it is assumed that the test set creation system 320 has access to these values. p k ( k =1,…, N C This allows the test set creation system 320 to use a specific ML classification model 322 to generate multiple values for any given input sample 208. p k ( k =1,…, N C ).
[0075] Without loss of generality, we assume these values in the following text.p k ( k =1,…, N C ) is normalized to have a predetermined sum, i.e. It is for a certain predetermined constant W Preferably, W =1, which will be assumed in the following text, although it will be understood that this is merely an example.
[0076] Therefore, at step 604a, for the current candidate sample P j Generating classification metrics may involve a test set creation system 320 using a specific ML classification model 322 for the current candidate samples. P j Generate multiple values p k ( k =1,…, N C ), and use a scoring function to base it on those multiple values. p k ( k =1,…, N C The score (which will be the classification metric) is generated for the sample. The scoring function is arranged such that the score indicates the likelihood of a small modification to the sample that would result in a change in the classification outcome produced for the sample using a specific ML classification model. Therefore, the scoring function is arranged such that the score indicates how "sensitive" the sample might be.
[0077] Step 604b can then use the same scoring function to generate a second candidate sample in the same manner as in step 604a. Q j The score.
[0078] As part of evaluating the classification metric at step 606, the test set creation system 320 can then compare the scores generated at steps 604a and 604b, and at step 612, if the second candidate sample... Q j The classification metric indicates the difference between the current candidate samples. P j If the classification metric is higher, then step 612 will select the current candidate sample. P j Set as the second candidate sample Q j .
[0079] For example, suppose there are two categories, namely NC =2, then the scoring function can be score =1−| p 1− p 2|. Therefore, if the sample belongs to category C The probability of 1 and the sample belonging to the category C If the probability of two samples being similar, then the sample will have a higher score; if the sample belongs to category 2... C The probability of 1 and the sample belonging to the category C If the probability of 2 is significantly different, then the sample will score low. This can be viewed from another perspective: if the sample is close to the class... C 1 and category C The boundary between 2 (making the sample "sensitive"), then p 1 and p A value of 2 might indicate similarity, resulting in a high score; however, if the sample is far from the category... C 1 and category C The boundary between 2 (making the sample "insensitive"), then p 1 and p The value of 2 can be very different, resulting in a low score.
[0080] For example: • If for the first sample 208, p 1 = 0.99 and p If 2 = 0.01, then the score of the first sample is score =1−|0.99−0.01|=0.02, which is low. The first sample 208 is not very "sensitive"; • If for the second sample 208, p 1 = 0.60 and p If 2 = 0.40, then the score of the second sample is score =1−|0.60−0.40|=0.80, which is quite high. The second sample 208 is quite "sensitive" and tends to be a better test sample for test set 234 compared to the first sample 208; • If for the third sample 208, p 1 = 0.51 and p If 2 = 0.49, then the score of the third sample is score =1−|0.51−0.49|=0.98, which is very high. The third sample 208 is very "sensitive" and tends to be a better test sample for the test set 234 compared to the first or second sample 208.
[0081] What will be understood is that other scoring functions can be used. For example, in NC When =2, the scoring function can be score =1−( p 1− p 2) 2 .
[0082] As a further example, for values with normalized values p k ( k =1,…, N C The sample of ) makes ,make μ It is a normalized value p k ( k =1,…, N C The mean of ), i.e. Therefore, the scoring function used could be... .
[0083] As a further example, consider values with normalized values. p k ( k =1,…, N C The sample of ) makes .make And make p i and p j yes P The two largest elements (such that the two most likely classes of the sample are) C i and C j Then the scoring function used can be score =1−| p i - p j |or score =1−( p i - p j ) 2 One advantage of this scoring function is that, generally, fewer iterations / updates are needed to generate the test set of 324.
[0084] As a further example, given the existence N C Known categories { C k : k= 1,…, NC The scoring function can be based on only a subset of values for these categories. p k For example, there exists An unordered pair of categories—for example, if N C =3, then there are 3 unordered class pairs, that is ( C 1, C 2), ( C 1, C 3) and ( C 2, C 3). For a pair of such categories ( C i , C j Any of the scoring functions in the examples above can be based solely on the values for these two categories. p i and p j This is used for the sample (to help update the sample towards the boundary between the two classes). This can also be done for some or all of other class pairs, for the sample. Additionally, this, like obtaining updated candidate samples, can use the pair values. p k ( k =1,…, N C This is accomplished by using a scoring function that operates on more than two values.
[0085] In some embodiments, the scoring function is arranged to provide a higher score for a sample 208 having one or more of the following characteristics: (a) a high but nearly equal probability of belonging to two or more categories; (b) a low probability of belonging to all categories; and (c) a low but non-zero probability of belonging to a subset of (correct) categories.
[0086] Therefore, in this example, the method used to generate test set 324 could involve: (1) Step 502: Obtain candidate samples { B j : j =1,…, N B The first set of} is as discussed above; (2) By targeting each candidate sample B j ( j =1,…, N B The following steps are performed to update each candidate sample: (a) Initially,P j = B j , (b) Step 604a: Use a specific ML classification model 322 to target P j Generate multiple values p k ( k =1,…, N C ). Use for P j These generated values p k ( k =1,…, N C To generate P j The score, (c) Step 602: By... P j Perform (possibly random) changes / perturbations to generate second candidate samples. Q j , (d) Step 604b: Use a specific ML classification model 322 to target Q j Generate multiple values p k ( k =1,…, N C ). Use for Q j These generated values p k ( k =1,…, N C To generate Q j The score, (e) Step 606: i. Steps 608 and 612: Having already compared the two scores, if... Q j The score indicator is higher than P j If the score is higher, then set P j = Q j , ii. Step 610: Check whether the termination condition has been met, for example, whether one or more of the following conditions have been met: Q jThe score did not indicate a ratio P j The likelihood of a higher score (and potentially, not yet within a predetermined number of previous update steps); for this candidate sample B j The number of update steps performed is now a predetermined threshold; the difference between these two scores is below a predetermined threshold (and potentially, has not been the case within a predetermined number of previous update steps); and so on. If the termination condition has been met, the updated candidate samples... B * j What is obtained P j Otherwise, return to (b); (3) Either set test set 324 as the candidate sample for all updates. B * j Alternatively, select some newer candidate samples. B * j To form a test set of 324 (e.g., a predetermined number of updated candidate samples). B * j (It was selected as the candidate sample with the lowest score).
[0087] Example 2 Example 2 builds upon Example 1. Specifically, the initial technique of Example 1 can be viewed as targeting candidate samples. B j ( j =1,…, N B Sequence 502 of one or more update steps j The first subsequence step (i.e., the first stage), where Example 2 then adds one or more update steps after the first subsequence sequence 502 j The second subsequence step (i.e., the second stage). The second stage can be regarded as a refinement of the first stage.
[0088] Specifically, during the second phase, the current candidate samples P j Second candidate sample Q j The classification metric is the corresponding identifier of the category of the sample generated using a specific ML classification model 322 (i.e., the category to which the sample belongs according to the specific ML classification model 322). Evaluating these generated classification metrics at step 606 then includes: if the second candidate sample... Q j Classification metrics and current candidate samplesP j If the classification metrics are the same (i.e., the two samples are determined to be in the same category), then the second subsequence terminates at step 610; while if the second candidate sample Q j Classification metrics and current candidate samples P j If the classification metrics are different (i.e., the two samples are identified as belonging to different categories), then: for the current candidate sample P j Second candidate sample Q j For each of the samples, generate the corresponding score as discussed above with reference to Example 1, and if the second candidate sample Q j The score indicates a difference compared to the current candidate sample. P j If the candidate sample has a higher probability of scoring, then (at step 612) the current candidate sample will be... P j Set as the second candidate sample Q j .
[0089] Therefore, in this example, the method used to generate test set 324 involves: (1) Step 502: Obtain candidate samples { B j : j =1,…, N B The first set of} is as discussed above; (2) By targeting each candidate sample B j ( j =1,…, N B The following steps are performed to update each candidate sample: (a) Initially, P j = B j , (b) Step 604a: Use a specific ML classification model 322 to target P j Generate multiple values p k ( k =1,…, N C ). Use for P j These generated values p k (k =1,…, N C To generate P j The score, (c) Step 602: By... P j Perform (possibly random) changes / perturbations to generate second candidate samples. Q j , (d) Step 604b: Use a specific ML classification model 322 to target Q j Generate multiple values p k ( k =1,…, N C ). Use for Q j These generated values p k ( k =1,…, N C To generate Q j The score, (e) Step 606: i. Steps 608 and 612: Having already compared the two scores, if... Q j The score indicator is higher than P j If the score is higher, then set P j = Q j , ii. If one or more criteria are met, then phase 1 ends and proceeds to phase 2, for example, whether one or more of the following criteria are met: Q j The score did not indicate a ratio P j The likelihood of a higher score (and potentially, not yet within a predetermined number of previous update steps); for this candidate sample B j The number of update steps performed is now a predetermined threshold; the difference between these two scores is below a predetermined threshold (and potentially, this has not been the case within a predetermined number of previous update steps); and so on. If this criterion is met, proceed to (f) below; otherwise, return to (b). (f) Step 604a: Use a specific ML classification model 322 to determine Pj Category (g) Step 602: By... P j Perform (possibly random) changes / perturbations to generate second candidate samples. Q j , (h) Step 604b: Use a specific ML classification model 322 to determine Q j Category (i) Step 606: • Steps 608 and 610: If, after comparing the two categories, Q j Category and P j If the categories are the same, then the updated candidate samples are... B * j What is obtained P j , • Otherwise, step 612: Use a specific ML classification model 322 to target P j Generate multiple values p k ( k =1,…, N C (It will be understood that these scores may have already been generated at (f) above, and therefore do not need to be regenerated.) Using targeted P j These generated values p k ( k =1,…, N C To generate P j The score. Using a specific ML classification model 322 to target Q j Generate multiple values p k ( k =1,…, N C (It will be understood that these scores may have already been generated at (h) above, and therefore do not need to be regenerated.) Using targeted Q j These generated values p k ( k =1,…, N C To generate Qj The score. If Q j The score indicator is higher than P j If the score is higher, then set P j = Q j Return to step (f); (3) Either set test set 324 as the candidate sample for all updates. B * j Alternatively, select some newer candidate samples. B * j To form a test set of 324 (e.g., a predetermined number of updated candidate samples). B * j (It was selected as the candidate sample with the lowest score).
[0090] Above, steps (2)(a)-(e) form the first stage, and steps (2)(f)-(i) form the second stage.
[0091] Example 3 In this example, at step 602, by analyzing the current candidate samples... P j Implement (possibly random) changes to generate a second candidate sample. Q j Current candidate samples P j Second candidate sample Q j The classification metric is the corresponding identifier of the category of the sample generated using a specific ML classification model 322 (i.e., the category to which the sample belongs according to the specific ML classification model 322). Evaluating these generated classification metrics at step 606 then includes: at step 610, if the second candidate sample... Q j Classification metrics and current candidate samples P j If the classification metrics are the same (i.e., the two samples are identified as belonging to the same class), then the process for selecting candidate samples is terminated. B j ( j =1,…, N B Sequence 502 of one or more update steps j However, if the second candidate sample Q j Classification metrics and current candidate samplesP j If the classification metrics are different (i.e., the two samples are identified as belonging to different categories), then at step 612, the current candidate sample... P j Set as the second candidate sample Q j And when a second candidate sample is generated for the next update step at step 602. Q j At the same time, it also reduces the amount that needs to be applied to candidate samples. P j The magnitude of the change. Therefore, the magnitude of the change gradually becomes smaller and smaller until the current candidate sample. P j Second candidate sample Q j Both were found to belong to the same category. The magnitude of the change can be viewed as a distance metric of 250 within the current candidate sample space. P j Compared with the second candidate sample Q j The distance between them. Due to this gradually smaller size change implemented at step 602, the "second candidate sample" at step 608 Q j Classification metrics and current candidate samples P j The flag indicating "different classification metrics" indicates the second candidate sample. Q j Compared to the current candidate samples P j It is more likely to be "sensitive" (because of the second candidate sample) Q j Is it using a ratio for the current candidate sample? P j The changes are obtained by smaller changes, and therefore are more likely to be closer to the category boundary.
[0092] Therefore, in this example, the method used to generate test set 324 involves: (1) Step 502: Obtain candidate samples { B j : j =1,…, N B The first set of} is as discussed above; (2) By targeting each candidate sample B j ( j =1,…, N BThe following steps are performed to update each candidate sample: (a) Initially, P j = B j , (b) Step 604a: Use a specific ML classification model 322 to determine P j Category (c) Step 602: By... P j Perform (possibly random) changes / perturbations to generate second candidate samples. Q j If this is not sequence 502 j The first update step involves changes that are less significant than changes made to previous update steps. (d) Step 604b: Use a specific ML classification model 322 to determine Q j Category (e) Step 606: i. Steps 608 and 610: If, after comparing the two categories, Q j Category and P j If the categories are the same, then the updated candidate samples are... B * j What is obtained P j , ii. Otherwise, step 612: setting P j = Q j Return to step (b); (3) Either set test set 324 as the candidate sample for all updates. B * j Alternatively, select some newer candidate samples. B * j To form a test set of 324 (e.g., a predetermined number of updated candidate samples). B * j They were selected as candidate samples with the smallest change at the end of their respective sequences 502.
[0093] As will be understood, this third example can be used when the test set creation system 320 does not have access to the values generated by a specific ML classification model 322. pk ( k =1,…, N C However, it will be understood that the test set creation system 320 does indeed have access to the values generated by the specific ML classification model 322. p k ( k =1,…, N C In the case of ), the third example can still be used.
[0094] What will be understood is that, for this third example, regarding the candidate sample B j ( j =1,…, N B Sequence 502 of one or more update steps j Additional steps (e.g., before or after) of steps (2)(a)-(e) discussed above may be involved, such that steps (2)(a)-(e) can be considered as targeting candidate samples. B j ( j =1,…, N B A larger sequence of one or more update steps 502 j The subsequence of steps in. In fact, for example, although Example 2 above adds a step as a second stage to the end of Example 1, steps (2)(a)-(e) of Example 3 can be used as the second stage instead of the second stage steps discussed in Example 2.
[0095] Example 4 Any method can be used to generate so-called "adversarial examples" against this model, and these adversarial examples can be treated as test samples for test set 324. Adversarial examples and their generation methods (whether with or without full access to the model and the generated intermediate values) are well known—see, for example, https: / / openai.com / blog / adversarial-example-research / and https: / / towardsdatascience.com / getting-to-know-a-black-box-model-374e180589ce, all of which are incorporated herein by reference in their entirety.
[0096] (c) Experimental results The following is an example experiment illustrating the efficacy and effectiveness of embodiments of the present invention. In this experiment, the well-known "double spiral" classification task was used: given the x and y coordinates of a point, the task aims to determine whether the point belongs to category (spiral) 1 or category (spiral) 2. Given a pair of coordinates, it is assumed that different models can assign different categories to it, while the same model always assigns the same category to that pair of coordinates.
[0097] data: First, create a standard double helix dataset. Next, add noise values to the x and y coordinates of each point. Figure 7 The illustration shows the example double-helix dataset. 2,000 points were created to form the training set. Different sets of 2,000 points were created to form the validation set for verifying / validating the trained model.
[0098] Model: A multilayer perceptron (MLP) model was used. As is well known, MLPs are a type of feedforward artificial neural network. Throughout this experiment, the MLP model used consisted of the following architecture: • It has a 12-unit input layer, 2-dimensional input data, and an activation function of tanh; • Each hidden layer has one or more hidden layers with 12 hidden units and an activation function tanh; • It has an output layer with one unit and the activation function sigmoid.
[0099] The number of hidden layers used ranged from 1 to 4. Nine different numbers of epochs were used: 48, 49, 50, 98, 99, 100, 148, 149, and 150. The batch size used was 10. The total number of trained models was 36 (4 × 9). Models with reasonable performance (i.e., max(false positive rate, false negative rate) < 5%) were then selected based on model validation / verification using a validation set. This resulted in 18 models, leading to 153 (18 × 17 / 2) model pairs. Each model pair contained two different models—the purpose of which was that, for each pair, a test could be performed where one of the models was (or acted as) a specific ML classification model 322, and the other model was (or acted as) an ML classification model 312.
[0100] Throughout the experiment, the model was trained using the training set generated as described above.
[0101] Generate test set 324: The following techniques were used to generate test set 324: • Input: The model; M (Floating-point), indicating the multiplier; T (Integer), indicating the number of points generated from a given interval; P (Integer), indicating the number of times the search process is repeated; interval [ x min , x max ] indicates the minimum and maximum numbers of the x-coordinates of the test set; • Output: Two samples from the test set of 324; • Steps: o in the interval [ x min , x max [Generated within] T Numbers with uniform intervals, o Obtaining samples ( x 1, x 1× M ), ( x 2, x 2× M )……( x T , x T × M ), o Use this model to obtain the corresponding prediction / class of the sample, as z 1, z 2,…, z T , o Order ( x A , x A × M ), ( x B , x B × M ) are those with different predictions / categories (i.e. z A ≠z B Two adjacent samples. Assume... x A < x B , o Perform the following binary search P Second-rate: ▪ Settings , ▪ Use this model to obtain points ( x MID ,x MID × M Prediction / Category z MID , ▪ If z MID = z A Then set x A = x MID , ▪ If z MID = z B Then set x B = x MID , o Return ( x A , x A × M )and( x B , x B × M () as test samples.
[0102] In this experiment, P = N =10, and M Take a value of 1.0 or 0.5. For this experiment, 28 test samples were generated for each model's test set 234.
[0103] The above content can be considered an implementation of Example 3 above. Specifically: the sample ( x A , x A × M ) can be taken as candidate samples B j Second candidate sample Q j It is a point ( x MID , x MID × M ); and updated candidate samples B * j yes( x A , x A × M )and(x B , x B × M Similarly, this can be viewed as a sequence of one or more update steps performed in parallel for two candidate samples. Specifically: sample ( x A , x A × M )and( x B , x B × M Two initial candidate samples can be selected. B j ; for the second candidate samples of both Q j It is a point ( x MID , x MID × M ); and updated candidate samples B * j They are ( x A , x A × M )and( x B , x B × M ).
[0104] test: Examples of single tests include: • Input: Model A, Model B, sample points ( x , y ); • Output: The number of errors; • Steps: o Obtain model A for sample points ( x , y Prediction / Category z A , o Obtain model B for sample points ( x , y Prediction / Category z B , o If z A equals z B If the condition is met, return 0; otherwise, return 1.
[0105] The sample sizes used range from 1 to 20. For each sample size, "sensitive" samples are first randomly selected from the test set 324 for model A, and then "regular" samples are randomly selected from the entire sample space 250. The testing steps described above are then performed on each selected sample. This process is repeated 10 times for each sample size. In fact, when testing with "regular" samples, a larger range of sample sizes (from 1 to 1000) is required to allow for comparison with embodiments of the invention using "sensitive" samples.
[0106] Measurement: The error rate is used to indicate the probability that the selected method (either using "sensitive samples" or "normal samples") will detect two different models as the same. This metric is defined as: .
[0107] The lower the error rate, the better the method can distinguish between two different models.
[0108] Discover: Figure 8 and Figure 9 This demonstrates the results when using test set 324 (the proposed method) or only using "regular" (purely randomly selected) test samples (the baseline method). Figure 4 A graph illustrating the error rate of the testing method. For example... Figure 8 As shown, the embodiments of the present invention (the proposed method) outperform the baseline method in terms of error rate. Specifically, the proposed method requires 11 samples to achieve an error rate of 0. The baseline method requires 691 samples (46 times more) to achieve comparable performance (e.g., ...). Figure 9 (As shown in the diagram). A single sample from the test set of 324 achieved an average error rate of 0.47712, while the baseline method required 27 regular samples to achieve a comparable error rate of 0.4977.
[0109] If we define 1 to... as follows N Scenario: Consider a basic model and N If the task is to detect a list of models, then the task is to detect... N Is any of the models identical to the base model? To achieve perfect performance (0 error rate), the number of queries required for the baseline method is 691 × N The number of proposed methods is 80 + 11 × N (Here, in this experiment, there are 80 queries to generate test set 324). For example... Figure 10As shown, the number of queries required by the baseline method is significantly greater than that required by the proposed method (7 to 58 times greater).
[0110] (d) Modify It will be understood that the described method has been shown as individual steps performed in a specific order. However, those skilled in the art will appreciate that these steps can be combined or performed in different orders while still achieving the desired result.
[0111] It will be understood that embodiments of the invention can be implemented using a variety of different information processing systems. In particular, although the accompanying drawings and discussion provide exemplary computing systems and methods, these are presented merely to provide useful reference in discussing the various aspects of the invention. Embodiments of the invention can be implemented on any suitable data processing device, such as a personal computer, laptop, personal digital assistant, mobile phone, set-top box, television, server computer, etc. Of course, for the purposes of discussion, the descriptions of the systems and methods have been simplified, and they are merely one of many different types of systems and methods that can be used in embodiments of the invention. It will be understood that the boundaries between logical blocks are merely illustrative, and alternative embodiments may combine logical blocks or elements, or may impose alternative decompositions of functionality under various logical blocks or elements.
[0112] It will be understood that the functions mentioned above can be implemented as one or more corresponding modules in hardware and / or software. For example, the functions mentioned above can be implemented as one or more software components for execution by the system's processor. Alternatively, the functions mentioned above can be implemented as hardware, such as one or more field-programmable gate arrays (FPGAs), and / or one or more application-specific integrated circuits (ASICs), and / or one or more digital signal processors (DSPs), and / or one or more graphics processing units (GPUs) and / or other hardware arrangements. The method steps implemented in the flowcharts included herein or described above can all be implemented by their respective corresponding modules; multiple method steps implemented in the flowcharts included herein or described above can be implemented together by a single module.
[0113] It will be understood that, to the extent that embodiments of the invention are implemented by computer programs, one or more storage media and / or one or more transmission media storing or carrying the computer program form aspects of the invention. A computer program may have one or more program instructions or program code that, when executed by one or more processors (or one or more computers), implement embodiments of the invention. As used herein, the term "program" can be a sequence of instructions designed to execute on a computer system and may include subroutines, functions, procedures, modules, object methods, object implementations, executable applications, applets, service programs, source code, object code, bytecode, shared libraries, dynamic link libraries, and / or other sequences of instructions designed to execute on a computer system. Storage media may be disks (such as hard disks or floppy disks), optical disks (such as CD-ROMs, DVD-ROMs, or Blu-ray discs), or memory (such as ROMs, RAMs, EEPROMs, EPROMs, flash memory, or portable / removable memory devices), etc. Transmission media may be communication signals, data broadcasts, communication links between two or more computers, etc.
Claims
1. A computer-implemented method for generating a test set for use in a method for identifying whether a classification system is using a specific machine learning classification model, the test set being associated with the specific machine learning classification model, wherein the test set includes a plurality of test samples, the test samples including at least one of images, audio data, and emails, and for each test sample in the test set, there exists a corresponding minor modification for that test sample, the minor modification resulting in a change in the classification result generated for that test sample using the specific machine learning classification model, wherein the method includes: Obtain a first set of multiple candidate samples applicable to the specific machine learning classification model; as well as Updating the first set, the update comprising, for each candidate sample, performing a corresponding sequence of one or more update steps, wherein each update step includes: A second candidate sample is generated based on the candidate sample; For each of the candidate samples and the second candidate samples, the specific machine learning classification model is used to generate a corresponding classification metric; and Evaluate the generated classification metric, wherein the evaluation includes: based on a comparison of the generated classification metric, performing one or more of the following steps: (a) terminating a sequence of one or more update steps; (b) setting the candidate sample as the second candidate sample if the comparison indicates that the second candidate sample is more likely to have a corresponding minor modification that would lead to a change in the classification result produced using the particular machine learning classification model. The test set mentioned therein includes some or all of the updated first set.
2. The method of claim 1, wherein the generating a second candidate sample based on the candidate sample comprises: A second candidate sample is generated by performing random changes on the candidate sample.
3. The method of claim 1, wherein for each candidate sample, for each update step of the first subsequence of the corresponding sequence of one or more update steps: For each of the candidate samples and the second candidate samples, the corresponding classification metric is a score generated through the following steps: generating, using the particular machine learning classification model, a corresponding plurality of values, each value indicating that the sample belongs to a corresponding class; as well as A scoring function is used to generate a score for the sample based on the corresponding multiple values, the score indicating the possibility of a small modification to the sample that results in a change in the classification result generated for the sample using the specific machine learning classification model; as well as The evaluation of the generated classification metric includes setting the candidate sample as the second candidate sample if the classification metric of the second candidate sample indicates a higher probability than the classification metric of the candidate sample.
4. The method of claim 1, wherein for each candidate sample, the corresponding sequence of the one or more update steps comprises a plurality of update steps, and for each update step of the second subsequence of the corresponding sequence of the one or more update steps following the first subsequence: For each of the candidate sample and the second candidate sample, the corresponding classification metric is an identifier of the category of the sample generated using the specific machine learning classification model; as well as The evaluation of the generated classification metrics includes: If the classification metric of the second candidate sample is the same as that of the candidate sample, then the second subsequence is terminated. If the classification metric of the second candidate sample is different from that of the candidate sample: For each of the candidate samples and the second candidate samples: The specific machine learning classification model is used to generate multiple corresponding values, each indicating that the sample belongs to a corresponding category; and A scoring function is used to generate a score for the sample based on the corresponding multiple values, the score indicating the possibility of a small modification to the sample that results in a change in the classification result produced for the sample using the specific machine learning classification model; and If the score of the second candidate sample indicates a higher probability than the score of the first candidate sample, then the first candidate sample is set as the second candidate sample.
5. The method according to claim 3 or 4, wherein each value represents the probability that the corresponding sample belongs to the corresponding category.
6. The method of claim 5, wherein for each of the first candidate sample and the second candidate sample, the corresponding plurality of values are normalized to have a predetermined sum.
7. The method of claim 1, wherein for each candidate sample, for each update step of the first subsequence of the corresponding sequence of one or more update steps: For each of the candidate sample and the second candidate sample, the corresponding classification metric is an identifier of the category of the sample generated using the specific machine learning classification model; The evaluation of the generated classification metrics includes: If the classification metric of the second candidate sample is the same as that of the candidate sample, then the first subsequence is terminated if the termination condition is met. If the classification metric of the second candidate sample is different from that of the candidate sample: Set the candidate sample as the second candidate sample; and When a second candidate sample is generated at the next update step, the size of the random changes to be applied to the candidate sample is reduced.
8. The method of claim 1, wherein one or more test samples in the test set are generated as adversarial examples for the particular machine learning classification.
9. A method for identifying whether a classification system is configured to use a specific machine learning classification model, the method comprising: The classification system is used to generate a corresponding classification result for each test sample in a predetermined test set comprising multiple test samples, wherein the test samples include at least one of images, audio data, and emails, and the predetermined test set is generated using the method according to any one of claims 1 to 8; and The classification system is using the specific machine learning classification model if (i) for each test sample in the test set, the corresponding classification result matches the classification result generated by the specific machine learning classification model for that test sample; or (ii) if there are test samples in the test set for which the corresponding classification result does not match the classification result generated by the specific machine learning classification model for that test sample, then the classification system is not using the specific machine learning classification model. The test set is associated with the specific machine learning classification model, and for each test sample in the test set, there is a corresponding minor modification that results in a change in the classification result generated by the specific machine learning classification model for that test sample.
10. An apparatus configured to perform the method according to any one of claims 1 to 9.
11. A computer program product that, when executed by one or more processors, causes the one or more processors to perform the method according to any one of claims 1 to 9.
12. A computer-readable medium storing a computer program product according to claim 11.
