Evaluation method, evaluation program, and information processing device

Abstract

Provided are an evaluation method, an evaluation program, and an information processing apparatus for enhancing safety in machine learning. The present invention involves: for a plurality of queries requesting execution of inference pertaining to individual pieces of input data, performing inference for each piece of input data using a classification model; obtaining certainty factors for individual classes by which the input data for each of the queries is classified through the inference; calculating the difference in certainty factor between a first class and a second class for each of the queries; and evaluating the plurality of queries on the basis of the tendency of increase or decrease in the calculated differences in certainty factor.

Description

[Technical Field]

[0001] The present invention relates to an evaluation method, an evaluation program, and an information processing device. [Background technology]

[0002] In recent years, there has been an increase in the number of cases where data is classified using machine learning. Machine learning processing is divided into a learning phase and an inference phase. In the learning phase, for example, a learning process is performed using training data and correct labels as input, thereby generating a trained machine learning model. In the inference phase, an inference process is performed using the trained machine learning model created in the learning phase using inference target data as input, and a classification result label is output as the result of the inference process.

[0003] In the inference process, when data to be inferred is input, multiple inference candidates are obtained through inference processing using a trained machine learning model, and a confidence score is calculated for each inference candidate.The inference candidate with the highest confidence score is then output as the classification result label.

[0004] During the inference phase of an image recognition system using machine learning, the addition of a type of noise called perturbations to an input image can cause the input image to be misclassified, which can be a problem. In particular, adversarial vulnerability attacks against trained machine learning models can cause input images to be misclassified in image recognition systems. For example, there is an attack in which data that appears to the human eye as A is intentionally created, but is judged as B in the inference process using a trained model. Data that is intentionally created to cause a misclassification in the inference process is known as an adversarial example.

[0005] Although image data has been used as an example here, adversarial samples can be generated for inference processing of any type of data, not just images. For example, in the process of recognizing speech and classifying the content of a conversation, adversarial samples can be generated by adding noise that humans cannot perceive to the speech data, so that what sounds to humans as conversation C is judged as conversation D.

[0006] Adversarial samples are generated by adding a certain amount of noise to the original image. This can lead to, for example, an image that would be recognized as a panda by the human eye being mistakenly classified as a gibbon by the inference process. Therefore, it is desirable to have an appropriate defense against attacks using adversarial samples.

[0007] Many methods for generating adversarial samples use internal information about machine learning models, but it is difficult to access internal information about models in publicly available services, etc. In such cases, attackers may create adversarial samples in the following ways:

[0008] An attacker adds appropriate noise to an image that they wish to misjudge. Next, they send a query to the machine learning model being attacked using the image with the added noise. Next, the attacker obtains an inference result for the image with the added noise that they wish to misjudge as a response to the sent query. The attacker then updates the added noise based on the obtained inference result. By repeatedly submitting a query using the image with the added noise, obtaining an inference result for the query, and updating the noise using the obtained inference result, the attacker obtains noise that will cause a misjudgement and creates an adversarial sample. An attack that uses an adversarial sample in this way to misjudge the inference results of a machine learning model is sometimes called a query-based black-box attack.

[0009] As a method to defend against this query-based black-box attack, there is a technology that detects the attacker's manipulation to generate adversarial samples for creating adversarial samples. For example, since generating appropriate noise involves continuously inputting images with gradually changing noise, a technology has been proposed to detect query-based black-box attacks by detecting the input of similar images.

[0010] For example, one technique for detecting adversarial samples is to calculate a similarity metric for a set of images input in time series, and detect an adversarial sample when the number of similar images exceeds a threshold. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Japanese Patent Publication No. 2020-200033 Summary of the Invention [Problem to be solved by the invention]

[0012] However, there may be cases where the input data cannot be stored due to storage capacity or privacy concerns. If the input data is not available, it is difficult to detect adversarial samples using, for example, the method of detecting adversarial samples based on similar images described above.

[0013] The disclosed technology has been made in consideration of the above, and aims to provide an evaluation method, an evaluation program, and an information processing device that improve the safety of machine learning. [Means for solving the problem]

[0014] In one aspect, the evaluation method, evaluation program, and information processing device disclosed in the present application perform inference on each of a plurality of queries that request execution of inference on individual input data using a classification model, obtain a confidence level for each class into which the input data has been classified for each of the queries through the inference, and calculate a difference in the confidence levels between a first class and a second class for each of the queries; An attack candidate state in which the difference in the certainty decreases at a certain rate or more is detected, and when the attack candidate state is detected consecutively for a number of times or more than a threshold, it is determined that there is a first possibility of an attack, and when the difference in the certainty becomes equal to or less than a certainty difference threshold, it is determined that there is a second possibility of an attack, and when it is determined that there is the first possibility and that there is the second possibility, it is determined that there is a possibility that the multiple queries are an attack. evaluate. [Effects of the Invention]

[0015] According to one aspect of the evaluation method, evaluation program, and information processing device disclosed in the present application, it is possible to improve the safety of machine learning. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating an outline of a query-based black-box attack. [Figure 3] FIG. 3 is a diagram showing an example of changes in the confidence levels of the classes with the highest and second highest confidence levels when a query-based black-box attack is performed. [Figure 4] FIG. 4 is a diagram showing another example of changes in the certainty of the classes with the highest and second highest certainty when a query-based black-box attack is performed. [Figure 5] FIG. 5 is a diagram showing an example of a change in the confidence difference. [Figure 6] FIG. 6 is a diagram showing predicted ranking points according to changes in the difference in confidence. [Figure 7] FIG. 7 is a diagram showing an example of a case where linear regression is performed using queries obtained over a certain period of time. [Figure 8] FIG. 8 is a diagram showing some examples of changes in the confidence difference. [Figure 9] FIG. 9 is a flowchart of an attack detection process performed by the information processing device according to the first embodiment. [Figure 10]FIG. 10 is a diagram illustrating the transition of the difference in the confidence level when an attack is detected according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating the hardware configuration of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the evaluation method, evaluation program, and information processing device disclosed in the present application will be described in detail with reference to the drawings. Note that the evaluation method, evaluation program, and information processing device disclosed in the present application are not limited to the following embodiments. [Example]

[0018] 1 is a block diagram of an information processing device. The information processing device 1 according to this embodiment includes a data acquisition unit 11, an inference unit 12, a data storage unit 13, a feature extraction unit 14, an attack possibility determination unit 15, a confidence difference determination unit 16, a detection unit 17, and a notification unit 18. The information processing device 1 is connected to an input terminal device 2, which is an external device, via a network such as the Internet. The input terminal device 2 is a device for inputting inference target data for causing the information processing device 1 to perform inference. There are multiple input terminal devices 2. The user of the input terminal device 2 is not limited to a person who causes the information processing device 1 to perform normal inference, but may also be an attacker aiming to perform a query-based black box attack.

[0019] The data acquisition unit 11 acquires inference target data input from the input terminal device 2. Then, the data acquisition unit 11 outputs the acquired inference target data to the inference unit 12.

[0020] The inference unit 12 has a machine learning model, which is a learned classification model trained using learning data. The inference unit 12 receives input of inference target data from the data acquisition unit 11. The inference unit 12 then inputs the inference target data into the learned machine learning model and obtains an inference result as its output. Here, the inference unit 12 obtains, as an inference result, a confidence level for each class into which the inference target data is classified. Thereafter, the inference unit 12 transmits the inference result to the input terminal device 2, which is the source of the inference target data.

[0021] Furthermore, the inference unit 12 acquires information on the class with the highest confidence level and its confidence level, as well as information on the class with the second highest confidence level and its confidence level, from among the inference results for the input inference target data. Then, the inference unit 12 stores the information on the class with the highest confidence level and the class with the second highest confidence level and their respective confidence levels in the data storage unit 13 in association with the identification information of the user who input the query to cause inference to be performed for the inference target data. The identification information of the user who input the query to cause inference to be performed for the inference target data is, for example, the identification information of the input terminal device 2.

[0022] The data storage unit 13 is a storage device such as a hard disk, etc. The data storage unit 13 stores information on the class with the highest and second highest confidence levels for each query input from the inference unit 12, as well as the respective confidence levels, in association with the identification information of the user who submitted the query using the inference target data that is the subject of the inference.

[0023] Fig. 2 is a diagram showing an outline of a query-based black box attack. Here, a query-based black box attack will be briefly explained with reference to Fig. 2. Here, the explanation will be given for the case where inference is made on image data.

[0024] The attacker generates attack data by adding appropriate noise 102 to misjudgment target data 101, which is the target of misjudgment. The attacker then inputs a query that causes the machine learning model 100, which is the target of the attack, to perform inference on the attack data. The attacker then causes the machine learning model 100 to perform inference and obtains the inference result on the attack data (step S101). That is, the attacker inputs the attack data to the information processing device 1 and causes the inference unit 12 to perform inference and obtain the inference result. The machine learning model possessed by the inference unit 12 corresponds to the machine learning model 100.

[0025] In this case, the inference result is the confidence level of each class into which the attack data is classified. For example, as shown in Figure 2, the inference result will be described assuming that the class with the highest confidence level is dog and the class with the second highest confidence level is cat. Assume that the attacker aims to generate adversarial samples that will cause the machine learning model 100 to erroneously classify dog images, which are misclassified data 101, as cats.

[0026] The attacker evaluates the fitness of the obtained inference result using a fitness function or the like (step S102). This allows the attacker to determine to what extent the noise 102 at that time can convert the misclassified target data 101 into an image that will result in the desired misclassification. Then, based on the fitness evaluation, the attacker performs an evolutionary strategic adjustment process on the noise 102, adjusting and evolving the noise 102 to increase the probability of misclassification (step S103). Thereafter, the attacker generates attack data by adding newly generated noise 102 to the misclassified target data 101, and causes the machine learning model 100 to perform inference again. (See, for example, https: / / arxiv.org / abs / 2104.15064.)

[0027] By repeating steps S101 to S103, the attacker decreases the certainty of dog, which is ranked first in the inference result, and increases the certainty of cat, which is ranked second. Eventually, the certainty of cat and the certainty of dog are reversed, allowing the attacker to generate noise that causes the misclassification target data 101 to be misclassified, thereby creating an adversarial sample.

[0028] In this way, when a query-based black box attack is performed, the confidence level of the class with the highest confidence level, which is the correct inference result in the inference result, decreases, while the confidence level of the class with the second highest confidence level, which is the desired incorrect judgment result, increases. Finally, the classes with the highest and second highest confidence levels are reversed in the inference result, making it possible to create an adversarial sample. Therefore, the information processing device 1 according to this embodiment detects the possibility of an attack by utilizing the characteristics of this query-based black box attack.

[0029] Returning to FIG. 1, the following description will focus on queries submitted by a specific user and their inference results. The feature extraction unit 14 determines whether a predetermined number of queries submitted by a specific user to a specific input terminal device 2 have been processed. The predetermined number can be, for example, 100.

[0030] Here, the feature extraction unit 14 repeatedly determines whether a predetermined number of queries have been processed. There are no particular limitations on the method for selecting the predetermined number of queries, as long as the predetermined number of queries is selected so as to include a set that includes newer queries each time the feature extraction unit 14 determines the predetermined number of queries. For example, the feature extraction unit 14 may repeat the determination so as to select the subsequent predetermined number of queries, without including queries that have already been determined as candidates for selection. Alternatively, the feature extraction unit 14 may perform the predetermined number of determinations so as to select queries that overlap some of the queries used in the previous determination.

[0031] The feature extraction unit 14 acquires the inference results of a predetermined number of queries from the data storage unit 13. Next, the feature extraction unit 14 extracts the highest confidence level and its class, and the second highest confidence level and its class in the inference results for each query. The feature extraction unit 14 then calculates the difference between the highest confidence level and the second highest confidence level in the inference results for each query. This difference between the highest confidence level and the second highest confidence level corresponds to the feature of each query entered by the user. Hereinafter, the difference between the confidence level of the first-ranked class and the confidence level of the second-ranked class will be simply referred to as the "difference in confidence level."

[0032] Thereafter, the feature extraction unit 14 outputs information on the difference in confidence in the inference results for each query for the predetermined number of queries to the attack possibility determination unit 15. In addition, the feature extraction unit 14 outputs information on the difference in confidence in the inference results for each query for the predetermined number of queries, and information on the class with the highest confidence and the class with the second highest confidence for each query to the confidence difference determination unit 16.

[0033] The attack possibility determination unit 15 has in advance a time threshold and a count threshold for detecting the possibility of an attack. The attack possibility determination unit 15 receives information on the difference in the confidence level in the inference results for each query for a predetermined number of queries from the feature extraction unit 14.

[0034] Fig. 3 is a diagram showing an example of changes in the certainty of the classes with the highest and second highest certainty levels when a query-based black-box attack is carried out. Fig. 4 is a diagram showing another example of changes in the certainty of the classes with the highest and second highest certainty levels when a query-based black-box attack is carried out. In Figs. 3 and 4, the horizontal axis represents the passage of time, and the vertical axis represents the certainty level.

[0035] 3, graph 201 represents the transition of the certainty of the class with the highest certainty, and graph 202 represents the transition of the certainty of the class with the second highest certainty. For example, when a query-based black box attack is carried out, the certainty of the class with the highest certainty may decrease linearly as shown in graph 201, and the certainty of the class with the second highest certainty may increase linearly as shown in graph 202.

[0036] 4 shows the transition of the confidence level of the class with the highest confidence level, and graph 204 shows the transition of the confidence level of the class with the second highest confidence level. For example, in another example of a query-based black box attack, the confidence level of the first-ranked class may suddenly drop at a specific point in time as shown in graph 203, and the confidence level of the second-ranked class may suddenly rise at a specific point in time as shown in graph 204.

[0037] Fig. 5 is a diagram showing an example of changes in the confidence difference. Fig. 5 shows changes in the confidence difference when the confidence of the first and second ranked classes changes as in Fig. 4. The horizontal axis of Fig. 5 represents the passage of time, and the vertical axis represents the confidence difference. The confidence difference in Fig. 5 is a normalized value, with the confidence difference being 1.0 when the confidence of the first ranked class is 100%.

[0038] When the certainty of the first and second ranked classes changes as shown in Figure 4, the difference in certainty rapidly decreases over time at a specific point in time, as shown in graph 205 of Figure 5. As shown in Figures 3 and 4, the changes in the certainty of the first and second ranked classes vary, but in any case, the difference between the certainty of the first ranked class and the certainty of the second ranked class eventually approaches 0, even if the process of decreasing it varies.

[0039] Therefore, the attack possibility determination unit 15 plots the difference in certainty for each query for a predetermined number of queries in a coordinate space as shown in FIG. 5, where the vertical axis represents the difference in certainty and the horizontal axis represents the passage of time. Next, the attack possibility determination unit 15 performs linear regression on the difference in certainty for each query for the predetermined number of queries to obtain a regression line. Next, the attack possibility determination unit 15 obtains the intercept of the regression line with the horizontal axis, which is the time axis. The intercept of this regression line with the time axis is the time at which the class with the highest certainty predicted from the current query and the class with the second highest certainty will reverse positions, and will be referred to as the "predicted ranking swap point" below.

[0040] The attack possibility determination unit 15 identifies a representative time of the predetermined number of queries. For example, the attack possibility determination unit 15 can set the representative time to the median time when the median query of the predetermined number of queries is submitted. However, the representative time may be any time that corresponds to the predetermined number of queries, and may be, for example, the submission time of the last query of the predetermined number of queries, or the time when the attack possibility determination unit 15 starts processing after a predetermined number of queries have accumulated.

[0041] Next, the attack possibility determination unit 15 determines whether the time from the representative time to the predicted ranking change point is within a time threshold. In other words, the attack possibility determination unit 15 determines whether the predicted ranking change point is included within the time from the representative time until the time threshold has elapsed. The time threshold may be, for example, five minutes. If the time from the representative time to the predicted ranking change point is longer than the time threshold, the attack possibility determination unit 15 waits until the next predetermined number of queries have accumulated.

[0042] On the other hand, if the time from the representative time to the predicted ranking change point is within the time threshold, the attack possibility determination unit 15 determines that an attack candidate has been detected. Then, the attack possibility determination unit 15 determines whether the number of consecutive detections of the attack candidate is equal to or greater than the count threshold. For example, the attack possibility determination unit 15 can set the count threshold to three times.

[0043] If the number of consecutive detections of attack candidates is less than the count threshold, the attack possibility determination unit 15 waits until the next predetermined number of queries accumulates. On the other hand, if the number of consecutive detections of attack candidates is equal to or greater than the count threshold, the attack possibility determination unit 15 detects the possibility of an attack. Thereafter, the attack possibility determination unit 15 notifies the detection unit 17 that there is a possibility of an attack.

[0044] If the time threshold is set short, the attack possibility determination unit 15 will detect an attack candidate when the possibility is higher, and if the time threshold is set long, the attack possibility determination unit 15 will detect an attack candidate even when the possibility is low. Furthermore, if the count threshold is set high, the attack possibility determination unit 15 will have a lower probability of falsely detecting a possible attack, but a higher probability of overlooking an attack. Conversely, if the count threshold is set low, the attack possibility determination unit 15 will have a higher probability of falsely detecting a possible attack, but a lower probability of overlooking an attack. Therefore, it is preferable that the time threshold and count threshold be determined in accordance with the operation, taking into account the probability of false detection and the possibility of overlooking an attack.

[0045] Fig. 6 is a diagram showing predicted ranking points according to changes in the confidence level difference. Fig. 6 shows the changes in the confidence level difference shown in Fig. 5 in time-series graphs 211 to 215. In all of graphs 211 to 215, the horizontal axis represents the passage of time, and the vertical axis represents the confidence level difference.

[0046] As shown in graphs 211 and 212, when there is little change in the difference in confidence level, the regression line 220 obtained by the attack possibility determination unit 15 does not have an intercept with the time axis within the time threshold.

[0047] In contrast, as shown in graph 213, when the difference in confidence level becomes smaller, the regression line 220 has a downward slope over time, and the ranking change prediction point 221, which is the intercept with the time axis, is included between the representative time and the time threshold T.

[0048] If the difference in confidence continues to decrease thereafter, a state in which the rank change predicted point 221 falls within the period from the representative time to the time threshold T continues, as shown in graphs 214 and 215. In other words, if a state in which the rank change predicted point 221 falls within the period from the representative time to the time threshold T continues, it can be said that there is a high possibility of an attack. Therefore, as described above, the attack possibility determination unit 15 determines that there is a possibility of an attack when the number of consecutive rank change predicted points that fall within the time threshold exceeds the count threshold.

[0049] Here, the attack possibility determination unit 15 according to this embodiment performs linear regression using inference results for a predetermined number of queries to calculate the predicted ranking change points. Alternatively, the attack possibility determination unit 15 may perform linear regression using inference results for queries accumulated over a certain period of time to calculate the predicted ranking change points. In this case, the feature extraction unit 14 acquires data from the data storage unit 13 at regular intervals, and the attack possibility determination unit 15 calculates the predicted ranking change points using the acquired data.

[0050] However, when the attack possibility determination unit 15 calculates the predicted ranking change points using queries obtained over a certain period of time, there may be a case where the number of accumulated queries is small. In this case, the attack possibility determination unit 15 creates a regression line using a small amount of data, and the regression line may not accurately reflect changes in the confidence level difference.

[0051] FIG. 7 is a diagram showing an example of a case where linear regression is performed using queries obtained over a certain period of time. For example, the attack possibility determination unit 15 generates a regression line 232 by using the confidence level difference included in range 231. On the other hand, the attack possibility determination unit 15 generates a regression line 234 by using the confidence level difference included in range 233. As such, a difference in just one point causes the slopes of the regression lines 232 and 234 to differ significantly, and it cannot be said that they accurately reflect changes in the confidence level difference. Therefore, if more accurate attack detection is desired, it is preferable that the attack possibility determination unit 15 calculates the predicted position swap point for each predetermined number of queries.

[0052] Continuing the explanation by returning to Fig. 1, the certainty difference determination unit 16 has a predetermined certainty difference threshold. For example, when the difference in certainty is normalized by setting the difference between the first-place class and the second-place class to 1 when the certainty of the first-place class is 100%, the certainty difference determination unit 16 can set the certainty threshold to 0.2.

[0053] The confidence difference determination unit 16 receives from the feature extraction unit 14 information on the difference between the highest confidence and the second highest confidence in the inference results for each query for a predetermined number of queries.

[0054] Next, the confidence difference determination unit 16 compares the confidence difference for each query and identifies the minimum confidence difference among a predetermined number of queries. Next, the confidence difference determination unit 16 determines whether the minimum confidence difference is equal to or less than the confidence threshold. If the minimum confidence difference is equal to or less than the confidence threshold, the confidence difference determination unit 16 determines that there is a possibility of an attack. Then, the confidence difference determination unit 16 notifies the detection unit 17 of the detection of the possibility of an attack. On the other hand, if the minimum confidence difference is greater than the confidence threshold, the confidence difference determination unit 16 determines that there is a low possibility of an attack and terminates the process.

[0055] Here, when the confidence difference threshold is set small, the confidence difference determination unit 16 detects the possibility of an attack when the possibility is high. Conversely, when the confidence difference threshold is set large, the confidence difference determination unit 16 detects the possibility of an attack even when the possibility is relatively low. In this way, it is preferable that the confidence threshold be determined depending on the degree of possibility to be detected.

[0056] When the attack possibility determination unit 15 detects the possibility of an attack, the detection unit 17 receives a notification of the detection of the possibility of an attack from the attack possibility determination unit 15. Similarly, when the confidence difference determination unit 16 detects the possibility of an attack, the detection unit 17 receives a notification of the detection of the possibility of an attack from the confidence difference determination unit 16. When the detection unit 17 receives notifications of the detection of the possibility of an attack from the attack possibility determination unit 15 and the confidence difference determination unit 16, the detection unit 17 detects the possibility of a query-based black box attack. Then, the detection unit 17 notifies the notification unit 18 of the detection of the possibility of a query-based black box attack. In this way, the information processing device 1 according to the present embodiment evaluates queries based on the tendency of increase or decrease in the confidence difference.

[0057] The notification unit 18 receives a notification of the detection of a query-based black box attack from the detection unit 17. Then, the notification unit 18 notifies the user of the information processing device 1 of the detection of a query-based black box attack by, for example, displaying the detection information on a monitor.

[0058] 8 is a diagram showing some examples of changes in the confidence level difference. In all of the graphs 301 to 303, the horizontal axis represents the passage of time, and the vertical axis represents the confidence level difference.

[0059] For example, as shown in graph 301, a case will be described in which the confidence difference decreases linearly. In this case, the position of the range 311 to 313 in which the predicted ranking change points from the representative time to the time threshold fall gradually changes. The regression line obtained from the confidence difference for a predetermined number of queries has an intercept on the time axis in each of the ranges 311 to 313. From this, the attack possibility determination unit 15 detects the possibility of an attack. Furthermore, the minimum value of the confidence difference is equal to or less than the confidence difference threshold 320, and the confidence difference determination unit 16 detects the possibility of an attack. Therefore, in this case, the detection unit 17 detects the possibility of a query-based black box attack.

[0060] Next, as shown in graph 302, a case will be described in which the curve representing the confidence difference is downwardly convex and approaches near 0. In this case, at the time when the ranking change predicted point 321 occurs, the number of times equal to or greater than the count threshold falls continuously within the range from the representative time to the time threshold. From this, the attack possibility determination unit 15 detects the possibility of an attack. However, at this time, the confidence difference is not equal to or less than the confidence difference threshold, and the confidence difference determination unit 16 does not detect the possibility of an attack. Therefore, the detection unit 17 does not detect a query-based black box attack. Thereafter, at the time when the ranking change predicted point 321 occurs, the number of times equal to or greater than the count threshold falls continuously again within the range from the representative time to the time threshold. From this, the attack possibility determination unit 15 detects the possibility of an attack. Then, this time, the confidence difference is equal to or less than the confidence difference threshold, and the confidence difference determination unit 16 detects the possibility of an attack. Therefore, the detection unit 17 detects the possibility of a query-based black box attack.

[0061] Next, as shown in graph 303, a case will be described in which the curve representing the confidence difference is downwardly convex and deviates from 0 halfway through. In this case, when the ranking change predicted point 323 occurs, the number of consecutive points equal to or greater than the count threshold falls within the range from the representative time to the time threshold. From this, the attack possibility determination unit 15 detects the possibility of an attack. However, at this point, the confidence difference is not equal to or less than the confidence difference threshold, and the confidence difference determination unit 16 does not detect the possibility of an attack. After that, the confidence difference does not exceed the confidence difference threshold, and the number of consecutive ranking change predicted points equal to or greater than the count threshold does not fall within the range from the representative time to the time threshold. As a result, the detection unit 17 does not detect a query-based black box attack. In this case, the confidence difference simply happened to become smaller, and it is considered not to be an attack. Therefore, the information processing device 1 can avoid erroneous determination.

[0062] In this embodiment, the possibility of an attack is detected using the difference in certainty between the class with the highest certainty and the class with the second highest certainty. However, this is not limiting, and the possibility of an attack may also be detected using the difference in certainty between the class with the highest certainty and a class ranked other than second. The classes ranked other than second here do not include the class with the highest certainty. Here, some believe that since classes ranked other than second will eventually become second, it is sufficient to ultimately use the difference between the class with the highest certainty and the class ranked second. However, there may be cases where the difference in certainty between the class with the highest certainty and a class ranked other than second becomes rapidly small. In such cases, the information processing device 1 can detect the issuance of a query intended as an attack early on, before a class ranked other than second becomes second.

[0063] 9 is a flowchart of the attack detection process by the information processing device according to the embodiment 1. Next, the flow of the attack detection process by the information processing device 1 according to the embodiment will be described with reference to FIG.

[0064] The data acquisition unit 11 acquires inference target data input from the input terminal device 2 (step S1). Then, the data acquisition unit 11 outputs the acquired inference target data to the inference unit 12.

[0065] The inference unit 12 receives an input of inference target data from the data acquisition unit 11. Then, the inference unit 12 inputs the inference target data into the trained machine learning model, executes inference on the inference target data, and obtains an inference result (step S2).

[0066] The inference unit 12 acquires the first and second most certainties from the inference results and stores them in the data storage unit 13 (step S3).

[0067] The feature extraction unit 14 checks the data stored in the data storage unit 13 and determines whether inference processing for a predetermined number of queries has been performed (step S4). If processing for the predetermined number of queries has not been completed (step S4: No), the attack detection processing returns to step S1.

[0068] On the other hand, if a predetermined number of queries have been processed (step S4: Yes), the feature extractor 14 calculates the difference in confidence between the first and second places for each query (step S5).

[0069] The attack possibility determination unit 15 plots the difference in certainty for each of the predetermined number of queries calculated by the feature extraction unit 14 in a two-dimensional coordinate space in which the vertical axis represents the difference in certainty and the horizontal axis represents the passage of time. Next, the attack possibility determination unit 15 performs linear regression on the difference in certainty for each of the predetermined number of queries (step S6).

[0070] Next, the attack possibility determination unit 15 identifies a predicted ranking change point, which is the intercept of the regression line obtained by the linear regression and the horizontal axis, which is the time axis (step S7).

[0071] Next, the attack possibility determination unit 15 determines a representative time for a predetermined number of queries. Then, the attack possibility determination unit 15 determines whether the time from the representative time to the predicted ranking change point is within a time threshold (step S8). If the time from the representative time to the predicted ranking change point is longer than the time threshold (step S8: No), the attack detection process returns to step S1.

[0072] On the other hand, if the time from the representative time to the predicted ranking change point is within the time threshold (step S8: Yes), the attack possibility determination unit 15 determines whether predicted ranking change points whose time from the representative time is within the time threshold have appeared consecutively more than the threshold number of times (step S9).

[0073] If predicted ranking change points in which the time from the representative time is within the time threshold have not appeared consecutively for a number of times equal to or greater than the threshold (step S9: No), the attack detection process returns to step S1.

[0074] On the other hand, if predicted ranking change points whose time from the representative time is within the time threshold appear consecutively for a number of times equal to or greater than the threshold (step S9: Yes), the attack possibility determination unit 15 notifies the detection unit 17 of the detection of the possibility of an attack. Furthermore, the certainty difference determination unit 16 compares the difference in certainty for each query and identifies the minimum value of the difference in certainty among a predetermined number of queries. Next, the certainty difference determination unit 16 determines whether the minimum value of the difference in certainty is equal to or less than the certainty threshold (step S10). If the minimum value of the difference in certainty is greater than the certainty threshold (step S10: No), the attack detection process returns to step S1.

[0075] On the other hand, if the minimum value of the confidence difference is equal to or less than the confidence threshold (step S10: Yes), the confidence difference determination unit 16 determines that there is a possibility of an attack. Then, the confidence difference determination unit 16 notifies the detection unit 17 of the detection of the possibility of an attack. The detection unit 17 receives notifications of the detection of the possibility of an attack from both the attack possibility determination unit 15 and the confidence difference determination unit 16. As a result, the detection unit 17 detects the possibility of a query-based black box attack (step S11).

[0076] As described above, the information processing device according to this embodiment calculates the difference in confidence between the first-place inference result for each query and the other rankings among a predetermined number of queries, and performs linear regression using the calculated confidence difference in a coordinate space representing the confidence difference and the passage of time.The information processing device then detects the possibility of a query-based black box attack based on the conditions that an event in which the intercept of the regression line and the time axis falls within a time threshold has occurred consecutively for more than a threshold number of times, and that the minimum confidence difference is equal to or less than a confidence difference threshold.In other words, the information processing device according to this embodiment evaluates queries based on the tendency of increases and decreases in the confidence difference.

[0077] This allows for the detection of attacks and the suppression of adversarial sample generation activities for queries suspected of being query-based black-box attacks without using input data. Therefore, it is possible to protect machine learning systems from attacks and improve their security while minimizing increases in storage capacity and protecting user privacy. Furthermore, since a possible attack is determined when the confidence difference falls below a certain value, it is possible to capture attack behavior that is not seen in similar image inputs that are not attacks. Therefore, it is possible to suppress false positives of attacks when the confidence difference becomes small by chance, rather than due to an attack. Furthermore, it is possible to eliminate the need to mistakenly block legitimate users, thereby improving the reliability of machine learning systems. [Example]

[0078] Next, a second embodiment will be described. An information processing device 1 according to this embodiment is also represented by the block diagram of FIG. 1. The information processing device 1 according to this embodiment differs from the first embodiment in that it detects an attack when the order of the class with the highest confidence level and the class with the second highest confidence level is swapped. In the following explanation, explanations of the operations of the same parts as those in the first embodiment will be omitted.

[0079] The feature extraction unit 14 outputs the difference in the certainty of each query for the predetermined number of queries to the attack possibility determination unit 15. The feature extraction unit 14 outputs information on the first and second highest classes of certainty of each query for the predetermined number of queries to the certainty difference determination unit 16.

[0080] As in Example 1, the attack possibility determination unit 15 performs linear regression using the difference in confidence to calculate the predicted ranking change point, and detects the possibility of an attack when the predicted ranking change point falls within the time threshold from the representative time for a number of consecutive times greater than the threshold.

[0081] The confidence difference determination unit 16 receives information on the first- and second-ranked classes of confidence for each of a predetermined number of queries from the feature extraction unit 14. The confidence difference determination unit 16 then determines whether the first-ranked class and the second-ranked class in confidence have swapped places. For example, the confidence difference determination unit 16 determines that the first-ranked class and the second-ranked class in confidence have swapped places when the last class of the predetermined number of queries has swapped from the previous first- and second-ranked classes. Alternatively, the confidence difference determination unit 16 may determine that the first-ranked class and the second-ranked class in confidence have swapped places when the proportion of classes that have swapped from the previous first- and second-ranked classes exceeds a certain proportion in a certain number of queries from the end of the predetermined number of queries. If it is determined that the first- and second-ranked classes have swapped places, the confidence difference determination unit 16 detects an attack. The confidence difference determination unit 16 then notifies the detection unit 17 of the detection of the attack.

[0082] When the detection unit 17 receives a notification of the detection of the possibility of an attack from the attack possibility determination unit 15 and also receives a notification of the detection of an attack from the confidence difference determination unit 16, the detection unit 17 detects a query-based black box attack.

[0083] Fig. 10 is a diagram showing the transition of the difference in the certainty factor in the case of attack detection according to Example 2. In Fig. 10, the horizontal axis represents the passage of time, and the vertical axis represents the difference in the certainty factor. Here, an example will be described in which a query-based black box attack is being carried out to erroneously identify an image of a dog as a cat.

[0084] For example, the attack possibility determination unit 15 performs linear regression using the confidence difference group 401 shown to calculate a regression line 411. Then, the attack possibility determination unit 15 calculates a ranking change predicted point 421 and determines that it falls within a time threshold from the representative time. Furthermore, the confidence 431 for the last query is 0.51 for dog and 0.48 for cat. In this case, if the class with the highest confidence level up to that point was dog and the class with the second highest confidence level was cat, the confidence difference determination unit 16 would not detect an attack. Therefore, the detection unit 17 would not detect a query-based black box attack.

[0085] Next, the attack possibility determination unit 15 performs linear regression using the confidence difference group 402 to calculate a regression line 412. Then, the attack possibility determination unit 15 calculates a ranking change predicted point 422 and determines that it falls within the time threshold from the representative time. Furthermore, the confidence 432 for the last query is 0.50 for dog and 0.49 for cat. In this case, too, the confidence difference determination unit 16 does not detect an attack. Therefore, the detection unit 17 does not detect a query-based black box attack.

[0086] Next, the attack possibility determination unit 15 performs linear regression using the confidence difference group 403 to calculate a regression line 413. Then, the attack possibility determination unit 15 calculates a rank change predicted point 423 and determines that it falls within the time threshold from the representative time. In this case, since the rank change predicted points 421 to 423 consecutively fall within the time threshold from the representative time, when the count threshold is 3, the attack possibility determination unit 15 detects the possibility of an attack. Furthermore, the confidence 433 in the last query is 0.50 for cat and 0.49 for dog. In this case, the confidence difference determination unit 16 detects the exchange of ranks between the class with the highest confidence and the class with the second highest confidence, and detects an attack. Therefore, the detection unit 17 detects a query-based black box attack.

[0087] As described above, the information processing device according to this embodiment detects a query-based black box attack using the condition that the number of consecutive ranking change prediction points within a time threshold from the representative time is equal to or greater than the count threshold, as well as the condition that the first and second most likely classes have switched positions. If the possibility of an attack is detected when the difference in the most likely classes becomes small, as in the first embodiment, alerts may be issued frequently by the information processing device. In contrast, the information processing device according to this embodiment uses the condition that the first and second most likely classes have switched positions, thereby issuing an alert when an attack is deemed to be certain to some extent. This increases the probability of preventing erroneous judgments and enables more accurate detection of query-based black box attacks. Therefore, it is possible to protect machine learning systems from attacks and improve their security.

[0088] (Hardware configuration) 11 is a hardware configuration diagram of the information processing device. Next, an example of a hardware configuration for realizing each function of the information processing device 1 will be described with reference to FIG.

[0089] 11, the information processing device 1 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0090] The network interface 94 is an interface for communication between the information processing device 1 and an external device. The network interface 94 relays communication with the input terminal device 2, for example.

[0091] The hard disk 93 is an auxiliary storage device. The hard disk 93 realizes the function of the data storage unit 13 illustrated in Fig. 1. The hard disk 93 also stores various programs including programs for realizing the functions of the data acquisition unit 11, the inference unit 12, the feature extraction unit 14, the attack possibility determination unit 15, the confidence difference determination unit 16, the detection unit 17, and the notification unit 18 illustrated in Fig. 1.

[0092] The memory 92 is a main storage device and may be, for example, a dynamic random access memory (DRAM).

[0093] The CPU 91 reads various programs from the hard disk 93, expands them into the memory 92, and executes them. As a result, the CPU 91 realizes the functions of the data acquisition unit 11, the inference unit 12, the feature extraction unit 14, the attack possibility determination unit 15, the confidence difference determination unit 16, the detection unit 17, and the notification unit 18 illustrated in FIG. [Explanation of symbols]

[0094] 1. Information processing equipment 2 Input terminal device 11 Data Acquisition Section 12 Reasoning part 13 Data storage section 14 Feature extraction unit 15 Attack possibility determination section 16 Confidence difference determination unit 17 Detection unit 18. Information Department

Claims

1. For a plurality of queries requesting inference to be performed on individual input data, performing inference on each of the input data using the classification model; obtaining a confidence level for each of the queries for each of the classes into which the input data is classified by the inference; Calculating a difference in the confidence between the first class and the second class for each query; detecting an attack candidate state in which the difference in the certainty level decreases by a certain ratio or more, and determining that there is a first possibility of an attack when the attack candidate state is detected consecutively for a number of times equal to or greater than a threshold value; When the difference in the confidence level is equal to or less than a confidence level difference threshold, it is determined that there is a second possibility of an attack; When it is determined that there is the first possibility and also determined that there is the second possibility, it is evaluated that there is a possibility that the plurality of queries are an attack. An evaluation method characterized by causing a computer to execute processing.

2. For a plurality of queries requesting inference to be performed on individual input data, performing inference on each of the input data using the classification model; obtaining a confidence level for each of the queries for each of the classes into which the input data is classified by the inference; Calculating a difference in the confidence between the first class and the second class for each query; detecting an attack candidate state in which the difference in the certainty level decreases by a certain ratio or more, and determining that there is a first possibility of an attack when the attack candidate state is detected consecutively for a number of times equal to or greater than a threshold value; detecting an attack when the order of the certainty factors of the first class and the second class is reversed; determining that there is a first possibility of the attack and, if the attack is detected, evaluating the plurality of queries as an attack; An evaluation method characterized by causing a computer to execute processing.

3. The evaluation method according to claim 1 , wherein the plurality of queries are evaluated based on a tendency of increase or decrease in the difference in certainty between the class with the highest certainty and the class with the second highest certainty.

4. performing linear regression using the difference in confidence for each of the plurality of input data and the input time to obtain a regression line; When the time at which the difference in the certainty level disappears on the regression line falls between a representative time related to the input time and a time threshold, the attack candidate state is detected.

3. The evaluation method according to claim 1 or 2.

5. For a plurality of queries requesting inference to be performed on individual input data, performing inference on each of the input data using the classification model; obtaining a confidence level for each of the queries for each of the classes into which the input data is classified by the inference; Calculating a difference in the confidence between the first class and the second class for each query; detecting an attack candidate state in which the difference in the certainty level decreases by a certain ratio or more, and determining that there is a first possibility of an attack when the attack candidate state is detected consecutively for a number of times equal to or greater than a threshold value; When the difference in the confidence level is equal to or less than a confidence level difference threshold, it is determined that there is a second possibility of an attack; When it is determined that there is the first possibility and also determined that there is the second possibility, it is evaluated that there is a possibility that the plurality of queries are an attack. An evaluation program that causes a computer to execute a process.

6. For a plurality of queries requesting inference to be performed on individual input data, performing inference on each of the input data using the classification model; obtaining a confidence level for each of the queries for each of the classes into which the input data is classified by the inference; Calculating a difference in the confidence between the first class and the second class for each query; detecting an attack candidate state in which the difference in the certainty level decreases by a certain ratio or more, and determining that there is a first possibility of an attack when the attack candidate state is detected consecutively for a number of times equal to or greater than a threshold value; detecting an attack when the order of the certainty factors of the first class and the second class is reversed; determining that there is a first possibility of the attack and, if the attack is detected, evaluating the plurality of queries as an attack; An evaluation program that causes a computer to execute a process.

7. an inference unit that performs inference on each of a plurality of queries that request execution of inference on individual input data using a classification model, and obtains a confidence level for each class into which the input data is classified for each of the queries through the inference; a feature calculation unit that calculates a difference in the confidence between the first class and the second class for each query; an evaluation unit that detects an attack candidate state in which the difference in the certainty decreases at a certain rate or more, determines that there is a first possibility of an attack when the attack candidate state is detected consecutively a number of times or more than a threshold, determines that there is a second possibility of an attack when the difference in the certainty becomes equal to or less than a threshold, and evaluates that there is a possibility that the multiple queries are an attack when it has determined that there is the first possibility and that there is the second possibility; An information processing device comprising:

Images