GENERATION METHOD, GENERATION PROGRAM, AND INFORMATION PROCESSING APPARATUS
The method enhances machine learning system security by analyzing attack trees to generate a priority list of specification changes, addressing the inefficiencies of existing attack tree analysis methods in machine learning systems.
Patent Information
- Application Number
- JP2023572306
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-01-07
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-01-07
AI Technical Summary
Existing attack tree analysis methods for machine learning systems are ineffective in prioritizing and implementing appropriate specification changes due to complex nested structures and overlapping subtrees, making it difficult to enhance security against various attacks.
A method and device that associate attack nodes with conditions and countermeasures, using tree structure information, difficulty levels, and condition matching status to generate a priority list of specification changes that can prevent attacks by analyzing and prioritizing conditions that must be satisfied or unsatisfied.
Improves the security of machine learning systems by efficiently identifying and implementing effective countermeasures against attacks through a systematic prioritization of specification changes based on attack tree analysis.
Smart Images

Figure 0007593508000001 
Figure 0007593508000002 
Figure 0007593508000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a generation method, a generation program, and an information processing device. [Background technology]
[0002] There are known technologies that use artificial intelligence (AI) to estimate specific information and recognize various objects based on given data. AI realized by machine learning in particular has attracted a great deal of attention. In the following, a system that performs estimation or recognition using a learning model generated by machine learning is called a machine learning system. There is a risk that AI realized by such machine learning systems will be subject to various attacks.
[0003] For example, there is an attack called Adversarial Example. This is an attack that intentionally makes the AI make a wrong inference by adding cleverly calculated noise to the original image, creating an image that is recognized by humans as the same object as the original image, but is recognized as something else by the machine learning system. For example, by adding noise to an image of a panda, it is possible to generate an image that looks like a panda to humans, but is determined to be a gibbon when classified by a machine learning system. There are various other methods of attacking machine learning systems.
[0004] As such, machine learning systems are exposed to many risks from various attacks. Therefore, when developing a machine learning system, it is important to perform a security analysis to determine what types of attacks are applicable to the machine learning system and consider how to respond.
[0005] There are two main ways to respond to attacks in machine learning systems: changing the specifications and applying attack-specific countermeasures to the machine learning system. Changing the specifications involves changing the specifications of the machine learning system to make attacks impossible, since attacks on machine learning systems are closely related to the specifications. Meanwhile, applying attack-specific countermeasures to machine learning systems includes retraining the learning model to make it less likely for attacks to succeed, and implementing attack detection methods to mitigate damage. Here, we consider countermeasures against adversarial attacks by changing the specifications of the machine learning system.
[0006] Here, there is a method known as attack tree analysis as a general security analysis method in IT (Internet Technology) security. Attack tree analysis is performed in the following steps. A tree is constructed with the top node representing the possible damage to the system being attacked, and branches downward from there. For each node, branches downward are set by considering the conditions for that node to be valid and setting branches and leaves, generating an attack tree. Once the branches and leaves have been determined, the conditions for the attack tree to no longer be valid can be identified, and the specifications of the system can be modified so that the attack tree no longer be valid. This makes it possible to generate a system that is resistant to attacks that will cause the expected damage.
[0007] In a typical attack tree, the structure is not determined at first, and the information of each node and branch is set after the specifications are decided. In contrast, the types of attacks and damages on machine learning systems are limited. Therefore, when performing attack tree analysis on a machine learning system, it is possible to generate an attack tree in which the information of each node and branch is registered before the specifications of the machine learning system are determined. Therefore, in a machine learning system, an attack tree is generated in advance, and the conditions registered in each node are checked against the specifications to determine whether each attack is valid or not.
[0008] As a technology related to system safety, there is a technology that extracts combinations of attack activities to be dealt with from information on attack paths that indicate the steps of an attack, calculates an evaluation value based on the usefulness of countermeasure candidates for each attack activity and the importance of the combination of attack activities, and determines countermeasures for each combination of attack activities. There is also a technology that generates information indicating the accessibility of a trained model from predetermined information related to a machine learning system, and determines the possibility of an attack based on the generated information and information indicating the operability of a query or a trained model. There is also a technology that generates an attack tree, assigns the probability of an attack succeeding to each node, and calculates and provides the final probability of an attack. There is also a technology that generates an attack tree, calculates an attack score based on the impact of the attack, and ranks the attacks according to the score. [Prior art documents] [Patent documents]
[0009] [Patent Document 1] International Publication No. 2018 / 134909 [Patent Document 2] Patent Publication No. 2021-60872 [Patent Document 3] US Patent Application Publication No. 2009 / 0113549 [Patent Document 4] U.S. Pat. No. 1,074,9890 Summary of the Invention [Problem to be solved by the invention]
[0010] However, in an attack tree for a machine learning system, when an attack tree is constructed for each attack, there are often overlapping subtrees that represent small hierarchies composed of branches and leaves, resulting in a nested structure, such as an attack tree for one attack existing inside an attack tree for another attack. In addition, there are many cases where nodes with the same content exist in multiple subtrees. Furthermore, it may be difficult to make a specific node unsatisfied as a specification. For example, if there is a node whose content is to output a result to a user, it is difficult to make this node unsatisfied, since making this node unsatisfied would make the system meaningless. Since attack trees for machine learning systems have the above complex characteristics, it is difficult to prioritize and present which conditions should be unsatisfied using general attack tree analysis techniques. Therefore, it is difficult to determine appropriate specification changes as countermeasures against attacks, and it is difficult to improve the security of machine learning systems.
[0011] In addition, while it is possible to obtain an evaluation of attacks using the above-mentioned technology, it is difficult to prioritize and present conditions that will invalidate the attack tree as a countermeasure against attacks. Therefore, it is difficult to determine appropriate specification changes as a countermeasure against attacks, and it is difficult to improve the security of machine learning systems.
[0012] The disclosed technology has been made in consideration of the above, and aims to provide a generation method, a generation program, and an information processing device that improve the security of machine learning systems. [Means for solving the problem]
[0013] In one aspect of the generation method, generation program, and information processing device disclosed in the present application, tree structure information representing the structure of an attack tree including a first node associated with information indicating a successful attack and a plurality of second nodes hierarchically connected to the first node and associated with conditions for making the attack successful, difficulty level information indicating the difficulty of taking measures to prevent the conditions associated with each of the plurality of second nodes from being satisfied, and condition matching status information indicating whether the plurality of second nodes are satisfied, are received, and based on the tree structure information, difficulty level information, and condition matching status information, information to be provided indicating the priority of adopting each of the unsatisfied condition candidates, which are combinations of conditions associated with each of the plurality of second nodes that result in an unsuccessful attack. Effect of the Invention
[0014] In one aspect, the present invention can improve the security of machine learning systems. [Brief description of the drawings]
[0015] [Figure 1] FIG. 1 is a block diagram of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of an attack tree. [Diagram 3] FIG. 3 is a diagram illustrating an example of tree structure information. [Figure 4] FIG. 4 is a diagram illustrating an example of the condition match status information. [Diagram 5] FIG. 5 is a diagram illustrating an example of the difficulty level information. [Figure 6] FIG. 6 is a diagram showing an example of a prioritized specification change candidate list. [Figure 7] FIG. 7 is a diagram showing a specific example of an attack tree. [Figure 8] FIG. 8 is a diagram illustrating an example of the input information. [Figure 9] FIG. 9 is a diagram showing the development of a logical formula that denies the highest level attack achievement condition for the Adversarial Example condition. [Figure 10]FIG. 10 is a diagram for explaining decomposition of a logical formula. [Figure 11] FIG. 11 is a diagram for explaining the omission of redundant expressions in a logical formula. [Figure 12] FIG. 12 is a diagram for explaining extraction of unmet condition candidates for the Poisoning condition. [Figure 13] FIG. 13 is a diagram for explaining the integration of unmet condition candidates. [Figure 14] FIG. 14 is a diagram for explaining the development of the unmet condition candidates after integration. [Figure 15] FIG. 15 is a diagram showing final candidates of unmet conditions. [Figure 16] FIG. 16 is a diagram in which condition matching status information is added to an attack tree. [Figure 17] FIG. 17 is a diagram showing the calculated updated scores. [Figure 18] FIG. 18 is a diagram for explaining the calculation of the priority determination score for each unmet condition candidate. [Figure 19] FIG. 19 is a diagram showing a specific example of a prioritized specification change candidate list. [Figure 20] FIG. 20 is a flowchart of a process of generating a prioritized specification change candidate list by the information processing device according to the embodiment. [Figure 21] FIG. 21 is a diagram illustrating a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, embodiments of the generating method, generating program, and information processing device disclosed in the present application will be described in detail with reference to the drawings. Note that the generating method, generating program, and information processing device disclosed in the present application are not limited to the following embodiments. EXAMPLES
[0017] 1 is a block diagram of an information processing device according to the present embodiment. The information processing device 1 includes a logical analysis unit 11, an extraction unit 12, a ranking unit 13, an output unit 14, and a storage unit 15.
[0018] The storage unit 15 stores the tree structure information 101, the established tree group 102, and the unestablished condition candidate group 103. The storage unit 15 is available to each of the logical analysis unit 11, the extraction unit 12, and the ranking unit 13.
[0019] Tree structure information 101 is information that represents the tree structure of an attack tree for a machine learning system whose specifications are to be changed to prevent attacks. Fig. 2 is a diagram showing an example of an attack tree. The attack tree shown in Fig. 2 is an attack tree when the machine learning system is attacked by attack scenario A1.
[0020] Hereinafter, one attack tree may be referred to as a "tree." The attack tree in FIG. 2 has a plurality of second nodes in which the conditions #2B, #3, #4B, #6B, and #7A for achieving the attack by the attack scenario A1 shown in the top first node are written. The attack tree also has nodes whose contents are the achievement of other attack scenarios included in the attack tree. Hereinafter, the second nodes in which the conditions for achieving the attack described in the top first node in the attack tree are written and the nodes whose contents are the achievement of other attack scenarios included in the attack tree are called "leaves." A logical symbol expressing the relationship between each condition is called a "branch." The second nodes, which are leaves, are hierarchically connected to the first node by the branch. A branch extending downward from a node whose contents are the achievement of a specific attack scenario included in the attack tree and the leaves are called a "subtree." In other words, a subtree can be considered as an attack tree for a specific attack described at the top of the subtree.
[0021] The tree structure information 101 is a logical formula that links the conditions described in each leaf using logical conditions indicated by branches. For example, the tree structure information 101 representing the attack tree in FIG. 2 is expressed as the logical formula shown in FIG. 3 when each condition is represented by a code. FIG. 3 is a diagram showing an example of tree structure information. Since an attack tree for a machine learning system can be created in advance, the tree structure information 101 is created in advance, input to the information processing device 1, and stored in the storage unit 15. Since the number of attack trees may be one or more, one or more pieces of tree structure information 101 are created according to the number of attack trees and stored in the storage unit 15.
[0022] The established tree group 102 is a collection of tree structure information 101 of attack trees in which the highest attack among the attack trees is established. The established tree group 102 is generated by the logic analysis unit 11 and stored in the storage unit 15.
[0023] The candidate unsatisfied condition group 103 is a set of candidate unsatisfied condition candidates that are candidates for a combination of conditions that are not satisfied in order to prevent the top-level attack in the attack tree from being achieved. The candidate unsatisfied condition group 103 is generated by the extraction unit 12 and stored in the storage unit 15.
[0024] The logical analysis unit 11 acquires the tree structure information 101 of each attack tree from the storage unit 15. Furthermore, the logical analysis unit 11 acquires the condition matching status information 22 inputted through the input terminal 2.
[0025] Fig. 4 is a diagram showing an example of condition matching status information. For example, the condition matching status information 22 for the attack tree in Fig. 2 stores information indicating whether or not each of conditions #2B, #3, #4B, #6B, and #7A is satisfied in the specifications of the current machine learning system, as shown in Fig. 4. The condition matching status information 22 in Fig. 4 indicates that conditions #2B, #4B, and #6B are satisfied, and conditions #3 and #7A are not satisfied.
[0026] Returning to FIG. 1, the explanation will be continued. The logical analysis unit 11 judges the establishment status of the attack tree, which indicates whether or not the attack shown in the topmost first node in each attack tree is accomplished and the attack tree is established, from the tree structure information 101 of each attack tree and the condition matching status information 22. For example, when the condition matching status information 22 shown in FIG. 4 is input for the attack tree in FIG. 2, the logical analysis unit 11 judges that the attack tree shown in FIG. 2 is established because the condition #6B is satisfied and the condition #7A is satisfied. Then, the logical analysis unit 11 extracts the tree structure information 101 in which the attack tree is established, and stores them together as the established tree group 102 in the storage unit 15. After that, the logical analysis unit 11 notifies the extraction unit 12 of the completion of generation of the established tree group 102.
[0027] The extraction unit 12 receives a notification of the completion of the generation of the established tree group 102 from the logical analysis unit 11. Then, the extraction unit 12 acquires each tree structure information 101 included in the established tree group 102 from the storage unit 15. Next, the extraction unit 12 generates a logical formula representing a condition for making the attack tree represented by the tree structure information 101 invalid from the acquired tree structure information 101. Next, the extraction unit 12 breaks down the logical formula into individual logical formulas including only AND, and generates unvalidated condition candidates that are candidates for the minimum unit of conditions for making the attack tree invalid. The unvalidated condition candidates are conditions that, if any one of them is satisfied, make the original attack tree invalid. When there are multiple attack trees, the extraction unit 12 integrates the unvalidated condition candidates of each attack tree to generate unvalidated condition candidates for making all attack trees invalid. After that, the extraction unit 12 stores the unvalidated condition candidate group 103 including the generated unvalidated condition candidates in the storage unit 15. Furthermore, the extraction unit 12 notifies the ranking unit 13 of the completion of extraction of the non-met condition candidate group 103.
[0028] For example, when the tree structure information 101 shown in FIG. 3 is used, the extraction unit 12 generates a logical expression that negates the entire logical expression of the tree structure information 101 shown in FIG. 3. Then, the extraction unit 12 generates ((~#6B)&((~#2B)|(~3)))|((~#4B)&(~7A)) from the generated logical expression as a logical expression representing a condition that makes the attack tree shown in FIG. 2 invalid. Here, "~" represents the negation (NOT) of a condition. Furthermore, the extraction unit 12 breaks down this logical expression into the smallest unit logical expressions connected by AND, and generates three, ~#6B and ~#2B, ~#6B and ~#3, and ~#4B and ~7A. The extraction unit 12 sets these three logical expressions as invalid condition candidates. In other words, if any of these three invalid condition candidates is satisfied, the attack tree shown in FIG. 2 is invalid.
[0029] 1, the prioritization unit 13 receives a notification from the extraction unit 12 that extraction of the non-met condition candidate group 103 has been completed. Next, the prioritization unit 13 acquires the difficulty level information 21 input from the input terminal 2.
[0030] FIG. 5 is a diagram showing an example of difficulty level information. The difficulty level information 21 is information that indicates, for each condition, how difficult it is to not satisfy the condition written on each leaf. In this embodiment, the difficulty level is expressed on a 10-point scale, with the higher the number, the more difficult it is. For example, in the difficulty level information shown in FIG. 5, conditions #3 and #7A are very difficult to not satisfy. In contrast, condition #2B is relatively easy to not satisfy. Hereinafter, the difficulty level value for each condition will be referred to as a difficulty parameter.
[0031] Continuing the explanation by returning to Fig. 1, the prioritization unit 13 next acquires the establishment tree group 102 from the storage unit 15. Furthermore, the prioritization unit 13 acquires the condition conformance status information 22 input from the input terminal 2.
[0032] Next, the ranking unit 13 develops all attack trees included in the group of established trees 102, and counts the number of leaves in which the same condition is described and the condition is satisfied, excluding subtrees that are not established. In other words, the ranking unit 13 does not count leaves of attack trees and subtrees that are not established, and leaves in established attack trees that do not satisfy the condition. Hereinafter, the number of leaves in which the same condition is described and the condition is satisfied will be referred to as the "number of identical leaves."
[0033] Next, the ranking unit 13 stores the value obtained by dividing 1 by the number of identical leaves as the basic score of the condition described in those leaves for each condition. For example, if there are three leaves in which the same condition is described and the condition is satisfied in the established attack tree, the ranking unit 13 calculates the basic score of that condition as 1 / 3. In this way, the ranking unit 13 can lower the basic score of the condition the more overlapping the conditions are.
[0034] Next, the ranking unit 13 multiplies the basic score by the difficulty parameter for each condition to calculate an updated score for each condition. That is, the more the number of identical leaves for that condition, the lower the updated score, and the more difficult it is to not satisfy that condition, the higher the updated score. For example, if the number of identical leaves for a certain condition is 5 and the difficulty parameter is 4, the ranking unit 13 calculates the updated score for that condition as (1 / 5)×4=4 / 5. Also, if the number of identical leaves for another condition is 2 and the difficulty parameter is 7, the ranking unit 13 calculates the updated score for that condition as (1 / 2)×7=7 / 2. The ranking unit 13 calculates the updated scores for all conditions included in the attack tree that is satisfied.
[0035] Next, the ranking unit 13 acquires the unsatisfied condition candidate group 103 from the storage unit 15. Then, the ranking unit 13 calculates a priority determination score for each unsatisfied condition candidate using the updated score of each condition included in the candidate. At this time, the ranking unit 13 uses the calculated updated score of each condition as a score for the negation of each condition. In addition, the ranking unit 13 calculates the priority determination score of the unsatisfied condition candidate by adding the updated score for "AND". For example, the unsatisfied condition candidate is NOT (condition P) and NOT (condition Q) and NOT (condition R), and the updated scores of condition P, condition Q, and condition R are 2, 4 / 5, and 4, respectively. In this case, the ranking unit 13 calculates the priority determination score of the unsatisfied condition candidate as 2+4 / 5+4=6.8. At this time, the ranking unit 13 sets the updated score of the condition that has not already been satisfied to 0.
[0036] Thereafter, the ranking unit 13 sorts the non-established condition candidates in ascending order of the priority determination scores to arrange them in order of priority. Then, the ranking unit 13 outputs information on the non-established condition candidates sorted in order of priority to the output unit 14.
[0037] The output unit 14 acquires an input of information on the unmet condition candidates arranged in order of priority from the ranking unit 13. Then, the output unit 14 generates and outputs a prioritized specification change candidate list 100 shown in Fig. 6 by adding priorities to the unmet condition candidates in the arranged order. Fig. 6 is a diagram showing an example of the prioritized specification change candidate list. For example, the output unit 14 displays the prioritized specification change candidate list 100 on a monitor or the like, and presents to the user which conditions should be unmet with priorities.
[0038] Next, the calculation process of the priority determination score by the information processing device 1 according to the present embodiment will be described in detail with reference to a specific example. Fig. 7 is a diagram showing a specific example of an attack tree. Here, a case where two attack trees 201 and 202 exist will be described.
[0039] Attack tree 201 represents conditions for an attack by Adversarial Example / Evasion. Attack tree 201 has eight leaves in which conditions #2A, #2B, #3, #4B, #6B, or #7A are described. Attack tree 202 represents conditions for an attack by Poisoning. Attack tree 202 has six leaves in which conditions #1, #2B, #3, #4B, #6B, and #7A are described.
[0040] Condition #1 is a condition that "the attacker can execute the learning process." Condition #2A is a condition that "the attacker can execute the estimation process at least once." Condition #2B is a condition that "the attacker can execute the estimation process approximately a thousand times." Condition #3 is a condition that "the attacker can obtain the output results." Condition #4B is a condition that "the attacker can obtain one or more pieces of data from the system." Condition #6B is a condition that "the attacker can obtain a trained model." Condition #7A is a condition that "the attacker can obtain one or more pieces of similar data."
[0041] 8 is a diagram showing an example of input information. The information processing device 1 acquires tree structure information 101, difficulty level information 21, and condition matching status information 22 shown in FIG.
[0042] In this case, the tree structure information 101 includes a logical expression of an Adversarial Example condition that represents the structure of an attack tree 201 and a logical expression of a Poisoning condition that represents the structure of an attack tree 202, as shown in FIG.
[0043] Furthermore, the difficulty information 21 includes the difficulty parameters of conditions #1, #2A, #2B, #3, #4B, #6B, and #7A described in the leaves of the attack trees 201 and 202. The difficulty parameter of condition #1 is 10. The difficulty parameter of condition #2A is 5. The difficulty parameter of condition #2B is 5. The difficulty parameter of condition #3 is 10. The difficulty parameter of condition #4B is 4. The difficulty parameter of condition #6B is 3. The difficulty parameter of condition #7A is 8.
[0044] Furthermore, the condition fulfillment status information 22 includes the condition fulfillment status of conditions #1, #2A, #2B, #3, #4B, #6B, and #7A described in the leaves of the attack trees 201 and 202. Condition #1 is fulfilled. Condition #2A is fulfilled. Condition #2B is fulfilled. Condition #3 is fulfilled. Condition #4B is fulfilled. Condition #6B is not fulfilled. Condition #7A is fulfilled.
[0045] Using the tree structure information 101 and the condition matching status information 22, the logical analysis unit 11 determines that both attack trees 201 and 202 are valid, and stores the tree structure information 101 shown in Figure 2, which includes both tree structures, as a group of valid trees 102.
[0046] The extraction unit 12 acquires the logical formula of the Adversarial Example condition from among the two logical formulas registered in the establishment tree group 102. FIG. 9 is a diagram showing the expansion of a logical formula that denies the highest-level attack establishment condition for the Adversarial Example condition. The extraction unit 12 generates logical formula 211 shown in FIG. 9 that denies the entire acquired logical formula. Next, the extraction unit 12 generates logical formula 212 that represents the condition for making logical formula 211 true. Next, the extraction unit 12 expands logical formula 212 to generate logical formula 213. The extraction unit 12 further expands logical formula 213 to generate logical formula 214. This completes the expansion of the logical formula that denies the Adversarial Example condition.
[0047] Fig. 10 is a diagram for explaining the decomposition of a logical formula. Next, the extraction unit 12 decomposes the expanded logical formula 214 of Fig. 9 by dividing it by OR, and sets each logical formula as shown in the logical formula group 215 of Fig. 10. Here, when the same condition is included in logical formulas connected by AND, the extraction unit 12 groups the conditions into one. Next, the extraction unit 12 groups the same logical formulas among the decomposed logical formulas into one, and sets it as the logical formula group 216. This completes the decomposition of the expanded logical formula.
[0048] 11 is a diagram for explaining the omission of redundant expressions in logical expressions. Next, as shown in the logical expression group 217, when the same conditional expression is followed by AND, it is considered to be a redundant condition, and the extraction unit 12 consolidates it into a logical expression from which the condition following AND is removed.
[0049] For example, in the three logical expressions 219 in the logical expression group 217, (NOT(#6B)) AND (NOT(#2B)) are the same. In this case, if (NOT(#6B)) AND (NOT(#2B)) holds true, the original logical expression is negated regardless of whether the condition connected by the latter AND holds true or not. That is, all three logical expressions 219 are included in the logical expression (NOT(#6B)) AND (NOT(#2B)). Therefore, the extraction unit 12 consolidates the three logical expressions 219 into the logical expression (NOT(#6B)) AND (NOT(#2B)). In this way, the extraction unit 12 omits redundant expressions in the logical expression group 216 in FIG. 10 and generates the logical expression group 218.
[0050] The conditions indicated by the four logical expressions included in this logical expression group 218 are candidates for the unsatisfied condition for the Adversarial Example condition. In other words, by making any one of the four logical expressions included in this logical expression group 218 be satisfied, an attack by Adversarial Example / Evasion can be avoided.
[0051] Next, the extraction unit 12 acquires the logical formula of the poisoning condition from among the two logical formulas registered in the establishment tree group 102. FIG. 12 is a diagram for explaining the extraction of an unestablished condition candidate for the poisoning condition. The extraction unit 12 generates a logical formula 221 shown in FIG. 12 that negates the entire acquired logical formula. Next, the extraction unit 12 generates a logical formula 222 for establishing the logical formula 221. Next, the extraction unit 12 expands the logical formula 222 to generate a logical formula 223. Next, the extraction unit 12 generates a logical formula group 224 by decomposing the logical formula 223 and omitting any redundant expressions.
[0052] The conditions indicated by the four logical expressions included in this logical expression group 224 are candidates for the non-established condition for the poisoning condition. In other words, by establishing any one of the four logical expressions included in this logical expression group 224, it is possible to avoid an attack by poisoning.
[0053] Furthermore, the extraction unit 12 integrates the unsatisfied condition candidates for the Adversarial Example condition indicated by the logical formula group 218 and the unsatisfied condition candidates for the Poisoning condition indicated by the logical formula group 224. Fig. 13 is a diagram for explaining the integration of the unsatisfied condition candidates. The extraction unit 12 integrates the unsatisfied condition candidates by connecting the logical formula group 218 and the logical formula group 224 with AND as indicated by the logical formula group 231 in Fig. 13.
[0054] Fig. 14 is a diagram for explaining the expansion of the unsatisfied condition candidates after integration. Next, the extraction unit 12 generates all combinations of each logical expression in the logical expression group 218 and the logical expression group 224 in the logical expression shown in Fig. 13, and connects the logical expressions in each combination with AND, thereby executing the expansion of the integrated unsatisfied condition candidates. In this way, the extraction unit 12 generates the logical expression group 232 shown in Fig. 14. At this time, the extraction unit 12 combines logical expressions indicating the same condition into one, as shown in logical expression 233. Furthermore, the extraction unit 12 executes the omission of redundant expressions.
[0055] FIG. 15 is a diagram showing the final unsatisfied condition candidates. By performing the above processing, the extraction unit 12 calculates the logical formula group 234 shown in FIG. 15. The conditions indicated by the four logical formulas included in this logical formula group 234 are unsatisfied condition candidates for the Adversarial Example condition and the Poisoning condition. In other words, by making any one of the four logical formulas included in this logical formula group 234 be satisfied, both the Adversarial Example attack and the Poisoning attack can be avoided. The extraction unit 12 stores the four logical formulas included in this logical formula group 234 in the storage unit 15 as the unsatisfied condition candidate group 103.
[0056] The prioritization unit 13 combines the tree structure and the condition matching status information contained in the establishment tree group 102. Fig. 16 is a diagram in which the condition matching status information is added to the attack trees. In this case, both attack trees 201 and 202 are established. Therefore, when the condition matching status of each leaf and branch of the attack trees 201 and 202 is added, the attack trees 201 and 202 are expressed as shown in Fig. 16. In Fig. 16, if there is a leaf above a branch, the condition matching status is added to that leaf.
[0057] Since the subtree 240 of the attack tree 201 is not established, the ranking unit 13 removes it from the count of the number of identical leaves. Then, the ranking unit 13 counts the number of identical leaves in the attack trees 201 and 202. Here, the ranking unit 13 counts the number of leaves for condition #1 as 1, the number of leaves for condition #2A as 0, the number of leaves for condition #2B as 2, the number of leaves for condition #3 as 2, the number of leaves for condition #4B as 2, the number of leaves for condition #6B as 0, and the number of leaves for condition #7A as 2. Next, the ranking unit 13 calculates the basic score for each condition from the number of identical leaves counted. Here, the ranking unit 13 counts the basic score for condition #1 as 1, the basic score for condition #2A as 0, the basic score for condition #2B as 1 / 2, the basic score for condition #3 as 1 / 2, the basic score for condition #4B as 1 / 2, the basic score for condition #6B as 0, and the basic score for condition #7A as 1 / 2.
[0058] Then, the ranking unit 13 multiplies each difficulty parameter included in the difficulty level information 21 by the basic score to calculate the updated score for each condition. FIG. 17 is a diagram showing the calculated updated scores. Here, the ranking unit 13 calculates the updated score for each condition as shown in updated score 241. The updated score for condition #1 is 10. The updated score for condition #2A is 0. The updated score for condition #2B is 5 / 2. The updated score for condition #3 is 5. The updated score for condition #4B is 2. The updated score for condition #6B is 0. The updated score for condition #7A is 4.
[0059] FIG. 18 is a diagram for explaining the calculation of the priority determination score of each unmet condition candidate. Next, the ranking unit 13 calculates the priority determination score of each unmet condition candidate, as shown in the priority determination score 242 of FIG. 18, using the post-update score of the condition calculated for each unmet condition candidate included in the unmet condition candidate group 103 represented by the logical expression group 234 of FIG. 15. For example, the unmet condition candidate ((NOT(#6B)AND(NOT(#2B))) will be explained as an example. In this case, since the post-update score of NOT(#6B) is 0 and the post-update score of NOT(#2B) is 5 / 2, the ranking unit 13 calculates the priority determination score as 0+2 / 5=5 / 2. Here, the ranking unit 13 uses the post-update score of each condition already calculated as the negated value of each condition.
[0060] The ranking unit 13 sorts the unmet condition candidates indicated by the logical expression group 234 in Fig. 15 in ascending order of the calculated priority determination scores. Then, the ranking unit 13 outputs information on the sorted unmet condition candidates to the output unit 14.
[0061] 19 is a diagram showing a specific example of a prioritized specification change candidate list. The output unit 14 generates a prioritized specification change candidate list 250 in which each unmet condition candidate sorted according to priority is ranked. Here, the output unit 14 expresses priority as a rank. The output unit 14 also generates the prioritized specification change candidate list 250 by adding a message encouraging a change in specifications to each unmet condition candidate. The output unit 14 provides the generated prioritized specification change candidate list 250 to a user.
[0062] 20 is a flowchart of a process for generating a prioritized specification change candidate list by the information processing device according to the embodiment. Next, a flow of a process for generating a prioritized specification change candidate list by the information processing device 1 will be described with reference to FIG.
[0063] The information processing device 1 acquires tree structure information 101 in advance and stores it in the storage unit 15. The information processing device 1 also acquires the difficulty level information 21 and the condition matching status information 22 input from the input terminal 2 (step S1).
[0064] Next, the logical analysis unit 11 identifies attack trees that are established using the tree structure information 101 and the condition matching status information 22, and generates a group of established trees 102 including the tree structure information 101 of each of the established attack trees (step S2). The logical analysis unit 11 stores the group of established trees 102 in the storage unit 15.
[0065] Next, the extraction unit 12 generates an unfounded condition candidate group 103 including each unfounded condition candidate for making a found attack tree unfounded by using the tree structure information 101 included in the found tree group 102 (step S3). The extraction unit 12 stores the unfounded condition candidate group 103 in the storage unit 15.
[0066] The ranking unit 13 acquires the tree structure information 101 included in the establishment tree group 102. Then, the ranking unit 13 counts the number of identical leaves in the tree structure information 101 and calculates a standard score for each condition (step S4).
[0067] Next, the ranking unit 13 acquires the difficulty parameters of each condition registered in the difficulty level information 21, and calculates an updated score for each condition using the difficulty parameters and the reference score (step S5).
[0068] Next, the ranking unit 13 acquires the unmet condition candidate group 103 and calculates the priority determination score for each unmet condition candidate using the updated score of each condition (step S6). After that, the ranking unit 13 sorts the unmet condition candidates in ascending order from the lowest priority determination score to assign priorities from the top. Then, the ranking unit 13 outputs information on the unmet condition candidates sorted in ascending order from the lowest priority determination score to the output unit 14.
[0069] The output unit 14 receives an input of information on the unmet condition candidates sorted in ascending order from the lowest priority determination score from the ranking unit 13. Then, the output unit 14 adds priorities to the unmet condition candidates in the sorted order, and creates a prioritized specification change candidate list shown in Fig. 6. Thereafter, the output unit 14 outputs the prioritized specification change candidate list and provides it to the user (step S7).
[0070] In this embodiment, the difficulty parameter is different for each condition, but the difficulty parameter may be the same for each condition. In this case, the ranking unit 13 can rank the unmet condition candidates based on the basic score without using the updated score.
[0071] In addition, although the difficulty information 21 has been described as being input from the input terminal 2, the information processing device 1 may acquire the difficulty information 21 in advance and store it in the memory unit 15 as a calculation table, and use the calculation table to calculate the updated score.
[0072] Furthermore, when there is a large number of combinations of conditions that become candidates for the unmet condition in the unmet condition candidate group 103, the extraction unit 12 may evaluate the post-update score for each tree or subtree, and exclude those with large scores before combining. Since unmet condition candidates are extracted for each attack and finally combined and integrated with AND, the final integration increases the number of unmet condition candidates by a large amount. Therefore, the extraction unit 12 excludes trees or subtrees that clearly have large scores and low priorities before the final integration. This makes it possible to prevent the number of combinations of conditions that become candidates for the unmet condition from becoming large.
[0073] Furthermore, the ranking unit 13 may exclude unsatisfied condition candidates including conditions with a base score of 0 from the prioritized specification change candidate list to be finally output. Furthermore, the ranking unit 13 may evaluate the priority determination scores of the unsatisfied condition candidates by excluding conditions with a base score of 0. This is because the unsatisfied condition candidates may be simplified by excluding conditions with a base score of 0.
[0074] In addition, in this embodiment, the ranking unit 13 divides by the number of leaves when calculating the basic score, but other calculation methods may be used as long as the basic score can be calculated so that the fewer the number of leaves, the higher the score. Also, for the updated score, the ranking unit 13 may use a calculation method other than multiplication of the difficulty parameter and the basic score as long as the updated score can be calculated so that the score with the fewer the number of leaves and the score with the lower degree of difficulty are prioritized. Also, the derivation of the non-established condition candidates does not necessarily have to be transformed to be the minimum. In other words, the ranking unit 13 may calculate the priority determination score and create the specification change candidate list while keeping the redundant expressions.
[0075] As described above, the information processing device according to the present embodiment uses tree structure information of an attack tree assuming attacks against a machine learning system to extract candidates for unsatisfied conditions to prevent all attacks. Then, the information processing device calculates the score of each candidate for unsatisfied conditions using condition match status information and difficulty information representing the establishment or non-establishment of each leaf, so that the more overlapping there is, the higher the score is, and the easier it is to make it unsatisfied, and determines the priority of the candidates for unsatisfied conditions and notifies the user. The more overlapping there is, the easier it is to prevent attacks with a single change in specifications, and the easier it is to make specifications unsatisfied, the easier it is to change them, so that the user can grasp the change in specifications that makes it easier to take measures against attacks. Therefore, it is possible to efficiently select a response to an attack, and it is easy to prevent attacks against the machine learning system, and it is possible to improve the safety of the machine learning system.
[0076] (Hardware configuration) Fig. 21 is a hardware configuration diagram of an information processing device. The information processing device 1 described in the above embodiment can be realized with a hardware configuration as shown in Fig. 21. For example, the information processing device 1 has a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.
[0077] The network interface 94 is an interface for communication between the information processing device 1 and an external device. The network interface 94 relays communication between the CPU 91 and the input terminal 2, for example.
[0078] The hard disk 93 is an auxiliary storage device. The hard disk 93 realizes the function of the storage unit 15 illustrated in Fig. 1. The hard disk 93 also stores various programs including programs for realizing the functions of the logical analysis unit 11, the extraction unit 12, the ranking unit 13, and the output unit 14 illustrated in Fig. 1.
[0079] The memory 92 is a main storage device, and is, for example, a dynamic random access memory (DRAM).
[0080] The CPU 91 reads out various programs stored in the hard disk 93, expands them in the memory 92, and executes them. In this way, the CPU 91 realizes the functions of the logical analysis unit 11, the extraction unit 12, the ranking unit 13, and the output unit 14 illustrated in FIG. [Explanation of symbols]
[0081] 1. Information processing device 2 Input terminal 11 Logical Analysis Department 12 Extraction part 13 Ranking Division 14 Output section 15 Storage section 21 Difficulty level information 22 Condition matching status information 101 Tree structure information 102 Establishment Trees 103 Group of unsatisfied conditions candidates
Claims
1. receiving tree structure information representing a structure of an attack tree including a first node associated with information indicating a successful attack and a plurality of second nodes hierarchically connected to the first node and associated with conditions for the successful attack, difficulty level information indicating the difficulty of taking measures to prevent the conditions associated with each of the plurality of second nodes from being satisfied, and condition match status information indicating whether the conditions of the plurality of second nodes are satisfied; Based on the tree structure information, the difficulty level information, and the condition match status information, for each of the combinations of conditions associated with the second nodes that result in the attack being unsuccessful, information to be provided indicating a priority of adopting the unsuccessful condition candidate is generated. A generating method characterized by causing a computer to execute processing.
2. 2. The method according to claim 1, wherein the logical expression indicating the tree structure information is expanded and decomposed to generate individual logical expressions linked by logical products, and the individual logical expressions are set as the candidates for the unsatisfied condition.
3. The generation method described in claim 1, characterized in that the total number of second nodes in the attack tree to which the same condition is associated is calculated based on the tree structure information and the condition matching status information, and the priority of the candidate unmet condition is determined so that the priority is higher the larger the total number for each condition included and lower the difficulty registered in the difficulty information.
4. calculating an updated score so that the updated score is higher as the total number is larger and the updated score is lower as the difficulty registered in the difficulty level information is lower; A priority determination score for the unmet condition candidate is calculated based on the updated score; The priority of each of the unmet condition candidates is determined according to the priority determination score. The method of claim 3 .
5. receiving tree structure information representing a structure of an attack tree including a first node associated with information indicating a successful attack and a plurality of second nodes hierarchically connected to the first node and associated with conditions for the successful attack, difficulty level information indicating the difficulty of taking measures to prevent the conditions associated with each of the plurality of second nodes from being satisfied, and condition match status information indicating whether the conditions of the plurality of second nodes are satisfied; Based on the tree structure information, the difficulty level information, and the condition match status information, for each of the combinations of conditions associated with the second nodes that result in the attack being unsuccessful, information to be provided indicating a priority of adopting the unsuccessful condition candidate is generated. A generating program that causes a computer to execute a process.
6. a ranking unit that receives tree structure information representing a structure of an attack tree including a first node associated with information indicating an attack that will be successful and a plurality of second nodes hierarchically connected to the first node and associated with conditions for making the attack successful, difficulty level information indicating the difficulty of making the conditions associated with each of the plurality of second nodes unsatisfied, and condition match status information indicating whether the conditions associated with each of the plurality of second nodes are satisfied, and generates, based on the tree structure information, the difficulty level information, and the condition match status information, information to be provided indicating a priority of adopting each of the unsatisfied condition candidates that are combinations of the conditions associated with each of the plurality of second nodes that will result in the attack being unsatisfied. An information processing device comprising:
Citation Information
Patent Citations
Method and system for establishing automobile information security attack tree model
CN113810365A
Generation method, generation program, and information processing apparatus
JP2021060872A
Systems and methods for improving the ranking and prioritization of attack-related events
US10749890B1
System and method to analyze software systems against tampering
US20090113549A1
Method and system for determining policies, rules, and agent characteristics, for automating agents, and protection
US20200410399A1
