Risk calculation program, risk calculation method, and risk calculation device

Abstract

The present invention causes a computer to execute processing for: storing a weight for each attack condition, which is an element for establishing a predetermined attack on an AI system, the weight being calculated on the basis of the usage ratio of each attack condition and the existence ratio, in the specifications of a plurality of existing AI systems, of each of a plurality of specification elements which are included in the AI system specifications and are related to a condition for establishing the attack; receiving information on the specifications of an AI system for which a risk is to be determined; for each attack tree created in advance, identifying the attack condition establishment state on the basis of information about the specification elements extracted from the information on the specifications of the AI system for which a risk is to be determined; and on the basis of the weight for each attack condition and the establishment state of the identified attack condition, calculating a risk value for each attack tree of the AI system for which a risk is to be determined.

Description

Risk calculation program, risk calculation method, and risk calculation device

[0001] The present invention relates to a risk calculation program, a risk calculation method, and a risk calculation device.

[0002] Artificial intelligence (AI) is a well-known technology that can estimate specific information and recognize various objects based on given data. AI, which is realized through machine learning, has attracted particular attention. AI systems, which are expected to be used in many fields, are at risk of being subjected to various attacks.

[0003] For example, there is an attack called an Adversarial Example. This is an attack in which cleverly calculated noise is added to an original image to create an image that humans recognize as the same object as the original image, but that the AI ​​system recognizes as something else, intentionally causing the AI ​​to make an incorrect inference. For example, by adding noise to an image of a panda, it is possible to create an image that looks like a panda to humans, but that when classified by an AI system is determined to be a gibbon. There are various other methods of attacking AI systems.

[0004] As such, the risk of attacks on AI systems has been increasing in recent years. Therefore, there is a demand to increase the attack resistance of AI systems, but responding to all attacks would require a huge amount of work, making this unrealistic. Therefore, it is important to appropriately improve the resistance of AI systems in accordance with their system requirements. To this end, it is desirable to conduct a security analysis and evaluation during the development of an AI system to determine what types of attacks are applicable to that system, and then consider countermeasures by comparing the results with the system's specification requirements, etc.

[0005] One way to respond to attacks on AI systems is to change their specifications. Because attacks on AI systems are closely related to their specifications, changing the specifications is a security measure that changes the specifications of an AI system to make attacks more difficult.

[0006] Here, a method known as attack tree analysis is a common security analysis technique in IT (Internet Technology) security. Attack tree analysis is performed using the following procedure. A tree is constructed with the top node representing the possible damage to the system being attacked, and branches downward from there. For each node, branches are set downward by considering the conditions under which that node is valid, and an attack tree is generated by setting branches downward. Once the branches and leaves are determined, the conditions under which the attack tree does not hold can be identified, and the system specifications can be modified so that the attack tree does not hold. This makes it possible to create a system that is resistant to attacks that cause the expected damage.

[0007] In a typical attack tree, the structure is not determined at first, and information about each node and branch is set after the specifications are decided. In contrast, the types of attacks and damage that can be inflicted on an AI system are limited. Therefore, when performing attack tree analysis on an AI system, it is possible to generate an attack tree in which information about each node and branch is registered before the specifications of the AI ​​system are determined. Therefore, in an AI system, an attack tree is generated in advance, and the conditions registered at each node are checked against the specification information of the AI ​​system to determine whether each attack is successful.

[0008] Furthermore, a security analysis technique that takes human damage into consideration has been proposed by collecting vulnerability information on IoT (Internet of Technology) devices from the web, etc., and analyzing the probability of threats occurring from the perspective of the system provider based on the collected information. Furthermore, a technique has been proposed that selects features to be added by calculating the evaluation value of new features in an incident determination model from the similarity of current features and the estimated contribution rate of incident determination.

[0009] JP 2019-145053 A JP 2019-168796 A

[0010] Jun Yajima, Takanori Oikawa, Ikuya Morikawa, Fumitaka Kasahara, Maki Inui, and Nobukazu Yoshioka, Machine Learning Security Threat Analysis Technology for Development Engineers, 2022 Symposium on Cryptography and Information Security (SCIS2022), Jan. 18-21, 2022.

[0011] However, technologies that compare conditions registered in an attack tree with specification information to determine whether each attack is successful only target existing attacks and are therefore difficult to address potential threats such as new attacks. Furthermore, attacks against AI systems are often carried out through combinations of normal behavior, making it difficult to extract specific risk factors from vulnerability information. Therefore, even technologies that analyze the probability of a threat occurring based on vulnerability information are difficult to address potential threats. Furthermore, technologies that calculate the evaluation value of new features in an incident detection model based on the similarity of current features and the estimated contribution rate of incident detection target cyberattacks and require information that can be used to determine whether an attack is occurring. Therefore, they are difficult to apply to attacks that are indistinguishable from normal behavior. As such, neither of these technologies has been able to improve the safety of AI systems.

[0012] The disclosed technology has been made in consideration of the above, and aims to provide a risk calculation program, a risk calculation method, and a risk calculation device that improve the safety of AI systems.

[0013] In one aspect, the risk calculation program, risk calculation method, and risk calculation device disclosed in the present application store a weight for each attack condition calculated based on the usage rate of each attack condition, which is an element for a predetermined attack against an AI system to be successful, and the presence rate of each specification element included in the specifications of the AI ​​system that is related to the conditions for the attack to be successful, where the presence rate is in the specifications of multiple existing AI systems; accepts information on the specifications of the AI ​​system to be risk assessed; for each attack tree created in advance, identifies the status of the attack condition to be met based on information on the specification elements extracted from the information on the specifications of the AI ​​system to be risk assessed; and causes a computer to execute a process of calculating a risk value for each attack tree of the AI ​​system to be risk assessed based on the weight for each attack condition and the status of the identified attack condition to be met.

[0014] According to one aspect of the risk calculation program, risk calculation method, and risk calculation device disclosed in the present application, it is possible to improve the safety of AI systems.

[0015] FIG. 1 is a block diagram of a risk calculation device according to an embodiment. FIG. 2 is a diagram illustrating an example of an attack tree. FIG. 3 is a diagram illustrating information on AI system specifications stored in an AI specification information database. FIG. 4 is a diagram illustrating an example of raw score calculation when the logical expression is AND and OR. FIG. 5 is a diagram illustrating an attack tree including NOT in the logical expression and an example of raw score calculation for the attack tree. FIG. 6 is a diagram illustrating an attack tree including another attack scenario and an example of raw score calculation for the attack tree. FIG. 7 is a diagram for explaining calculation of raw scores after score calculation for each attack tree. FIG. 8 is a diagram illustrating an example of weight calculation for attack conditions. FIG. 9 is a diagram illustrating an example of attack condition identification processing. FIG. 10 is a diagram illustrating an example of risk value calculation processing. FIG. 11 is a diagram illustrating an example of risk value notification. FIG. 12 is a diagram illustrating an overall overview of information provision using risk calculation processing according to an embodiment. FIG. 13 is a flowchart of risk calculation and information provision processing by the risk calculation device according to an embodiment. FIG. 14 is a flowchart of advance preparation processing. FIG. 15 is a flowchart of risk determination processing. FIG. 16 is a hardware configuration diagram of the risk calculation device.

[0016] Hereinafter, embodiments of the risk calculation program, the risk calculation method, and the risk calculation device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the risk calculation program, the risk calculation method, and the risk calculation device disclosed in the present application are not limited to the following embodiments.

[0017] 1 is a block diagram of a risk calculation device according to an embodiment. The risk calculation device 1 calculates a weight representing the degree of danger of each operation performed in an attack against an AI system, and calculates and provides the risk of an attack against the AI ​​system using the weight. As a result, the risk calculation device 1 can provide information that serves as the basis for determining how safe an AI system that is the target of risk assessment can be considered to be against both existing attacks and currently unknown attacks.

[0018] Here, as a countermeasure against attacks on AI systems, it is conceivable to use specifications of AI systems that are likely to be targeted by attackers in the future, rather than information on the results of attacks such as vulnerability information, in order to calculate the risk of future attacks that have not yet become widespread.

[0019] Therefore, we will consider the trends of attackers when carrying out attacks. First, it is thought that attackers are likely to target the specifications of AI systems that have been the cause of past attacks in future attacks. This is because attackers are likely to use attack methods that are already in circulation, as they are easy to adapt. Second, it is thought that attackers are likely to devise attack methods that can be adapted to the specifications of currently widely used AI systems. This is because, at the very least, it is thought that attackers are unlikely to use conditions that are rare in the world as conditions when carrying out attacks. For example, since there are almost no AI systems that output internal data as response information, it is unlikely that an attack would take advantage of this usage, such as outputting internal data as response information.

[0020] Operations in specific attacks against AI systems are generally performed within the normal operation of the AI ​​system, making it difficult to directly determine the risk of those operations, and even if the attack conditions are met, it is difficult to calculate the risk value of the attack from those conditions. Therefore, as will be explained below, the risk calculation device 1 calculates the risk value for potential threats by taking into account the two tendencies of attackers described above and focusing on whether the specifications meet the attack conditions that have been used in past attacks.

[0021] The risk calculation device 1 will be described in detail below. As shown in FIG. 1 , the risk calculation device 1 includes an attack condition element score calculation unit 11, an attack condition weight calculation unit 12, a specification element existence rate calculation unit 13, an attack condition identification unit 14, a risk calculation unit 15, and a notification unit 16. The risk calculation device 1 also includes an attack method condition distribution database 101, an AI specification information database 102, and an attack condition weight database 103. The processes performed by the risk calculation device 1 include a preparation stage process for calculating a risk value for a specific AI system, and a risk assessment stage process for actually calculating a risk value for the specific AI system using information obtained in the preparation stage. The operation of the risk calculation device 1 will be described below, divided into the preparation stage and the risk assessment stage.

[0022] (Preparation Stage) The attack technique condition distribution database 101 stores attack conditions and attack tree expression information. Attack conditions are the operations performed by an attacker to establish an attack tree. Attack tree expression information, also called tree structure information, is information that represents the structure of the attack tree using attack information and logical expressions that connect them.

[0023] FIG. 2 is a diagram showing an example of an attack tree. The square frames in the attack tree 200 are called nodes. The nodes are connected by logical expressions to form a hierarchy. The nodes at the bottom of the page are called "leaves." The leaves in the attack tree correspond to attack conditions or other attack scenarios. The leaves in this attack tree are an example of a "second node."

[0024] When the attack tree 200 is established, the attack scenario A1 indicated by the top node 201 is established. The top node 201 is called the root node. Nodes other than the root node may be called child nodes. This root node is an example of a "first node."

[0025] The attack tree 200 also has five leaves, nodes 205 to 209. Nodes 205 to 209 correspond to attack conditions when an attack is carried out and also correspond to specification conditions when building an AI system. Nodes 202 to 204 are partial scenarios for establishing attack scenario A1. For example, if condition 2-2 of node 208 is satisfied as an attack condition and condition 3-1 of node 209 is satisfied, the partial scenario of node 204 is established. If the partial scenario of node 204 is established or condition 6-3 of node 205 is satisfied as an attack condition, the partial scenario of node 202 is established. If condition 4-3 of node 208 or condition 7-1 of node 207 is satisfied as an attack condition, the partial scenario of node 203 is established. Then, if the partial scenario of node 202 is established and the partial scenario of node 203 is established, attack scenario A1 is established, and an attack based on attack scenario A1 becomes possible against the AI ​​system.

[0026] Furthermore, as mentioned above, the types of attacks and damages on AI systems are limited, and attack trees for AI systems can be generated in advance. There are multiple attack trees that can be created in advance. Therefore, the attack technique condition distribution database 101 can acquire and store expression information for each of multiple attack trees in advance. For example, the expression information for the attack tree 200 shown in FIG. 2 is expressed as AND(OR(AND(2-2, 3-1), 6-3), OR(4-3, 7-1)).

[0027] For example, a user creates an attack tree for each attack method against an AI system, and registers and stores the formula information for each attack tree in advance in the attack method condition distribution database 101 using the terminal device 2. In addition, the user uses the created attack tree for each attack method to collect specification elements of the AI ​​system that can be attacked, and registers and stores the collected specification elements in advance in the attack method condition distribution database 101 as attack conditions using the terminal device 2.

[0028] The AI ​​specification information database 102 stores various specification information, including specifications that correspond to attack conditions, collected from various existing AI systems, in association with each AI system. Here, the specifications of an AI system for an attack condition are information indicating whether the AI ​​system corresponds to a specification element that corresponds to the attack condition. In other words, an AI system that corresponds to a specification element means that the AI ​​system executes or satisfies the content of that specification element. The AI ​​specification information database 102 also stores the occurrence rate of each specification element.

[0029] Figure 3 is a diagram showing the specification information of AI systems stored in the AI ​​specification information database. For example, the AI ​​specification information database 102 holds specification information for each AI system, as shown in table 210 in Figure 3. Figure 3 shows a case in which information on four AI systems, systems A to D, is held. For example, system A corresponds to specification element #1, but does not correspond to specification elements #2 and #n.

[0030] For example, a user collects information on the specifications of various existing AI systems and uses an input terminal (not shown) or the like to register and store the collected information in the AI ​​specification information database 102. In addition, the specification element existence rate calculation unit 13 registers and stores the existence rate of the specification elements in the AI ​​specification information database 102.

[0031] The attack condition weight database 103 stores weight information for each attack condition. The attack condition weight information is registered and accumulated in the attack condition weight database 103 by the attack condition weight calculation unit 12. This attack condition weight database 103 is an example of a "storage unit."

[0032] The specification element existence rate calculation unit 13 acquires specification information of various existing AI systems registered in the AI ​​specification information database 102. Then, for each specification element, the specification element existence rate calculation unit 13 divides the number of applicable AI systems by the total number of AI systems for which specification information has been collected to calculate the existence rate of each specification element.

[0033] For example, when the AI ​​specification information database 102 stores information on the specifications of the AI ​​system shown in Figure 3, the specification element presence rate calculation unit 13 calculates the presence rate of specification element #1 as 3 / 4 = 0.75, the presence rate of specification element #2 as 1 / 4 = 0.25, and the presence rate of specification element #n as 2 / 4 = 0.50.

[0034] The specification element existence rate calculation unit 13 then registers and accumulates the calculated existence rate of each specification element in the AI ​​specification information database 102. This existence rate of a specification element indicates how many specification elements exist in AI systems around the world for the attack conditions corresponding to the specification conditions. Since it is believed that the more specification elements exist, the more likely attackers are to use them, in other words, the existence rate of a specification element can be said to indicate the likelihood that it will be targeted by attackers in the future. In other words, the higher the existence rate of a specification element, the more likely it is that attackers will target it in future attacks.

[0035] The attack condition raw score calculation unit 11 acquires attack conditions and attack tree formula information from the attack method condition distribution database 101. Then, the attack condition raw score calculation unit 11 calculates a raw score for each attack condition according to predetermined rules. The attack condition raw score calculation unit 11 then outputs the calculated raw score for each attack condition to the attack condition weight calculation unit 12. An example of calculation of the raw scores of attack conditions by the attack condition raw score calculation unit 11 will be described below.

[0036] The attack condition raw score calculation unit 11 assigns a score of 1 to each leaf of the attack tree corresponding to an attack condition. Next, the attack condition raw score calculation unit 11 calculates the raw score of the attack condition corresponding to each leaf while proceeding from the leaf to the root of the tree based on the formula information of the attack tree, in accordance with the following rules:

[0037] If the next logical expression in the direction of advance is AND, the attack condition element score calculation unit 11 divides the current score by the number of conditions connected to that logical expression. That is, if the next logical expression is AND, the next score = (current score) × (1 / (number of conditions connected to the logical expression)). Also, if the next logical expression is OR, the attack condition element score calculation unit 11 multiplies the current score by 1. That is, if the next logical expression is OR, the next score = (current score) × 1.

[0038] For example, the attack tree 200 in FIG. 2 will be used as an example. FIG. 4 is a diagram showing an example of raw score calculation when the logical formula is AND and OR. Formula 300 shown in FIG. 4 is formula information for the attack tree 200. The attack condition raw score calculation unit 11 calculates raw scores for the attack conditions represented by nodes 205 to 209, which are leaves of the attack tree 200. Here, the attack conditions will be explained by representing them with the condition numbers in FIG. 2 that correspond to those attack conditions. For example, the attack condition corresponding to node 208 will be represented as condition 2-2.

[0039] First, the attack condition element score calculation unit 11 assigns a score of 1 to each of conditions 2-2, 3-1, 6-3, 4-3, and 7-1. Next, the next logical expression 301 in the direction toward node 201 is AND between condition 2-2 corresponding to node 208 and condition 3-1 corresponding to node 209. Therefore, since there are two nodes connected to logical expression 301, the attack condition element score calculation unit 11 multiplies the scores of conditions 2-2 and 3-1 by 1 / 2. In this case, as indicated by score 311, the attack condition element score calculation unit 11 calculates the scores of conditions 2-2 and 3-1 as 0.5. Here, in FIG. 4 , the scores 311 to 314 calculated at each stage are represented by the condition number and the score after the condition number. For example, 2-2: 0.5 indicates that the score of condition 2-2 is 0.5.

[0040] Next, the next logical expression 302 in the direction toward node 201 between node 204, which connects conditions 2-2 and 3-1, and condition 6-3 represented by node 205, is OR. Therefore, the attack condition element score calculation unit 11 multiplies the current scores of conditions 2-2, 7-1, and 6-3 by 1. In this case, as shown in score 312, the attack condition element score calculation unit 11 calculates the scores of conditions 2-2 and 3-1 as 0.5, and the score of condition 6-1 as 1.

[0041] Furthermore, the next logical expression 303 in the direction toward node 201 for condition 4-3 corresponding to node 206 and condition 7-1 corresponding to node 207 is OR. Therefore, the attack condition element score calculation unit 11 multiplies the scores of conditions 4-3 and 7-1 by 1. In this case, as shown in score 313, the attack condition element score calculation unit 11 calculates the score of condition 4-3 and the score of condition 7-1 as 1.

[0042] Next, for node 202, which is connected to conditions 2-2, 3-1, and 6-3, and node 203, which is connected to conditions 4-3 and 7-1, the next logical expression 304 in the direction toward node 201 is AND. Therefore, since there are two nodes connected to logical expression 304, the attack condition element score calculation unit 11 multiplies the current scores of conditions 2-2, 3-1, 6-3, 4-3, and 7-1 by 1 / 2. In this case, as shown in score 314, the attack condition element score calculation unit 11 calculates the scores of conditions 2-2 and 3-1 as 0.25, and the scores of conditions 6-1, 4-3, and 7-1 as 0.5. In this way, the attack condition element score calculation unit 11 can calculate the raw scores of each attack element included in the attack tree 200.

[0043] Furthermore, if the next logical expression is NOT, the attack condition raw score calculation unit 11 treats the entire expression up to the NOT as a separate attack condition from the attack conditions connected to the logical expression from below. Figure 5 is a diagram showing an example of an attack tree containing a NOT in its logical expression and its raw score calculation. For example, in the attack tree 320 shown in Figure 5, node 325 is connected to a NOT logical expression. Formula 331 is formula information representing the attack tree 320.

[0044] The attack condition element score calculation unit 11 sets conditions 8-1 to 8-1, which are separate from condition 8-1, including the logical expression of NOT and condition 8-1 corresponding to node 326. Then, the attack condition element score calculation unit 11 assigns a score of 1 to each of conditions 2-4, 3-1, 4-1, and 8-1. Next, the next logical expression in the direction of node 321, which is the root node, for nodes 322 to 325 is AND, which is represented by logical expression 332 in expression 331. Here, node 325, which is the NOT of node 326, is represented as 8-1, as shown in element 333 in expression 331. Therefore, since the number of nodes connected to this logical expression is four, the attack condition element score calculation unit 11 multiplies the current scores of conditions 2-4, 3-1, 4-1, and 8-1 by 1 / 4. In this case, the attack condition element score calculation unit 11 calculates the scores of conditions 2-4, 3-1, 4-1, and 8-1 as 0.25, as shown in score 334. In this way, the attack condition element score calculation unit 11 can calculate the raw scores of each attack element included in the attack tree 320.

[0045] Furthermore, if the node of the attack tree under consideration includes the establishment of another attack scenario as a condition, the attack condition score calculation unit 11 includes all attack conditions included in the attack tree that establishes that attack scenario as attack conditions of the attack tree under consideration. In this case, the attack condition score calculation unit 11 carries over the scores of each attack condition calculated in the attack tree of the other attack scenario and calculates the score.

[0046] 6 is a diagram showing an example of an attack tree including another attack scenario and its raw score calculation. For example, the attack condition indicated by node 344 included in attack tree 340 shown in FIG. 6 corresponds to another attack scenario, and in this case, it is an attack condition that is met when attack tree 320 in FIG. 5 is met. Formula 351 is formula information representing attack tree 340. Condition 353 in formula 351 corresponds to the attack condition corresponding to the other attack scenario indicated by node 344.

[0047] The attack conditions in the attack tree 320 in FIG. 5, Condition 2-4, Condition 3-1, Condition 4-1, and Conditions 1 to 8-1, all have a score of 0.25. Therefore, the attack condition element score calculation unit 11 assumes that Condition 2-4, Condition 3-1, Condition 4-1, and Conditions 1 to 8-1, each with a score of 0.25, are connected below node 344. The attack condition element score calculation unit 11 also assigns a score of 1 to Condition 2-1 corresponding to node 342 and Condition 3-1 corresponding to node 343. The next logical expression from nodes 342 to 344 in the direction of node 341, which is the root node, is AND, and is represented by logical expression 352 in expression 351. Therefore, since the number of nodes connected to this logical expression is three, the attack condition element score calculation unit 11 multiplies the current scores of Condition 2-1, Condition 3-1, Condition 2-4, Condition 3-1, Condition 4-1, and Conditions 1 to 8-1 by 1 / 3. In this case, as indicated by score 354, the attack condition element score calculation unit 11 calculates the scores for conditions 2-1 and 3-1 as 0.33, and the scores for conditions 2-4, 3-1, 4-1, and 8-1 as 0.073.

[0048] Next, the attack condition weight calculation unit 12 calculates the sum of the scores for each attack tree for each attack condition to calculate a raw score for each attack condition. Figure 7 is a diagram for explaining the calculation of the raw score after calculating the score for each attack tree. The score calculation result 361 in Figure 7 shows the score for each attack condition in the attack tree for each of the attack scenarios A1, X5, and A3.

[0049] For example, the attack condition raw score calculation unit 11 calculates the score of each attack condition for each attack tree of attack scenarios A1, X5, and A3, as shown in the score calculation result 361. Then, the attack condition raw score calculation unit 11 calculates the raw score of each attack condition by adding up the scores of the same attack condition in each attack tree. As shown in the raw score calculation result 362, the attack condition raw score calculation unit 11 calculates the raw scores of conditions 2-1, 2-2, 2-4, 3-1, 4-1, 4-3, 7-1, and 8-1, which are attack conditions included in at least one of the attack trees. For example, because condition 3-1 is included in all of the attack trees of attack scenarios A1, X5, and A3, the attack condition raw score calculation unit 11 calculates the raw score of condition 3-1 by adding three values, as follows: 0.25 + 0.25 + 0.33 = 0.833. Furthermore, for example, since condition 2-1 exists in the attack tree of attack scenario A3 but does not exist in attack scenarios A1 and X5, the attack condition raw score calculation unit 11 sets 0.33, which is the score in the attack tree of attack scenario A3, as the raw score for condition 2-1.

[0050] Here, the raw score of an attack condition represents the frequency of use of that attack condition in attacks, or in other words, the likelihood of its use as an attack condition. In other words, for attack conditions connected with AND, the condition based on that attack condition will not be established unless all attack conditions are satisfied, making that attack condition less likely to be used. In addition, for attack conditions connected with OR, the condition based on that attack condition will be established if any of the attack conditions is satisfied, so the likelihood of that attack condition being used does not change. Therefore, when calculating the raw score, it is preferable to use a rule such that attack conditions connected with AND receive a lower score and attack conditions connected with OR receive an unchanged score. In other words, the attack condition raw score calculation unit 11 calculates the usage rate of a given attack condition in the tree structure information when it is alternatively selected from other attack conditions (i.e., when the logical formula is an attack condition connected with OR), so that the usage rate of the given attack condition is lower.

[0051] The attack condition weight calculation unit 12 receives the raw score of each attack condition from the attack condition raw score calculation unit 11. The attack condition weight calculation unit 12 also acquires the presence rate of the specification elements of the AI ​​system from the AI ​​specification information database 102.

[0052] Next, the attack condition weight calculation unit 12 calculates the weight of each attack condition by multiplying the raw score of each attack condition by the occurrence rate of the AI ​​specification element. That is, the attack condition weight calculation unit 12 calculates the weight so that the higher the usage rate of an attack condition in an attack, the higher the weight, and the higher the occurrence rate in existing AI systems, the higher the weight of the attack condition corresponding to that specification element. Thereafter, the attack condition weight calculation unit 12 associates each attack condition with the calculated weight for each attack condition and stores and accumulates them in the attack condition weight database 103.

[0053] 8 is a diagram showing an example of attack condition weight calculation. For example, the attack condition weight calculation unit 12 obtains information shown in table 401 as the presence rate of specification elements of the AI ​​system from the AI ​​specification information database 102. Next, the attack condition weight calculation unit 12 calculates weight 403 for each attack condition by multiplying the raw score of that attack condition by the presence rate of the specification element, as shown in calculation 402. Then, the attack condition weight calculation unit 12 associates each attack condition with its weight and stores and accumulates it in the attack condition weight database 103, as shown in table 404, for example.

[0054] (Risk Assessment Stage) The attack condition identification unit 14 receives input of the specifications of a specific AI system to be assessed for risk from the terminal device 2. The attack condition identification unit 14 also acquires attack tree information and attack conditions for each attack scenario from the attack method condition distribution database 101. The attack condition identification unit 14 then sets attack conditions corresponding to specification elements to which the specific AI system applies as True. The attack condition identification unit 14 also sets attack conditions corresponding to specification elements to which the specific AI system does not apply as False.

[0055] 9 is a diagram showing an example of the attack condition identification process. For example, the attack condition identification unit 14 acquires the specifications of a specific AI system that is the target of risk assessment, which are shown in table 501. Then, as shown in table 502, the attack condition identification unit 14 sets attack conditions corresponding to specification elements that the specific AI system applies to as True, and sets attack conditions corresponding to specification elements that do not apply to as False.

[0056] In this way, the attack condition identification unit 14 can identify the attack condition that is set to True as the attack condition that is established in the specific AI system. After that, the attack condition identification unit 14 outputs the establishment status of the attack condition in the specific AI system that is the target of risk assessment to the risk calculation unit 15. For example, the attack condition identification unit 14 may output a table 502 shown in FIG. 9 to the risk calculation unit 15.

[0057] The risk calculation unit 15 receives input of information on the establishment status of attack conditions in a specific AI system that is the target of risk assessment from the attack condition identification unit 14. The risk calculation unit 15 also acquires the weight of each attack condition from the attack condition weight database 103. Furthermore, the risk calculation unit 15 acquires structural information of the attack tree of each attack scenario from the attack method condition distribution database 101.

[0058] The risk calculation unit 15 then adds up the weights of the attack conditions that are True for each attack tree to calculate a risk value for each attack tree for the specific AI system. The risk calculation unit 15 then outputs the calculated risk value for each attack tree for the specific AI system to the notification unit 16.

[0059] 10 is a diagram showing an example of the risk value calculation process. For example, the risk calculation unit 15 acquires the establishment status of the attack conditions shown in table 511, the weight of each attack condition shown in table 512, and the structure information 513 of the attack tree.

[0060] Next, the risk calculation unit 15 obtains the weight of each attack condition that is set to True in the table 511 from the table 512. Next, the risk calculation unit 15 identifies the attack conditions that correspond to nodes 205 to 209 in the attack tree 200, for example, from the attack tree structure information 513. In this case, since the attack conditions that correspond to nodes 207 to 209 are set to True, the risk calculation unit 15 adds up the respective weights to calculate a risk value in the specific AI system for the attack scenario A1 shown in the attack tree 200.

[0061] In this embodiment, the risk calculation unit 15 acquires structural information of the attack tree stored in the device from the attack method condition distribution database 101, and acquires the establishment status of the attack conditions from the attack condition identification unit 14. However, the present invention is not limited to this, and the risk calculation unit 15 may acquire such information from another function that performs processing to check the conditions registered in the attack tree against the specification information and determine whether each attack is established.

[0062] Furthermore, the risk calculation unit 15 can determine whether or not an attack scenario shown in each attack tree is established based on the establishment status of each attack condition in a specific AI system. Therefore, the risk calculation unit 15 can also output information on the feasibility of an attack according to each attack scenario to the notification unit 16.

[0063] The notification unit 16 receives an input of a risk value for each attack tree for a specific AI system that is the target of risk assessment from the risk calculation unit 15. Then, the notification unit 16 transmits information on the risk value for each attack tree for the specific AI system to the terminal device 2, and notifies the user of the security risk for the specific AI system.

[0064] 11 is a diagram showing an example of a notification of a risk value. For example, the notification unit 16 acquires information on the feasibility of an attack for each attack scenario, along with the risk value of an attack tree corresponding to each attack scenario, from the risk calculation unit 15. Next, the notification unit 16 creates a table 520 that indicates the feasibility of an attack and the risk value for each attack scenario. The notification unit 16 then transmits the created table 520 to the terminal device 2 and displays it on the screen, thereby notifying the user of the security risk of a specific AI system.

[0065] 12 is a diagram illustrating an overall overview of information provision using the risk calculation process according to the embodiment. Here, an overview of an example of information provision using the risk calculation process according to the embodiment will be described with reference to FIG. 12. A preparation process 610 in FIG. 12 is performed by the attack condition element score calculation unit 11, the attack condition weight calculation unit 12, and the specification element existence rate calculation unit 13 shown in FIG. 1.

[0066] The advance preparation process 610 acquires AI system specification information 611 for an existing AI system from the AI ​​system specification information 601. The advance preparation process 610 also calculates raw scores 612 for attack conditions. The advance preparation process 610 then calculates weights 613 for the attack conditions using the AI ​​system specification information 611 and the raw scores 612 for the attack conditions.

[0067] The risk calculation unit 15 can operate as part of an assessment tool 602 that evaluates AI systems. The assessment tool 602 acquires specification information of a specific AI system that is the target of risk assessment from the AI ​​system specification information 601. The risk calculation unit 15 then calculates a risk value for each attack tree in the specific AI system using the specification information of the specific AI system acquired by the assessment tool 602 and the attack condition weights 613. The assessment tool 602 provides the evaluation of the specific AI system, including information on the risk value for each attack tree calculated by the risk calculation unit 15, to the user as output information 603.

[0068] (Risk Calculation and Information Provision Processing) Fig. 13 is a flowchart of the risk calculation and information provision processing by the risk calculation device 1 according to the embodiment. Next, the flow of the risk calculation and information provision processing by the risk calculation device 1 according to the embodiment will be described with reference to Fig. 13 .

[0069] Prior to calculating the risk of the AI ​​system, the attack condition element score calculation unit 11, the attack condition weight calculation unit 12, and the specification element existence rate calculation unit 13 perform a preparatory process to obtain the weight of each attack condition to be used in calculating the risk (step S1).

[0070] The attack condition identification unit 14 and the risk calculation unit 15 perform a risk assessment process to calculate and notify the risk of the input AI system using the weight of each attack condition obtained in the advance preparation process (step S2).

[0071] Fig. 14 is a flowchart of the advance preparation process. Each process in the flow shown in Fig. 14 corresponds to an example of the process executed in step S1 in Fig. 13. Next, the flow of the advance preparation process will be described with reference to Fig. 14.

[0072] The attack method condition distribution database 101 receives and stores expression information for each attack tree and attack conditions, which are specification elements that enable attacks (step S101).

[0073] The AI ​​specification information database 102 collects and stores information on the specifications of multiple existing AI systems (step S102).

[0074] The specification element existence rate calculation unit 13 acquires specification information of existing AI systems stored in the AI ​​specification information database 102. Then, for each specification element, the specification element existence rate calculation unit 13 divides the number of corresponding AI systems by the total number of AI systems for which specification information has been collected to calculate the existence rate of each specification element (step S103). Thereafter, the specification element existence rate calculation unit 13 registers the calculated existence rate of each specification element in the AI ​​specification information database 102.

[0075] The attack condition raw score calculation unit 11 calculates the score for each attack condition using the attack conditions and attack tree formula information stored in the attack method condition distribution database 101. Next, the attack condition raw score calculation unit 11 adds up the scores for each attack tree for each attack condition to calculate the raw score for each attack condition (step S104).

[0076] The attack condition weight calculation unit 12 obtains the raw score of each attack condition calculated by the attack condition raw score calculation unit 11. The attack condition weight calculation unit 12 also obtains the presence rate of each specification element from the AI ​​specification information database 102. Then, for each attack element, the attack condition weight calculation unit 12 calculates the weight of the attack condition by multiplying the raw score of the attack condition by the presence rate of the specification element corresponding to that attack element (step S105). Thereafter, the attack condition weight calculation unit 12 stores and accumulates the calculated weight of each attack condition in the attack condition weight database 103.

[0077] Fig. 15 is a flowchart of the risk assessment process. Each process in the flow shown in Fig. 15 corresponds to an example of the process executed in step S2 of Fig. 13. Next, the flow of the advance preparation process will be described with reference to Fig. 15.

[0078] The attack condition identification unit 14 receives input of information on the specifications of a specific AI system to be subjected to risk assessment from the terminal device 2 (step S201).

[0079] Then, the attack condition specifying unit 14 specifies an attack condition that is true for each specification requirement from the acquired specification information (step S202).

[0080] The risk calculation unit 15 acquires information on the establishment status of the attack conditions, including information on attack conditions that are True, from the attack condition identification unit 14. The risk calculation unit 15 also acquires the weight of each attack condition from the attack condition weight database 103. The risk calculation unit 15 also acquires structural information on the attack tree from the attack method condition distribution database 101. Then, the risk calculation unit 15 calculates the sum of the weights of the attack conditions that are True for each attack tree, and sets the calculated value as the risk value for each attack tree for the specific AI system (step S203).

[0081] The notification unit 16 transmits risk information regarding the specific AI system, including the risk value calculated by the risk calculation unit 15, to the terminal device 2, and notifies the user of the risk information (step S204).

[0082] As described above, the risk calculation device according to this embodiment executes the following processing as preparation for risk calculation. The risk calculation device calculates the usage rate of specification elements based on the collected specification information of existing AI systems, and determines the likelihood that each specification element will be targeted for future attacks. The risk calculation device also calculates a score for each attack condition based on the configuration of the attack tree, and calculates the likelihood that each attack condition will be used in attacks. The risk calculation device then calculates the weight of each attack condition using the usage rate of the specification elements and the raw score of the attack condition. After the preparation is completed, the risk calculation device acquires specification information of the AI ​​system to be subjected to risk calculation, and calculates a risk value for each attack tree of the AI ​​system to be subjected to risk calculation, using the weight of the attack condition that is determined to be true.

[0083] In this way, the risk calculation device according to this embodiment can calculate a risk value that includes potential risks to an AI system based on the likelihood that the AI ​​system's specification information will be the target of an attack under the attack conditions. That is, the risk calculation device according to this embodiment can calculate the degree of security of an AI system, including potential threats, by taking into account attack trends, rather than simply determining whether an attack will be successful. Potential threats include threats resulting from unknown attack scenarios. This makes it possible to visualize and present to users potential threats of attacks that are deemed difficult to achieve or that are difficult to analyze because they are not subject to consideration for feasibility. Furthermore, by presenting information about the risk of future attacks, developers of AI systems can, for example, take measures in advance, such as eliminating elements that could potentially be the cause of attacks. This improves the security of AI systems.

[0084] (Hardware Configuration) Fig. 16 is a diagram showing the hardware configuration of the risk calculation device 1. Next, an example of the hardware configuration for realizing each function of the risk calculation device 1 will be described with reference to Fig. 16.

[0085] 16, the risk calculation device 1 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0086] The network interface 94 is an interface for communication between the risk calculation device 1 and an external device. The network interface 94 relays communication between the terminal device 2 and the CPU 91, for example.

[0087] The hard disk 93 is an auxiliary storage device. The hard disk 93 can realize the functions of the attack method condition distribution database 101, the AI ​​specification information database 102, and the attack condition weight database 103 illustrated in Fig. 1. The hard disk 93 also stores various programs including programs for realizing the functions of the attack condition element score calculation unit 11, the attack condition weight calculation unit 12, the specification element existence rate calculation unit 13, the attack condition identification unit 14, the risk calculation unit 15, and the notification unit 16 illustrated in Fig. 1.

[0088] The memory 92 is a main storage device, and may be, for example, a dynamic random access memory (DRAM).

[0089] The CPU 91 reads various programs from the hard disk 93, expands them into the memory 92, and executes them. As a result, the CPU 91 realizes the functions of the attack condition element score calculation unit 11, the attack condition weight calculation unit 12, the specification element existence rate calculation unit 13, the attack condition identification unit 14, the risk calculation unit 15, and the notification unit 16 illustrated in FIG.

[0090] REFERENCE SIGNS LIST 1 Risk calculation device 2 Terminal device 11 Attack condition element score calculation unit 12 Attack condition weight calculation unit 13 Specification element existence rate calculation unit 14 Attack condition identification unit 15 Risk calculation unit 16 Notification unit 101 Attack method condition distribution database 102 AI specification information database 103 Attack condition weight database

Claims

1. A risk calculation program that causes a computer to execute the following processes: store a weight for each attack condition, which is calculated based on the usage rate of each attack condition that is an element for a specific attack against an AI system to be successful, and the presence rate of each specification element that is included in the specifications of the AI ​​system and is related to the conditions for the attack to be successful, where the presence rate in the specifications of multiple existing AI systems; accept information on the specifications of the AI ​​system that is the target of risk assessment; identify the status of establishment of the attack condition for each attack tree created in advance, based on information on the specification elements extracted from the information on the specifications of the AI ​​system that is the target of risk assessment; and calculate a risk value for each attack tree of the AI ​​system that is the target of risk assessment, based on the weight for each attack condition and the status of establishment of the identified attack conditions.

2. The risk calculation program of claim 1, further comprising causing the computer to execute the following processes: acquiring tree structure information representing a structure of a plurality of attack trees including a first node associated with information indicating the specified attack and a plurality of second nodes associated with the attack conditions, calculating the usage rate of each of the attack conditions based on each of the tree structure information; acquiring specification information for the plurality of existing AI systems; calculating the existence rate of each of the specification elements based on the specification information for the plurality of existing AI systems; and calculating and storing a weight for each of the attack conditions based on the usage rate of each of the attack conditions and the existence rate of each of the specification elements.

3. The risk calculation program according to claim 2, characterized in that the computer is caused to execute a process of calculating the usage rate of each of the attack conditions in the tree structure information so that when an alternative selection is made between a specified attack condition and other attack conditions, the usage rate of the specified attack condition is low.

4. The risk calculation program according to claim 1, characterized in that the computer is caused to execute a process of calculating the presence rate of each of the specification elements based on the number of existing AI systems that include the specification elements in their specification information and the total number of existing AI systems that have acquired the specification information.

5. A risk calculation method characterized by executing a process in which a risk calculation device: stores a weight for each attack condition, which is calculated based on the usage rate of each attack condition that is an element for a specified attack against an AI system to be successful, and the presence rate of each specification element that is included in the AI ​​system's specifications and is related to the conditions for the attack to be successful, where the presence rate is in the specifications of a plurality of existing AI systems; accepts information on the specifications of the AI ​​system to be subjected to risk assessment; identifies the status of establishment of the attack condition for each attack tree created in advance, based on information on the specification elements extracted from the information on the specifications of the AI ​​system to be subjected to risk assessment; and calculates a risk value for each attack tree of the AI ​​system to be subjected to risk assessment, based on the weight for each attack condition and the status of establishment of the identified attack conditions.

6. A risk calculation device comprising: a memory unit that stores a weight for each attack condition calculated based on the usage rate of each attack condition, which is an element for a specified attack against an AI system to be successful, and the presence rate of each specification element included in the specifications of the AI ​​system and related to the conditions for the attack to be successful, where the presence rate in the specifications of a plurality of existing AI systems; an attack condition identification unit that receives specification information of the AI ​​system to be subjected to risk assessment, and identifies, for each attack tree created in advance, the status of establishment of the attack condition based on information on the specification elements extracted from the specification information of the AI ​​system to be subjected to risk assessment; and a risk calculation unit that calculates a risk value for each attack tree of the AI ​​system to be subjected to risk assessment based on the weight for each attack condition stored in the memory unit and the status of establishment of the attack condition identified by the attack condition identification unit.