Verification device, verfication method and program

Abstract

A verification device according to one embodiment of the present disclosure verifies identity between each of a plurality of first machine learning models and a second machine learning model. The verification device comprises: an input unit that inputs the same input a predetermined number of times to each of the plurality of first machine learning models and the second machine learning model; and a verification unit that verifies identity between each of the plurality of first machine learning models and the second machine learning model, on the basis of a plurality of first model characteristics that respectively represent the distribution of the outputs of each of the plurality of first machine learning models when the input is input the predetermined number of times, and a second model characteristic that represents the distribution of the outputs of the second machine learning model when the input is input the prescribed number of times.

Description

Verification device, verification method, and program

[0001] The present disclosure relates to a verification device, a verification method, and a program.

[0002] Among the machine learning models that realize AI (Artificial Intelligence), some continue to change through continuous machine learning in order to improve their sophistication and adapt to external environments. Such machine learning models have the problem that it is difficult to detect changes due to erroneous learning or unauthorized substitution of machine learning models by third parties. In response to this problem, a method has been proposed for verifying the identity of multiple machine learning models by utilizing the characteristics of the machine learning models (Non-Patent Document 1).

[0003] Naoto Kiribuchi, Yuya Sato, Ryohei Suzuki, Nami Ashizawa, Satoshi Oki, Hirofumi Mineno, Masakatsu Nishigaki, "A Study on the Use of Verifiable Machine Learning Models: Proposal of a Model Identity Verification Method Based on Robustness to Image Manipulation," Research Report Computer Security (CSEC), vol.2023-CSEC-101, no.16, pp.1-7, 2023.

[0004] However, the method proposed in Non-Patent Document 1 is a method for verifying whether a certain machine learning model is identical to another certain machine learning model, and cannot verify which of multiple machine learning models a certain machine learning model is identical to.

[0005] The present disclosure has been made in consideration of the above points, and aims to verify identity with multiple machine learning models.

[0006] A verification device according to one aspect of the present disclosure is a verification device that verifies the identity of each of a plurality of first machine learning models and a second machine learning model, and includes an input unit that inputs the same input to each of the plurality of first machine learning models and the second machine learning model a predetermined number of times, and a verification unit that verifies the identity of each of the plurality of first machine learning models and the second machine learning model based on a plurality of first model characteristics that respectively represent the distribution of outputs of each of the plurality of first machine learning models when the input is inputted the number of times, and a second model characteristic that represents the distribution of outputs of the second machine learning model when the input is inputted the number of times.

[0007] It is possible to verify identity with multiple machine learning models.

[0008] It is a diagram illustrating an example of a hardware configuration of a verification device according to the present embodiment. It is a diagram illustrating an example of a functional configuration of a verification device according to the present embodiment. It is a flowchart illustrating an example of a verification process according to the present embodiment.

[0009] An embodiment of the present invention will be described in detail below with reference to the drawings. In the following embodiment, a verification device 10 will be described that can verify whether a certain machine learning model is the same as any of a plurality of machine learning models. The number of machine learning models is set to N (where N is an integer equal to or greater than 2), and these machine learning models are referred to as "machine learning model T （１） ", ..., "machine learning model T （Ｎ） " Also, the machine learning model T （１） , ..., machine learning model T （Ｎ） The machine learning model to be verified as being identical to any of the above (or not identical to any of the above) is referred to as "machine learning model V."

[0010] Machine learning model T （ｎ） (n=1, . . . , N) is assumed to be a legitimate machine learning model that operates as intended by the model creator. On the other hand, machine learning model V is assumed to be a machine learning model T （ｎ）(n = 1, ..., N), examples of which include unknown machine learning models whose behavior is unknown, machine learning models that have been altered by erroneous learning from known machine learning models, and machine learning models that have been fraudulently replaced by a third party.

[0011] Each machine learning model T （ｎ） Each of the n (n = 1, ..., N) may be a machine learning model whose output is deterministically determined for the same input, or may be a machine learning model whose output is not deterministically determined for the same input. Similarly, the machine learning model V may be a machine learning model whose output is deterministically determined for the same input, or may be a machine learning model whose output is not deterministically determined for the same input.

[0012] An example of a machine learning model in which the output is not deterministically determined for the same input is a machine learning model called a large language model (LLM) that realizes generative AI (or may be called generative AI). Such a machine learning model does not necessarily obtain the same output for the same input, and the output changes probabilistically for the same input. Therefore, a machine learning model in which the output is not deterministically determined for the same input can also be called a machine learning model in which the output is probabilistically determined for the same input.

[0013] Hereinafter, we will refer to a machine learning model in which the output is determined deterministically for the same input as a "deterministic model," and a machine learning model in which the output is determined probabilistically for the same input as a "probabilistic model."

[0014] <Example of Hardware Configuration of Verification Device 10> An example of the hardware configuration of the verification device 10 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the hardware configuration of the verification device 10 according to this embodiment.

[0015] 1, the verification device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0016] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the verification device 10 does not necessarily have to include at least one of the input device 101 and the display device 102, for example.

[0017] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0018] The communication I / F 104 is an interface for connecting to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The processor 108 is a variety of arithmetic devices such as a central processing unit (CPU) or a graphic processing unit (GPU).

[0019] 1 is an example, and the hardware configuration of the verification device 10 is not limited to this. For example, the verification device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0020] <Example of Functional Configuration of Verification Apparatus 10> An example of the functional configuration of the verification apparatus 10 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the functional configuration of the verification apparatus 10 according to this embodiment.

[0021] 2, the verification device 10 according to this embodiment includes a model input unit 201, a model property acquisition unit 202, a verification unit 203, and a verification result output unit 204. Each of these units is implemented, for example, by a process in which one or more programs installed in the verification device 10 are executed by the processor 108 or the like. The verification device 10 according to this embodiment also includes an input storage unit 205 and a model property storage unit 206. Each of these storage units is implemented, for example, by a storage area of ​​the auxiliary storage device 107 or the like. Note that at least one of the input storage unit 205 and the model property storage unit 206 may be implemented by a storage area of ​​a storage device (e.g., a storage device included in a database server) or the like that is communicatively connected to the verification device 10.

[0022] The model input unit 201 inputs the input x stored in the input storage unit 205 to each machine learning model T （ｎ） (n = 1, ..., N) is input k times. This creates a machine learning model T （ｎ） k outputs y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） The machine learning model T （ｎ） If y is a deterministic model, Ｔ，１ （ｎ） =...=y Ｔ，ｋ （ｎ） On the other hand, the machine learning model T （ｎ） If y is a probabilistic model, Ｔ，１ （ｎ） =...=yＴ，ｋ （ｎ） Here, k is a predetermined integer of 1 or more (particularly, the machine learning model T （ｎ） If y is a probabilistic model, k=1. Ｔ，ｉ （ｎ） is the i-th (where 1≦i≦k) input x to the machine learning model T （ｎ） Generally, the i-th input x is input to the machine learning model T （ｎ） Each time input is made to Ｔ，ｉ （ｎ） is obtained.

[0023] Furthermore, the model input unit 201 inputs the input x stored in the input storage unit 205 to the machine learning model V k times. As a result, k outputs y Ｖ，１ , ..., y Ｖ，ｋ If the machine learning model V is a deterministic model, then y Ｖ，１ =...=y Ｖ，ｋ On the other hand, if the machine learning model V is a probabilistic model, y Ｖ，１ =...=y Ｖ，ｋ Here, y Ｖ，ｉ is the output when the input x is input to the machine learning model V for the i-th time (where 1≦i≦k). In general, each time the i-th input x is input to the machine learning model V, the i-th output y Ｖ，ｉ is obtained.

[0024] The input x is in the form of a machine learning model T （ｎ） and V, but for example, the machine learning model T （ｎ） If V is a machine learning model such as a large-scale language model that implements generative AI, the input x is typically in text format. Similarly, the output y Ｔ，ｉ （ｎ） and y Ｖ，ｉ The format of the machine learning model T （ｎ） and V, but for example, the machine learning model T （ｎ） and V is a machine learning model such as a large-scale language model that realizes generative AI, the output y Ｔ，ｉ （ｎ） and y Ｖ，ｉHowever, it is also possible to input and output data in various formats, such as still images, video, and audio.

[0025] In addition, the machine learning model T （ｎ） and V are stored in a storage area such as the auxiliary storage device 107. （ｎ） At least one of the machine learning models V and V may be stored in a storage area such as a storage device communicatively connected to the verification device 10 (e.g., a storage device provided in an API (Application Programming Interface) server, etc.).

[0026] The model characteristic acquisition unit 202 acquires the machine learning model T （ｎ） k outputs y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） Machine learning model T （ｎ） Model characteristic Y Ｔ （ｎ） = (y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） Similarly, the model characteristic acquisition unit 202 acquires k outputs y Ｖ，１ , ..., y Ｖ，ｋ Let Y be the model characteristic of machine learning model V. Ｖ = (y Ｖ，１ , ..., y Ｖ，ｋ ) is obtained.

[0027] Furthermore, the model characteristic acquisition unit 202 acquires the model characteristic Y Ｔ （ｎ） and Y Ｖ At least one of the above is stored in the model characteristic storage unit 206.

[0028] The verification unit 203 determines the model characteristic Y Ｔ （ｎ） and Y Ｖ Using this, the machine learning model V is compared with the machine learning model T （ｎ） That is, the verification unit 203 verifies whether the model characteristic YＴ （ｎ） (n=1, . . . , N) and Y Ｖ Using the above, the machine learning model V and each machine learning model T （ｎ） Verify identity with each of the

[0029] The verification result output unit 204 outputs the verification result by the verification unit 203 to a predetermined output destination. The predetermined output destination may be, for example, the display device 102 such as a display, a storage area such as the auxiliary storage device 107, or another device or equipment connected to the verification device 10 so as to be able to communicate with the verification device 10.

[0030] The input storage unit 205 stores the machine learning model T （ｎ） and stores the input x given as an input to V.

[0031] The model characteristic storage unit 206 stores the model characteristic Y Ｔ （ｎ） and Y Ｖ Remember.

[0032] <Verification Process> The verification process according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the verification process according to this embodiment.

[0033] The model input unit 201 inputs the input x stored in the input storage unit 205 to each machine learning model T （ｎ） (n=1, . . . , N) is input k times (step S101). As a result, for n=1, . . . , N, the machine learning model T （ｎ） k outputs y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） is obtained.

[0034] The model characteristic acquisition unit 202 acquires the k outputs y obtained in step S101 for n=1, . . . , N. Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） The model characteristic Y Ｔ （ｎ） = (y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ（ｎ） ) (step S102). Ｔ （ｎ） are stored in the model characteristic storage unit 206 as needed. Ｔ （ｎ） is stored in the model characteristic storage unit 206, for example, Ｔ （ｎ） For example, after obtaining the ID, the identity verification in step S105 described later is performed later.

[0035] The model input unit 201 inputs the input x stored in the input storage unit 205 to the machine learning model V k times (step S103). As a result, k outputs y Ｖ，１ , ..., y Ｖ，ｋ is obtained.

[0036] The model characteristic acquisition unit 202 calculates the k outputs y obtained in step S103. Ｖ，１ , ..., y Ｖ，ｋ The model characteristic Y Ｖ = (y Ｖ，１ , ..., y Ｖ，ｋ ) (step S104). Ｖ The model characteristic Y is stored in the model characteristic storage unit 206 as needed. Ｖ is stored in the model characteristic storage unit 206, for example, Ｖ For example, after obtaining the ID, the identity verification in step S105 described later is performed later.

[0037] The above steps S101 to S102 and the above steps S103 to S104 may be performed in any order. That is, for example, the above steps S101 to S102 may be performed after the above steps S103 to S104 are performed. In addition, in the above steps S101 to S102, the machine learning model T （１） , ..., machine learning model T （Ｎ） After inputting the input x k times to each of Ｔ （１） , ..., model characteristic Y Ｔ（Ｎ） However, for example, the machine learning model T （ｎ） After inputting input x k times, the model characteristic Y Ｔ （ｎ） , N.

[0038] The verification unit 203 determines the model characteristic Y Ｔ （ｎ） (n=1, . . . , N) and Y Ｖ Using this, the machine learning model V is compared with the machine learning model T （ｎ） Here, the verification unit 203 verifies whether the distribution can be regarded as the same as any of the distributions (step S105). ｐ Using the norm, machine learning model V is compared with machine learning model T （ｎ） It is possible to verify which of the machine learning models T can be considered to be identical to the machine learning model V. （ｎ） If there are machine learning models T （ｎ） Information identifying the machine learning model T （ｎ） The final verification result is the machine learning model T that can be considered to be identical to the machine learning model V. （ｎ） does not exist, the verification unit 203 judges that "machine learning model V is not machine learning model T （ｎ） (n=1, . . . , N)” is the final verification result.

[0039] ・When using the distance between distributions, model characteristic Y Ｔ （ｎ） = (y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） ) is k points y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） Similarly, the model characteristic Y Ｖ = (y Ｖ，１ , ..., y Ｖ，ｋ ) also has k points y Ｖ，１ , ..., y Ｖ，ｋ Therefore, for n = 1, ..., N, the model characteristic YＴ （ｎ） = (y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） ) and model characteristic Y Ｖ = (y Ｖ，１ , ..., y Ｖ，ｋ ) and calculate the distribution distance between them. If the value is less than a predetermined threshold, the machine learning model V is judged to be superior to the machine learning model T. （ｎ） If not, "machine learning model V is considered to be identical to machine learning model T" （ｎ） The verification result can be "cannot be considered to be the same as model characteristic Y." Ｔ （ｎ） and model characteristic Y Ｖ As a measure for measuring the distance between the distributions, for example, KL (Kullback-Leibler) divergence can be used.

[0040] ・When using statistical hypothesis testing, model characteristics Y Ｔ （ｎ） = (y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） ) k outputs y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） can be considered as the distribution of a sample statistically extracted from a certain population. Similarly, the model characteristic Y Ｖ = (y Ｖ，１ , ..., y Ｖ，ｋ ) k outputs y Ｖ，１ , ..., y Ｖ，ｋ can be regarded as the distribution of a sample statistically extracted from a certain population. Therefore, for n = 1, ..., N, the model characteristic Y Ｔ （ｎ） and model characteristic Y ＶA statistical hypothesis test is performed on two samples to test whether the population probability distributions are different between the two. Specifically, the null hypothesis is that "the population probability distributions match," and if the p-value is smaller than a predetermined significance level, the null hypothesis is rejected. If the null hypothesis is rejected, the test result shows that the alternative hypothesis "the population probability distributions do not match" is correct, and therefore, "machine learning model V is not machine learning model T." （ｎ） On the other hand, if the null hypothesis is not rejected, it does not necessarily mean that the alternative hypothesis is correct, but we can assume that the alternative hypothesis is correct and conclude that "machine learning model V is not the same as machine learning model T." （ｎ） The verification result can be "can be considered to be the same as the previous two samples." As a statistical hypothesis test for two samples, for example, the Kolmogorov-Smirnov test or the Mann-Whitney U test can be used.

[0041] ・L ｐ When using norm, model characteristic Y Ｔ （ｎ） = (y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） ) k outputs y Ｔ，１ （ｎ） , ..., y Ｔ，ｋ （ｎ） can be expressed in the form of a scalar, vector, matrix, or tensor. Ｖ = (y Ｖ，１ , ..., y Ｖ，ｋ ) k outputs y Ｖ，１ , ..., y Ｖ，ｋ can be expressed in the form of a scalar, vector, matrix, or tensor, where y Ｔ，ｉ （ｎ） and y Ｖ，ｉ are expressed in the same format. In this case, for n=1, . . . , N, the model characteristic Y Ｔ （ｎ） and model characteristic Y Ｖ With L ｐ Norm L ｐ (Y Ｔ（ｎ） , Y Ｖ ) : = (| y Ｔ，１ （ｎ） -y Ｖ，１ | ｐ +...+ |y Ｔ，ｋ （ｎ） -y Ｖ，ｋ | ｐ ) １／ｐ If the value is less than a predetermined threshold, the machine learning model V is determined to be （ｎ） If not, "machine learning model V is considered to be identical to machine learning model T" （ｎ） The verification result can be "cannot be considered the same as p = 1 or p = 2." ｐ The norm can be calculated, for example, ∞ A norm may be used.

[0042] However, for some output y Ｔ，ｉ （ｎ） and y Ｖ，ｉ is expressed in vector, matrix, or tensor form, the output y Ｔ，ｉ （ｎ） and y Ｖ，ｉ Instead of ｐ Specifically, we can calculate the norm of a given output y Ｔ，ｉ （ｎ） and y Ｖ，ｉ is a vector and y Ｔ，ｉ （ｎ） = (y Ｔ，ｉ，ｊ （ｎ） ), y Ｖ，ｉ = (y Ｖ，ｉ，ｊ ), then |y Ｔ，ｉ （ｎ） -y Ｖ，ｉ | ｐ :=Σ ｊ |y Ｔ，ｉ，ｊ （ｎ） -y Ｖ，ｉ，ｊ | ｐ As the above L ｐ Norm L ｐ (Y Ｔ （ｎ） , Y Ｖ ) can be calculated. Ｔ，ｉ，ｊ （ｎ） is the vector y Ｔ，ｉ （ｎ）The jth element of y Ｖ，ｉ，ｊ is the vector y Ｔ，ｉ Similarly, for a given output y Ｔ，ｉ （ｎ） and y Ｖ，ｉ is a matrix and y Ｔ，ｉ （ｎ） = (y Ｔ，ｉ，ｊ，ｊ' （ｎ） ), y Ｖ，ｉ = (y Ｖ，ｉ，ｊ，ｊ' ), then |y Ｔ，ｉ （ｎ） -y Ｖ，ｉ | ｐ :=Σ ｊ，ｊ' |y Ｔ，ｉ，ｊ，ｊ' （ｎ） -y Ｖ，ｉ，ｊ，ｊ' | ｐ As the above L ｐ Norm L ｐ (Y Ｔ （ｎ） , Y Ｖ ) can be calculated. Ｔ，ｉ，ｊ，ｊ' （ｎ） is the matrix y Ｔ，ｉ （ｎ） the (j, j') element of y Ｖ，ｉ，ｊ，ｊ' is the matrix y Ｔ，ｉ The (j, j') element of a certain output y Ｔ，ｉ （ｎ） and y Ｖ，ｉ The same can be considered when is a tensor.

[0043] In addition, the above distribution distance, statistical hypothesis testing, L ｐ The norms are just examples of methods for verifying the identity between machine learning models. In addition to these, any method that can measure the similarity between machine learning models can be used to verify the identity. （ｎ） Depending on the situation, the method for verifying identity with the machine learning model V may differ.

[0044] The verification result output unit 204 outputs the final verification result obtained in step S105 to a predetermined output destination (step S106).

[0045] <Modifications> Modification 1 In the above embodiment, in step S101 of FIG. 3, the input x is input to each machine learning model T （ｎ） Each input was k times, but the machine learning model T （ｎ） For example, in step S101 of FIG. 3, the input x is input to the machine learning model T （ｎ） Nik （ｎ） Here, each k （ｎ） is a predetermined integer equal to or greater than 1. In step S105 of FIG. 3, all k （ｎ） As in the case where the two sequences are identical, statistical hypothesis testing can be used to verify the identity.

[0046] Modification 2 In the above embodiment, one type of input x is used for each machine learning model T （ｎ） and k inputs to the machine learning model V, for example, S types of input x １ , ..., x Ｓ For each machine learning model T （ｎ） and may be input k times to the machine learning model V. Note that S is a predetermined integer of 2 or more.

[0047] In this case, each input x ｓ 3 for (s=1, . . . , S), and then in step S105 of FIG. 3, θ or more inputs x for each n=1, . . . , N are input. ｓ Then machine learning model V is machine learning model T （ｎ） If machine learning model V can be considered to be identical to machine learning model T, （ｎ） If the result of the verification is not "machine learning model V is considered to be identical to machine learning model T", （ｎ） The verification result can be calculated by determining whether the result is "cannot be considered to be the same as the result of the previous verification." The final verification result can be calculated from these verification results. Note that θ is a predetermined threshold value, and can be, for example, θ = S, θ = 0.9 × S, or θ = 0.99 × S.

[0048] <Summary> As described above, the verification device 10 according to this embodiment is configured to verify the machine learning model V by verifying the machine learning model T （ｎ）(n=1, . . . , N) and outputs the verification result. As a result, when an unknown machine learning model V is given, the verification device 10 according to this embodiment can verify whether the model is identical to one of a plurality of known machine learning models T （ｎ） (n=1, ..., N) (or none of them). Therefore, even if the behavior of machine learning model V is unknown, machine learning model T that has been verified to be identical to machine learning model V can be verified. （ｎ） By referring to the above, it is possible to confirm or predict the behavior of the machine learning model V.

[0049] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0050] REFERENCE SIGNS LIST 10 Verification device 101 Input device 102 Display device 103 External I / F 103a Recording medium 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage device 108 Processor 109 Bus 201 Model input unit 202 Model characteristic acquisition unit 203 Verification unit 204 Verification result output unit 205 Input storage unit 206 Model characteristic storage unit

Claims

1. A verification device that verifies the identity of each of a plurality of first machine learning models and a second machine learning model, comprising: an input unit that inputs the same input to each of the plurality of first machine learning models and the second machine learning model a predetermined number of times; and a verification unit that verifies the identity of each of the plurality of first machine learning models and the second machine learning model based on a plurality of first model characteristics that respectively represent the distribution of outputs of each of the plurality of first machine learning models when the input is inputted the number of times, and a second model characteristic that represents the distribution of outputs of the second machine learning model when the input is inputted the number of times.

2. The verification device described in claim 1, wherein the verification unit verifies the identity of the first machine learning model and the second machine learning model based on the inter-distribution distance between the distribution represented by the first model characteristic and the distribution represented by the second model characteristic.

3. The verification device of claim 1, wherein the verification unit verifies the identity of the first machine learning model and the second machine learning model by testing, by statistical hypothesis testing, whether the population of distributions represented by the first model characteristics and the population of distributions represented by the second model characteristics match when the output of the first machine learning model and the output of the second machine learning model are each regarded as samples extracted from a certain population.

4. The verification unit: ｐ The verification device according to claim 1 , wherein the device verifies identity between the first machine learning model and the second machine learning model based on a norm.

5. A verification method in which a verification device that verifies the identity of each of a plurality of first machine learning models and a second machine learning model executes: an input procedure in which the same input is input to each of the plurality of first machine learning models and the second machine learning model a predetermined number of times; and a verification procedure in which the identity of each of the plurality of first machine learning models and the second machine learning model is verified based on a plurality of first model characteristics that respectively represent the distribution of outputs of each of the plurality of first machine learning models when the input is inputted the number of times, and a second model characteristic that represents the distribution of outputs of the second machine learning model when the input is inputted the number of times.

6. A program that causes a verification device that verifies the identity of each of a plurality of first machine learning models and a second machine learning model to execute: an input procedure that inputs the same input to each of the plurality of first machine learning models and the second machine learning model a predetermined number of times; and a verification procedure that verifies the identity of each of the plurality of first machine learning models and the second machine learning model based on a plurality of first model characteristics that respectively represent the distribution of outputs of each of the plurality of first machine learning models when the input is inputted the number of times, and a second model characteristic that represents the distribution of outputs of the second machine learning model when the input is inputted the number of times.

Images