Verification apparatus and verification method

Abstract

To provide a verification device and a verification method for efficiently verifying the identity of a machine learning model.SOLUTION: A verification device for verifying sameness between a first machine learning model and a second machine learning model, the verification device including a problem generation unit configured to generate a plurality of problems to be input to the first machine learning model and the second machine learning model, a model input unit configured to input each of the plurality of problems to each of the first machine learning model and the second machine learning model, and a determination unit configured to determine whether an answer of the first machine learning model to each of the plurality of problems is correct or incorrect and whether an answer of the second machine learning model to each of the plurality of problems is correct or incorrect, the device includes: a model characteristic creation unit which creates a first model characteristic representing the characteristic of the output distribution of a first machine learning model and a second model characteristic representing the characteristic of the output distribution of a second machine learning model; and a verification unit which verifies the identity of the first machine learning model and the second machine learning model by using the first model characteristic and the second model characteristic.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a verification device and a verification method. [Background technology]

[0002] Among the machine learning models that realize AI (Artificial Intelligence), there are some that continue to change through continuous machine learning in order to improve their sophistication and adapt to external environments, etc. The problem with such machine learning models is that it is difficult to notice if they change due to incorrect learning or if a third party fraudulently replaces the machine learning model.

[0003] Regarding the above problem, methods have been proposed for verifying the identity of multiple machine learning models by comparing the distribution of their outputs relative to inputs (Non-Patent Documents 1 and 2). Note that Non-Patent Document 1 proposes a method for verifying the identity of multiple machine learning models, targeting machine learning models whose outputs are deterministically determined for the same input, while Non-Patent Document 2 proposes a method for verifying the identity of multiple machine learning models, targeting machine learning models whose outputs are not deterministically determined for the same input. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Naoto Kiribuchi, Yuya Sato, Ryohei Suzuki, Nami Ashizawa, Satoshi Oki, Hirofumi Mineno, and Masakatsu Nishigaki, "A Study on the Use of Verifiable Machine Learning Models: Proposal of a Model Identity Verification Method Based on Robustness to Image Manipulation," Research Report Computer Security (CSEC), vol. 2023-CSEC-101, no. 16, pp. 1-7, 2023. [Non-patent document 2] Naoto Kiribuchi, Yuya Sato, Ryohei Suzuki, Nami Ashizawa, Satoshi Oki, Hirofumi Mineno, and Masakatsu Nishigaki, "Considerations toward the use of verifiable AI: Identity verification of large-scale language models using personality traits," SCIS2024, 2024. Summary of the Invention [Problem to be solved by the invention]

[0005] However, since the distribution of outputs for each input does not necessarily differ for each machine learning model, it may not be possible to efficiently verify identity depending on the input given to the machine learning model to be verified for identity.

[0006] The present disclosure has been made in consideration of the above points, and aims to efficiently verify the identity of machine learning models. [Means for solving the problem]

[0007] A verification device according to one aspect of the present disclosure is a verification device that verifies the identity of a first machine learning model and a second machine learning model, and includes: a question creation unit that creates a plurality of questions to be input to the first machine learning model and the second machine learning model; an input unit that inputs the plurality of questions to each of the first machine learning model and the second machine learning model; a model characteristic creation unit that creates a first model characteristic that represents the characteristics of the output distribution of the first machine learning model and a second model characteristic that represents the characteristics of the output distribution of the second machine learning model, using the correctness or incorrectness of the answers of the first machine learning model to each of the plurality of questions and the correctness or incorrectness of the answers of the second machine learning model to each of the plurality of questions, respectively; and a verification unit that verifies the identity of the first machine learning model and the second machine learning model using the first model characteristic and the second model characteristic. [Effects of the Invention]

[0008] The identity of machine learning models can be efficiently verified. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a verification apparatus according to the present embodiment. [Figure 2]FIG. 2 is a diagram illustrating an example of a functional configuration of a verification apparatus according to the present embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a detailed functional configuration of a question creation unit according to the present embodiment. [Figure 4] 10 is a flowchart illustrating an example of a question creation process according to the present embodiment. [Figure 5] 10 is a flowchart illustrating an example of a verification process according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will be described in detail below with reference to the drawings. In the following embodiment, a verification device 10 is described that can create multiple inputs whose output distributions may differ for each machine learning model, and then use these inputs to efficiently and accurately verify the identity of two machine learning models.

[0011] Hereinafter, the two machine learning models to be verified for identity will be referred to as "machine learning model T" and "machine learning model V." Machine learning model T is assumed to be, for example, a legitimate machine learning model that operates as intended by the model creator, etc. On the other hand, machine learning model V is a machine learning model whose identity with machine learning model T is to be verified; for example, it is assumed to be a machine learning model that may have changed due to erroneous learning or may have been illegally replaced by a third party. However, machine learning models T and V are both assumed to be machine learning models whose output is not deterministic for the same input.

[0012] An example of a machine learning model whose output is not deterministically determined for the same input is a machine learning model (particularly a machine learning model called a large language model (LLM)) that realizes generative AI (or may be called generative AI). Such machine learning models do not necessarily obtain the same output for the same input, and the output changes probabilistically for the same input. However, a machine learning model that realizes generative AI is just one example, and the following embodiment can be similarly applied to verifying the identity of any two machine learning models whose output is not deterministically determined for the input (in other words, any two machine learning models whose output changes probabilistically for the input).

[0013] <Example of hardware configuration of verification device 10> An example of the hardware configuration of the verification device 10 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the hardware configuration of the verification device 10 according to this embodiment.

[0014] 1, a verification device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0015] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the verification device 10 does not necessarily have to have at least one of the input device 101 and the display device 102, for example.

[0016] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0017] The communication I / F 104 is an interface for connecting to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 108 is various types of arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0018] 1 is an example, and the hardware configuration of the verification device 10 is not limited to this. For example, the verification device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0019] <Example of functional configuration of verification device 10> An example of the functional configuration of the verification device 10 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the functional configuration of the verification device 10 according to this embodiment.

[0020] As shown in FIG. 2, the verification device 10 according to this embodiment includes a question creation unit 201, a model input unit 202, a model property creation unit 203, a verification unit 204, and a verification result output unit 205. Each of these units is implemented, for example, by a processor 108 or the like executing one or more programs installed in the verification device 10. The verification device 10 according to this embodiment also includes a question and correct answer storage unit 206 and a model property storage unit 207. Each of these storage units is implemented, for example, by a storage area of ​​a storage device such as the auxiliary storage device 107. However, at least one of the question and correct answer storage unit 206 and the model property storage unit 207 may be implemented by a storage area of ​​a storage device (e.g., a storage device included in a database server) communicably connected to the verification device 10.

[0021] The question creation unit 201 creates a plurality of pairs of questions that serve as input to the machine learning models T and V and correct answers (correct answers) to the questions. The question creation unit 201 also stores the plurality of pairs of questions and correct answers in the question / correct answer storage unit 206. An example of a detailed functional configuration of the question creation unit 201 will be described later.

[0022] The model input unit 202 inputs questions stored in the question / correct answer storage unit 206 to the machine learning model T. Furthermore, when an answer to a question is output from the machine learning model T, the model input unit 202 inputs a question to the machine learning model T requesting the basis for the answer.

[0023] Similarly, the model input unit 202 inputs a question stored in the question / correct answer storage unit 206 to the machine learning model V. Similarly, when an answer to a question is output from the machine learning model V, the model input unit 202 inputs a question to the machine learning model V, which asks for the basis of the answer.

[0024] The model characteristic creation unit 203 uses the answer output from the machine learning model T, the correct answer to that answer, and the basis for that answer to create model characteristics that represent the characteristics of the output distribution of the machine learning model T. In addition, the model characteristic creation unit 203 stores the model characteristic of the machine learning model T in the model characteristic storage unit 207.

[0025] Similarly, the model property creation unit 203 uses the answer output from the machine learning model V, the correct answer to that answer, and the basis for that answer to create model properties that represent the characteristics of the output distribution of the machine learning model V. In addition, the model property creation unit 203 stores the model property of the machine learning model V in the model property storage unit 207.

[0026] The verification unit 204 uses the model characteristics of the machine learning model T and the model characteristics of the machine learning model V to verify whether the machine learning model V can be considered identical to the machine learning model T. In other words, the verification unit 204 uses the model characteristics of the machine learning model T and the model characteristics of the machine learning model V to verify the identity of the machine learning model T and the machine learning model V.

[0027] The verification result output unit 205 outputs the verification result by the verification unit 204 to a predetermined output destination. The predetermined output destination may be, for example, the display device 102 such as a display, a storage area such as the auxiliary storage device 107, or another device or equipment connected to the verification device 10 so as to be able to communicate with the verification device 10.

[0028] The question and correct answer storage unit 206 stores a plurality of pairs of questions and correct answers created by the question creating unit 201 .

[0029] The model characteristic storage unit 207 stores the model characteristics created by the model characteristic creation unit 203 .

[0030] The machine learning models T and V are stored in a storage area such as the auxiliary storage device 107. However, at least one of the machine learning models T and V may be stored in a storage area such as a storage device communicatively connected to the verification device 10 (e.g., a storage device provided in an API (Application Programming Interface) server, etc.).

[0031] <<Detailed functional configuration example of the question creation unit 201>> An example of a detailed functional configuration of the question creation unit 201 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a detailed functional configuration of the question creation unit 201 according to this embodiment.

[0032] As shown in FIG. 3, the question creation unit 201 according to this embodiment includes a question type determination unit 211, a basic information acquisition unit 212, and a question proliferation unit 213.

[0033] The problem type determination unit 211 determines the type of problem (hereinafter also referred to as "problem type") to be input to the machine learning models T and V. At this time, the problem type determination unit 211 determines the type of problem that can measure the capabilities (e.g., calculation ability, cognitive ability, ability to correct erroneous sentences, etc.) and the limits of individual expression of the machine learning models T and V. Hereinafter, as an example, the problem type is assumed to be either a "calculation problem" that is a problem for measuring the limits of calculation ability, or a "sentence correction problem" that is a problem for measuring the limits of the ability to correct erroneous sentences. A calculation problem is a problem that requires a machine learning model to solve the results of arithmetic operations. On the other hand, a sentence correction problem is a problem that requires a machine learning model to solve the original correct sentence from a sentence that has been rearranged in units such as sentences, phrases, words, or characters. However, calculation problems and sentence correction problems are both examples, and various other problem types that can measure the capabilities and limits of individual expression of a machine learning model may be determined.

[0034] The basic information acquisition unit 212 acquires or generates basic information (hereinafter also referred to as "basic information") for creating questions that are input to the machine learning models T and V. In the case of calculation questions, the basic information includes two numerical values ​​(particularly random numbers) for each of the 1st to Lth digits, where L is a predetermined integer of 1 or more. On the other hand, in the case of sentence correction questions, the basic information includes sentences to be rearranged.

[0035] The problem multiplication unit 213 creates multiple problems and correct answers to those problems based on the problem type and basic information. For example, if the problem type is a "calculation problem," the problem multiplication unit 213 uses 2L numerical values ​​to create 4L problems such as "add numbers with the same number of digits," "subtract numbers with the same number of digits," "multiply numbers with the same number of digits," and "divide numbers with the same number of digits," along with correct answers for each of those 4L problems. On the other hand, for example, if the problem type is a "sentence correction problem," the problem multiplication unit 213 rearranges (obfuscates) the sentences to be rearranged at M levels of strength, thereby creating M problems such as "restore the original sentence." Note that when the problem type is a "sentence correction problem," the sentences to be rearranged are the correct answers for each problem.

[0036] Below, the multiple questions and correct answers created by the question multiplication unit 213 are referred to as {(x i ,y i )|i=1, ,N}, where x i is the i-th problem, y i is the correct answer to the i-th problem. N is the number of problems (i.e., N=4L for calculation problems, and N=M for sentence correction problems). Note that {(x i ,y i )|i=1, . . . , N} is stored in the question and correct answer storage unit 206.

[0037] <Question creation process> An example of the question creation process according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the question creation process according to this embodiment.

[0038] The question type determination unit 211 of the question creation unit 201 determines the question type (step S101). The question type determination unit 211 may determine the question type according to a user specification, or may determine a pre-specified question type. Hereinafter, as an example, it is assumed that a "calculation question" or a "sentence correction question" is determined as the question type.

[0039] The basic information acquisition unit 212 of the question creation unit 201 acquires or generates basic information for creating questions to be input to the basic machine learning models T and V (step S102). For example, if a "calculation problem" is determined as the question type in step S101, the basic information acquisition unit 212 generates two numbers (particularly, random numbers) for each of the first to Lth digits. On the other hand, for example, if a "sentence correction problem" is determined as the question type in step S101, the basic information acquisition unit 212 acquires or generates a sentence to be rearranged. A sentence is typically a character string consisting of one or more sentences, but is not necessarily limited to this and may be any character string. Furthermore, when acquiring a sentence to be rearranged, the basic information acquisition unit 212 may acquire the sentence from, for example, a website on the Internet, a server storing a corpus, or any other document server.

[0040] The question multiplication unit 213 of the question creation unit 201 generates N questions and correct answers {(x i ,y i ) |i=1, . . . , N} is created (step S103).

[0041] For example, if the problem type is determined to be "calculation problem" in step S101, the problem multiplication unit 213 generates 4L problems x using the 2L numbers generated in step S102, such as "add numbers with the same number of digits," "subtract numbers with the same number of digits," "multiply numbers with the same number of digits," and "divide numbers with the same number of digits." iAnd those 4L problems x i The correct answer is y i This creates N=4L questions and correct answers {(x i ,y i )|i=1, ,N} is created.

[0042] On the other hand, for example, when the question type is determined to be a "sentence correction question" in step S101, the question multiplication unit 213 rearranges (obfuscates) the sentences acquired or generated in step S102 at M levels of strength, thereby generating M questions x such as "return to the original sentence." i At this time, the question multiplication unit 213 randomly selects two predetermined units (e.g., sentences, phrases, words, characters, etc.) from the text, and then rearranges the text by exchanging the positions of these two units. Here, rearranging at M levels of strength means that the strength of rearrangement (strength of obfuscation) is increased in order for m=1, ,M (e.g., increasing the number of times rearrangement is performed in order for m=1, ,M). In addition, the text acquired or generated in the above step S102 is treated as a correct answer y i This will result in N=M questions and correct answers {(x i ,y i )|i=1, ,N} is created. In this case, y1==y N is.

[0043] The question multiplication unit 213 of the question creation unit 201 multiplies the N questions and correct answers {(x i ,y i )|i=1, . . . , N} is stored in the question and correct answer storage unit 206 (step S104).

[0044] <Verification process> An example of the verification process according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the verification process according to this embodiment.

[0045] The model input unit 202 inputs N questions x stored in the question / answer storage unit 206. i(i=1, , N) are input to the machine learning model T (step S201). i Answers for each of y i,T is output from machine learning model T.

[0046] The model characteristic creation unit 203 creates N problems x i The correct answer for each of i Using this, the answer y output from the machine learning model T is i,T For example, the model characteristic creation unit 203 encodes whether y i,T =y i If z i,T (1) = 1, otherwise z i,T (1) = 0. This encodes the code (z 1,T (1) ,···,z N,T (1) ) is obtained.

[0047] The model input unit 202 inputs N problems x i Solution y for each of (i=1, ,N) i,T A question asking for the basis of is input to the machine learning model T (step S203). i Answers for each of y i,T The basis for i,T is output from machine learning model T.

[0048] The model characteristic creation unit 203 creates N problems x i Answers for each of y i,T The basis for i,T For example, the model characteristic creating unit 203 encodes each basis e i,T (i=1,...,N) into a K-bit string z i,T (2) This results in a code (z 1,T (2) ,···,z N,T (2)) is obtained, where K is a predetermined integer of 1 or greater.

[0049] The model characteristic creation unit 203 uses the code obtained in the above step S202 and the code obtained in the above step S204 to create a model characteristic Z T (Step S205). The model characteristic creating unit 203 creates, for example, Z T =(z 1,T (1) ,···,z N,T (1) ,z 1,T (2) ,···,z N,T (2) ) to model characteristic Z T This model characteristic Z T represents a (K+1)N-bit string that represents the characteristics of the output distribution of machine learning model T.

[0050] The model characteristic creating unit 203 uses the model characteristic Z created in step S205. T is stored in the model characteristic storage unit 207 (step S206).

[0051] The model input unit 202 inputs N questions x stored in the question / answer storage unit 206. i (i=1, , N) are input to the machine learning model V (step S207). i Answers for each of y i,V is output from machine learning model V.

[0052] The model characteristic creation unit 203 creates N problems x i The correct answer for each of i Using this, the answer y output from the machine learning model V is i,V For example, the model characteristic creation unit 203 encodes whether y i,V =y i If z i,V (1) = 1, otherwise z i,V (1)= 0. This encodes the code (z 1,V (1) ,···,z N,V (1) ) is obtained.

[0053] The model input unit 202 inputs N problems x i Solution y for each of (i=1, ,N) i,V A question asking for the basis of is input to the machine learning model V (step S209). i Answers for each of y i,V The basis for i,V is output from machine learning model V.

[0054] The model characteristic creation unit 203 creates N problems x i Answers for each of y i,V The basis for i,V For example, the model characteristic creating unit 203 encodes each basis e i,V (i=1,...,N) into a K-bit string z i,V (2) This results in a code (z 1,V (2) ,···,z N,V (2) ) is obtained.

[0055] The model characteristic creation unit 203 uses the code obtained in the above step S208 and the code obtained in the above step S210 to create a model characteristic Z V (Step S211). The model characteristic creating unit 203 creates, for example, Z V =(z 1,V (1) ,···,z N,V (1) ,z 1,V (2) ,···,z N,V (2) ) to model characteristic Z V This model characteristic Z V represents a (K+1)N-bit string that represents the characteristics of the output distribution of machine learning model V.

[0056] The model characteristic creating unit 203 uses the model characteristic Z created in step S211. V is stored in the model characteristic storage unit 207 (step S212).

[0057] The above steps S201 to S206 and the above steps S207 to S212 may be performed in any order. That is, for example, the above steps S201 to S206 may be performed after the above steps S207 to S212 are performed.

[0058] The verification unit 204 determines the model characteristic Y T and Y V (Step S213). Here, the verification unit 204 can verify the identity of the machine learning model T and the machine learning model V using, for example, the distribution distance or a statistical hypothesis test.

[0059] When using the distribution distance Z T (1) =(z 1,T (1) ,···,z N,T (1) ), Z T (2) =(z 1,T (2) ,···,z N,T (2) ) Similarly, Z V (1) =(z 1,V (1) ,···,z N,V (1) ), Z V (2) =(z 1,V (2) ,···,z N,V (2) ) In this case, Z T (1) and Z T (2) can be considered as a distribution of N points. Similarly, ZV (1) and Z V (2) can be regarded as a distribution of N points. T (1) and Z V (1) The inter-distribution distance d between (1) , Z T (2) and Z V (2) The inter-distribution distance d between (2) Calculate each and d (1) is a predetermined threshold θ (1) Less than and d (2) is a predetermined threshold θ (2) If it is less than , the verification result is that "machine learning model V can be considered identical to machine learning model T," and if it is not, "machine learning model V cannot be considered identical to machine learning model T." Note that, for example, KL (Kullback-Leibler) divergence can be used as a measure of the distance between distributions.

[0060] When using statistical hypothesis testing Z T (1) =(z 1,T (1) ,···,z N,T (1) ), Z T (2) =(z 1,T (2) ,···,z N,T (2) ) Similarly, Z V (1) =(z 1,V (1) ,···,z N,V (1) ), Z V (2) =(z 1,V (2) ,···,z N,V (2) ) In this case, Z T (1) N values ​​z in 1,T (1) ,···,z N,T(1) can be considered as the distribution of a sample statistically extracted from a certain population. Similarly, Z T (2) N values ​​z in 1,T (2) ,···,z N,T (2) can be considered as the distribution of a sample statistically extracted from a population. V (1) and Z V (2) The same is true for .

[0061] So, Z T (1) and Z V (1) Between Z T (2) and Z V (2) A statistical hypothesis test is performed on two samples to test whether the probability distributions of the populations are different between the two. Specifically, the null hypothesis is that the probability distributions of the populations are the same, and if the p-value is smaller than the predetermined significance level, the null hypothesis is rejected. Then, Z T (1) and Z V (1) Between Z T (2) and Z V (2) If the null hypothesis is not rejected between Z and T (1) and Z V (1) Between Z T (2) and Z V (2) The alternative hypotheses between Z and Z are considered to be correct, and the verification result of the identity verification is "machine learning model V can be considered to be identical to machine learning model T." T (1) and Z V (1) Between Z T (2) and Z V (2)If the null hypothesis is rejected on either side of the two, the verification result is that "machine learning model V cannot be considered identical to machine learning model T." Note that statistical hypothesis tests for two samples can be performed using, for example, the Kolmogorov-Smirnov test or the Mann-Whitney U test.

[0062] The verification result output unit 205 outputs the verification result obtained in step S213 to a predetermined output destination (step S214).

[0063] <Modification> In the above embodiment, the basis for the answer to each question is also obtained from the machine learning model and the basis is encoded, but the acquisition and encoding of the basis does not have to be performed. That is, steps S203 to S204 and steps S209 to S210 in Fig. 5 do not have to be performed. This reduces the accuracy of identity verification, but enables simple and easy identity verification.

[0064] <Summary> As described above, when verifying the identity of multiple machine learning models, the verification device 10 according to this embodiment creates, as input to each machine learning model, questions that can measure the capabilities of the machine learning model and the limits of its individual expression. Furthermore, the verification device 10 according to this embodiment can create a sufficient number of questions as input to each machine learning model. This allows different output distributions for different machine learning models, making it possible to efficiently and accurately verify the identity of multiple machine learning models. Therefore, by using the verification device 10 according to this embodiment, it becomes possible to efficiently and accurately detect machine learning models that may have been altered by erroneous learning or illegally replaced by a third party, for example, targeting machine learning models that realize generative AI.

[0065] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of symbols]

[0066] 10 Verification equipment 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 201 Question Creation Department 202 Model Input Section 203 Model characteristics creation unit 204 Verification Department 205 Verification result output section 206 Question and Answer Memory 207 Model characteristics memory unit 211 Problem Type Determination Unit 212 Basic information acquisition department 213 Problem Growth Department

Claims

1. A verification device that verifies identity between a first machine learning model and a second machine learning model, comprising: a question creation unit that creates a plurality of questions to be input to the first machine learning model and the second machine learning model; an input unit that inputs the plurality of problems to each of the first machine learning model and the second machine learning model; a model characteristic creation unit that creates first model characteristics that represent characteristics of an output distribution of the first machine learning model and second model characteristics that represent characteristics of an output distribution of the second machine learning model, using the correctness or incorrectness of the answers of the first machine learning model to each of the plurality of problems and the correctness or incorrectness of the answers of the second machine learning model to each of the plurality of problems, respectively; a verification unit that verifies the identity of the first machine learning model and the second machine learning model using the first model characteristics and the second model characteristics; A verification device having the above configuration.

2. The model characteristic creation unit 2. The verification device according to claim 1, further using a basis for the answer of the first machine learning model for each of the plurality of problems and a basis for the answer of the second machine learning model for each of the plurality of problems to create the first model characteristics and the second model characteristics, respectively.

3. The question creation unit: The verification device according to claim 1 , wherein the plurality of questions are created to measure the limits of at least one of capabilities and individual expression of the first machine learning model and the second machine learning model.

4. a verification device that verifies identity between the first machine learning model and the second machine learning model, a question creation step of creating a plurality of questions to be input to the first machine learning model and the second machine learning model; an input step of inputting the plurality of problems to each of the first machine learning model and the second machine learning model; a model characteristic creation step of creating first model characteristics representing characteristics of an output distribution of the first machine learning model and second model characteristics representing characteristics of an output distribution of the second machine learning model, using the correctness or incorrectness of the answers of the first machine learning model to each of the plurality of problems and the correctness or incorrectness of the answers of the second machine learning model to each of the plurality of problems, respectively; a verification step of verifying the identity of the first machine learning model and the second machine learning model using the first model characteristics and the second model characteristics; The validation method to perform.

Images