MODEL VERIFICATION APPARATUS, MODEL VERIFICATION METHOD, AND PROGRAM

The model verification device addresses the limitations of existing methods by verifying neural networks' robustness to small text variations using mixed integer programming, ensuring reliable text classification outcomes.

JP7584093B2Active Publication Date: 2024-11-15NIPPON TELEGRAPH & TELEPHONE CORP +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2021084645
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-05-19
Publication Date
2024-11-15
Estimated Expiration
2041-05-19

AI Technical Summary

Technical Problem

Existing machine learning models, particularly deep learning models for tasks beyond image identification, face limitations in handling small variations in text classification and nonlinear functions, making them vulnerable to minute changes that can alter their output significantly.

Method used

A model verification device that uses a mixed integer programming solver to verify the robustness of neural networks for text discrimination by adding small variations to intermediate vectors, considering linear and ReLU functions, ensuring the output remains unchanged.

Benefits of technology

The device accurately determines the robustness of neural networks to small input changes, enhancing the reliability of text discrimination tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584093000007
    Figure 0007584093000007
  • Figure 0007584093000008
    Figure 0007584093000008
  • Figure 0007584093000009
    Figure 0007584093000009
Patent Text Reader

Abstract

To verify robustness against minute change of input in a machine learning model in which natural language sentences are set as the input.SOLUTION: A model verification device which verifies robustness of a neural network which discriminates texts comprises: the neural network; parameters showing minute fluctuation; an input unit which inputs data becoming the input to the neural network; a verification processing execution unit which verifies whether or not a class outputted by the neural network is changed, when the minute fluctuation is added to a vector of an intermediate layer of the neural network to which the data were inputted; and a verification result output unit which outputs information indicating whether or not the neural network is robust against the minute change of the input, on the basis of a verification result due to the verification processing execution unit.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for verifying the robustness of a machine learning model. [Background technology]

[0002] Machine learning models, especially deep learning models using neural networks, are used for various tasks due to their high classification performance. However, it is known that deep learning models used for object recognition, for example, to recognize objects in images, can have a phenomenon in which the classification results can change significantly by adding minute noise to the input image that is unnoticeable to the human eye. If the classification results change due to minute noise, it becomes possible to control the behavior of the object system by applying minute processing to the input that is unnoticeable to the human eye, which creates a security problem.

[0003] To address this issue, techniques for validating machine learning models have been proposed that verify whether the output of a neural network model for image recognition remains unchanged even when the input is slightly changed.

[0004] As a technology for verifying the above-mentioned machine learning model, a technology called MIPVerify is disclosed in Non-Patent Document 1. In MIPVerify, in a deep learning model for image recognition, whether the output does not change with minute changes in the input (hereinafter, the fact that the output does not change with minute changes is referred to as robustness) is formulated as a mixed integer programming problem (MIP) and judged by solving this problem using a mathematical programming solver. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Vincent Tjeng, Kai Xiao, Russ Tedrake, "Evaluating Robustness of Neural Networks with Mixed Integer Programming", in Proc. of ICLR 2019 [Non-Patent Document 2] Jacob Devlin, Ming-wei Chang, Kenton Lee, Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in Proc. of NAACL 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] MIPVerify was a method that was intended to be used to verify image recognition models. However, deep learning models are used for a variety of tasks other than image recognition. For example, deep learning models are used to solve text recognition problems, in which a sentence written in natural language is input and the category or characteristics of the sentence are identified. However, when applying MIPVerify to a model for text recognition, there was an issue that the types of minute fluctuations that could be handled were limited.

[0007] The input for the text classification model is a sentence, i.e., a sequence of discrete words, and when using MIPVerify, it was only possible to handle changes such as replacing words in a sentence or characters in a word. Following the example of the image, it would be desirable to be able to handle small changes in text, such as changing particles or word order so that the meaning of the sentence remains almost unchanged, but replacing words or characters can significantly change the meaning of the sentence, so the fluctuations are no longer small. Another issue was that the types of nonlinear functions that MIPVerify could handle were limited.

[0008] A neural network expresses a function by hierarchically combining various linear and nonlinear functions. MIPVerify uses the activation function ReLU (Rectified Linear Unit), which is expressed by the following formula:

[0009]

number

[0010] The present invention has been made in consideration of the above points, and aims to provide a technique for verifying the robustness of a machine learning model that uses a natural language sentence as input to the model against small changes in the input. [Means for solving the problem]

[0011] According to the disclosed technology, there is provided a model verification device for verifying robustness of a neural network that performs text discrimination, comprising: an input unit for inputting the neural network, a parameter representing a minute variation, and data to be input to the neural network; a verification process execution unit that verifies whether or not a class output by the neural network changes when a small variation is applied to a vector in an intermediate layer of the neural network to which the data is input; a verification result output unit that outputs information indicating whether the neural network is robust to a small change in input based on a verification result by the verification process execution unit; The neural network is a composite function of the form h(g(·)), where h(·) is a function consisting only of linear functions, ReLUs, and maximization functions. A model validation apparatus is provided. Effect of the Invention

[0012] The disclosed technology provides a technology for verifying the robustness of a machine learning model that uses a natural language sentence as input to the model against small changes in the input. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram of a model verification apparatus according to an embodiment of the present invention. [Diagram 2] 13 is a flowchart illustrating an operation of the model verification apparatus. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0015] (Outline of the embodiment) First, an overview of the technology according to the present embodiment will be described.

[0016] In MIPVerify disclosed in Non-Patent Document 1, the robustness of a deep learning model when a small change is made to the input layer is judged by a mathematical programming solver, whereas in the model verification device according to this embodiment, a small change is made to an intermediate layer composed of a set of vectors obtained by applying some linear or non-linear transformations to the input, and a mathematical programming solver is used to verify whether the change changes the discrimination result. "The discrimination result does not change" means that there is no change compared to the discrimination result when no small change is made to the intermediate layer.

[0017] The vectors in the intermediate layer of a deep learning model are considered to be a replacement of a word in an input sentence with a real-valued vector that represents its meaning. It is also known that similar word vectors are obtained for words with similar meanings. Therefore, making a small change to the vector in the intermediate layer corresponds to a change equivalent to replacing a word in an input sentence with another word that has a similar meaning to that word, so it is suitable for the purpose of observing the change when making a small change. In addition, even if a deep learning model contains various nonlinear functions, as long as it does not contain nonlinear functions other than ReLU and max between the intermediate layer that adds the change and the output layer, it is possible to use the solution method using the mathematical programming solver used in MIPVerify.

[0018] Deep learning models for text discrimination often incorporate models that convert an input sentence (a string of words) into a string of real vectors, such as BERT (see Non-Patent Document 2), and such models use a variety of non-linear functions. However, by regarding the set of real vectors obtained by inputting a sentence into BERT as an intermediate layer and performing verification by making small changes to the intermediate layer, it is possible to apply the technology of the present invention to models that use BERT.

[0019] The following describes the details in more detail using examples.

[0020] (Example) <Explanation of the robustness of neural networks> First, we explain the robustness of the neural network. We define a neural network that classifies a given sentence (called a "text classification model") as f(·):[0,1] n →R m Here, n is the number of dimensions of the input to the neural network, and m is the total number of classes to be discriminated. In this specification, the arrow "→" representing a mapping is intended to mean an arrow with a short vertical bar attached to the left end. For the convenience of description in the specification, "→" is used.

[0021] If the input is a string of words, let N be the number of words in the sentence, and V be the number of types of words handled by the text discrimination model, so n = NV. If the input is a string of characters, let N be the total number of characters, and V be the number of types of characters.

[0022] In this embodiment, the neural network f is a set of two neural networks g(·):[0,1] n →R k and h(·):R k →R m Let us assume that the input to the neural network f is x∈[0,1]. Let f(·)=h(g(·)) be a composite function. Let h(·) consist of only linear functions, ReLU, and maximization functions. Let the input to the neural network f be x∈[0,1]. n Let z∈R k , z = g(x) is the intermediate vector, and y ∈ R m , y=h(z) is the output vector.

[0023] Vector z´∈R obtained by adding a small variation to the intermediate vector z k Let U(z)={z´|∀i:-ε≦(zz´) i ≦ε. Note that "adding a small variation to the intermediate vector z" is equivalent to "adding a small variation to the vectors in the intermediate layer."

[0024] The ε in the above is a non-negative real number and represents the small fluctuation to be allowed. (zz´) i represents the i-th element of the vector (zz´). The i-th element of h(z) is i Let λ(x) be the class that is output when input x is given to neural network f (the class when no small fluctuations are added). For convenience, λ(x) may be called the "correct class." A neural network is said to be robust against small fluctuations in the input x and the hidden layer, or against small fluctuations in the input x, if

[0025]

number

[0026] A model verification device 100 described below verifies whether a neural network f is robust to small input variations by verifying whether a class output by the neural network f changes when a small variation is made to a vector in the intermediate layer of the neural network f to which x is input. An example of the configuration and operation of the model verification device 100 will be described below.

[0027] <Equipment configuration example> Fig. 1 shows a configuration diagram of a model verification device 100 according to this embodiment. As shown in Fig. 1, the model verification device 100 has an NN model input section 110, a verification process execution section 120, and a verification result output section 130. An overview of each section is as follows.

[0028] The NN model input unit 110 receives a NN (neural network) model f for text discrimination, an input x, and a parameter ε from the outside. The verification process execution unit 120 verifies the robustness of the input neural network using a mathematical programming problem solver. The verification result output unit 130 outputs the verification result by the verification process execution unit 120.

[0029] If the model to be verified is robust, the verification result output unit 130 outputs information indicating that the model is robust as a verification result. If the model is not robust, that is, if the class output by applying a small change to the intermediate vector changes compared to the output class when no small change is applied, the verification result output unit 130 outputs one intermediate vector z' to which such a small change is applied.

[0030] Note that, when there are multiple intermediate vectors z' in which the class has been changed, multiple vectors may be output. Also, when there is an intermediate vector z' in which the class has been changed, information indicating that the model is not robust may be output. Outputting the intermediate vector z' in which the class has been changed is an example of outputting information indicating that the model is not robust.

[0031] <Example of device operation> An example of the processing operation of the model verification device 100 will be described with reference to the flowchart of FIG.

[0032] In S1 (step 1), the NN model input unit 110 receives a neural network to be verified, a parameter ε, and an input x. Here, it is assumed that the neural network to be verified satisfies f(·)=h(g(·)). In S2, the verification process execution unit 120 executes the verification process. The verification process execution unit 120 executes the verification process by solving the following mixed integer programming problem.

[0033]

number

[0034]

number

[0035]

number

[0036]

number

[0037] If h(·) consists only of a linear function, ReLU, and a maximization function, this problem can be expressed as a mixed integer programming problem using the method described in Non-Patent Document 1, and a solution can be found using a mixed integer programming problem solver.

[0038] If a feasible solution is found as a result of solving the problem using the solver in S2, then in S3 the verification result output unit 130 determines that the neural network is not robust, outputs the intermediate vector z' found by solving the problem, and ends the process. If a feasible solution is not found, then the verification result output unit 130 outputs information indicating that the neural network is robust, and ends the process.

[0039] (Example of device hardware configuration) The model verification device 100 can be realized, for example, by causing a computer to execute a program describing the processing contents described in this embodiment. More specifically, the program includes, for example, a mathematical programming problem solver program for solving a mixed integer programming problem.

[0040] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. The above program can also be provided via a network such as the Internet or e-mail.

[0041] Fig. 3 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 3 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus BS.

[0042] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0043] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0044] For example, the neural network to be verified (including g(·), h(·)), parameters ε, and input data x are input from interface device 1005, and these are stored in memory device 1003. Then, CPU 1004 operates according to the program to solve the problem expressed by the above equations (3) to (6) from the functions, parameters, and input x stored in memory device 1003.

[0045] (Effects of the embodiment) The model verification device 100 according to this embodiment can accurately determine whether a neural network for discriminating text is robust to minute input variations, thereby contributing to the realization of a safer text discrimination model.

[0046] (Summary of the embodiment) This specification describes at least the following model verification apparatus, model verification method, and program. (Section 1) A model verification device for verifying robustness of a neural network that performs text discrimination, comprising: an input unit for inputting the neural network, a parameter representing a minute variation, and data to be input to the neural network; a verification process execution unit that verifies whether or not a class output by the neural network changes when a small variation is applied to a vector in an intermediate layer of the neural network to which the data is input; a verification result output unit that outputs information indicating whether the neural network is robust to a small change in input based on a verification result by the verification process execution unit; A model verification apparatus comprising: (Section 2) The verification process execution unit solves a mixed integer programming problem to find a vector with a small variation that changes the class output by the neural network. 2. The model verification apparatus according to claim 1. (Section 3) If the vector to which the minute variation has been added is not found, the verification result output unit outputs information indicating that the neural network is robust against minute changes in the input. 3. The model verification apparatus according to claim 2. (Section 4) The neural network is a composite function of the form h(g(·)), where h(·) is a function consisting only of linear functions, ReLUs, and maximization functions. 4. The model verification apparatus according to claim 1, (Section 5) A model verification method executed by a model verification device for verifying robustness of a neural network that performs text discrimination, comprising the steps of: an input step of inputting the neural network, parameters representing minute variations, and data to be input to the neural network; a verification process execution step for verifying whether or not a class output by the neural network changes when a small variation is applied to a vector in an intermediate layer of the neural network to which the data is input; a verification result output step of outputting information indicating whether the neural network is robust to a small change in input based on a verification result by the verification processing execution step; The model validation method includes: (Section 6) A program for causing a computer to function as each unit in the model verification apparatus according to any one of claims 1 to 4.

[0047] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0048] 100 Model verification device 110 NN model input section 120 Verification Processing Execution Unit 130 Verification result output section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device

Claims

1. A model verification device for verifying robustness of a neural network that performs text discrimination, comprising: an input unit for inputting the neural network, a parameter representing a minute variation, and data to be input to the neural network; a verification process execution unit that verifies whether or not a class output by the neural network changes when a small variation is applied to a vector in an intermediate layer of the neural network to which the data is input; a verification result output unit that outputs information indicating whether the neural network is robust to a small change in input based on a verification result by the verification process execution unit; The neural network is a composite function of the form h(g(·)), where h(·) is a function consisting of only linear functions, ReLU, and maximization functions. Model validation equipment.

2. The verification process execution unit solves a mixed integer programming problem to find a vector with a small variation that changes the class output by the neural network. The model verification apparatus according to claim 1 .

3. If the vector to which the minute variation has been added is not found, the verification result output unit outputs information indicating that the neural network is robust against minute changes in the input. The model verification apparatus according to claim 2 .

4. A model verification method executed by a model verification device for verifying robustness of a neural network that performs text discrimination, comprising the steps of: an input step of inputting the neural network, a parameter representing a minute variation, and data to be input to the neural network; a verification process execution step for verifying whether or not a class output by the neural network changes when a small variation is applied to a vector in an intermediate layer of the neural network to which the data is input; a verification result output step of outputting information indicating whether the neural network is robust to a small change in input based on a verification result by the verification processing execution step, The neural network is a composite function of the form h(g(·)), where h(·) is a function consisting of only linear functions, ReLU, and maximization functions. Model validation methods.

5. A program for causing a computer to function as each unit in the model verification apparatus according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Adversarial training method and device of machine learning model, electronic equipment and medium

    CN112580822A