Unknown data detection device, method, and program

Abstract

To detect unknown data with a low memory capacity.SOLUTION: An unknown data detector comprises: an acquisition unit that acquires monitoring target data; an intermediate output calculation unit that applies a trained model to the monitoring target data to calculate an intermediate output; a projection component calculation unit that calculates a projection component of the intermediate output onto parameters constituting the trained model; and a determination unit that determines whether the monitoring target data is unknown data based on the projection component.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an unknown data detection device, method, and program. [Background technology]

[0002] The performance of machine learning depends not only on the model used, but also significantly on the training dataset and the dataset used in operation. For example, if the input data distribution changes due to the system's operating conditions, the trained model may not perform as initially expected due to the difference in the dataset. As time passes and the input data distribution changes from the training data distribution, performance degradation progresses. In particular, with deep learning models, which have seen a rapid expansion of applications in recent years, it has been reported that they exhibit behavior similar to the training data even when dealing with datasets that are completely different from the training data (OOD: Out-Of-Distribution). For example, in a deep neural network (DNN) model trained on a classification task, the classification probability for each class should ideally be low for OOD samples. However, in reality, it obtains a classification probability that is not significantly different from that of the training data, making it difficult to detect unknown data.

[0003] Various approaches have been taken to achieve more accurate OOD detection performance. Non-patent document 1 approximates the intermediate outputs from each intermediate layer of the model with a Gaussian distribution when training data is input, and performs OOD detection using the Mahalanobis distance from the center of each class as an indicator. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin, “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks,” in Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018).

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Non-Patent Document 1, after performing preprocessing to approximate the intermediate output of learning data with a multivariate normal distribution, the difference from the learning data distribution is evaluated based on the Mahalanobis distance. Calculating the Mahalanobis distance requires an average vector and a covariance matrix, and the memory cost required to ensure them is proportional to the square of the dimension of the feature map, so a certain amount of computing resources is required. Furthermore, in the case of, for example, a model trained for a class classification task or a network structure using convolutional layers, individual evaluations for each class and an increase in the number of dimensions according to the receptive field of the convolutional kernel occur. The increase in computational cost due to the details of these tasks is secondary, but the amount of increase can be on the order of 10 to 100 times and cannot be ignored. Therefore, it is important to reduce the computational cost by evaluation that strengthens more general aspects with less dependence on task details.

[0006] The problem to be solved by the present invention is to provide an unknown data detection device, method, and program capable of detecting unknown data with a low memory capacity.

Means for Solving the Problems

[0007] The unknown data detection device according to the embodiment comprises: an acquisition unit that acquires data to be monitored; an intermediate output calculation unit that calculates an intermediate output by applying a trained model to the data to be monitored; a projection component calculation unit that calculates the projection component of the intermediate output onto the parameters constituting the trained model; and a determination unit that determines whether the data to be monitored is unknown data based on the projection component. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of the unknown data detection device according to this embodiment. [Figure 2] This figure shows an example of the configuration of a noise impact evaluation unit, which evaluates the degree of noise impact by adding noise. [Figure 3] This figure shows an example of the configuration of a noise impact evaluation unit related to a method of evaluating the degree of noise impact by calculating projection components. [Figure 4] A schematic diagram illustrating the processing of unknown data. [Figure 5] A diagram showing an example of how to handle the evaluation phase. [Figure 6] This diagram shows the relationship between the network structure of a trained model and its intermediate output. [Figure 7] This diagram schematically illustrates the data processing of the noise impact evaluation unit in a system that evaluates the degree of noise impact by adding noise. [Figure 8] This diagram schematically illustrates the data processing of the noise impact evaluation unit in a system that evaluates the degree of noise impact by calculating projection components. [Figure 9] This figure shows an example of processing in the pre-treatment phase. [Figure 10] A schematic diagram illustrating the unknown data detection process related to Application Example 1. [Figure 11] Diagram showing the relationship between layer depth and stability rank in Application Example 2. [Figure 12] This figure shows the performance metrics for each method of detecting unknown data (OOD). [Figure 13] This figure shows the memory usage of the method using Mahalanobis distance and the method according to this embodiment. [Modes for carrying out the invention]

[0009] The unknown data detection device, method, and program according to this embodiment will be described below with reference to the drawings.

[0010] Figure 1 shows an example configuration of the unknown data detection device 100 according to this embodiment. As shown in Figure 1, the unknown data detection device 100 is a computer having a processing circuit 1, a storage device 2, an input device 3, a communication device 4, and a display device 5. Data communication between the processing circuit 1, the storage device 2, the input device 3, the communication device 4, and the display device 5 is performed via a bus.

[0011] The processing circuit 1 includes a processor such as a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory). The processing circuit 1 includes an acquisition unit 11, an intermediate output calculation unit 12, a noise impact evaluation unit 13, a discrimination unit 14, and an output control unit 15. The processing circuit 1 realizes each of the above units 11 to 15 by executing an unknown data detection program. The unknown data detection program is stored in a non-temporary computer-readable recording medium such as a storage device 2. The unknown data detection program may be implemented as a single program describing all the functions of the above units 11 to 15, or it may be implemented as multiple modules divided into several functional units. Furthermore, the above units 11 to 15 may be implemented by integrated circuits such as Application Specific Integrated Circuits (ASICs). In this case, they may be implemented in a single integrated circuit, or they may be implemented individually in multiple integrated circuits.

[0012] The acquisition unit 11 acquires a trained model. The trained model is a deep learning model whose parameters have been trained to perform any task. The task of the trained model is not particularly limited and may perform regression problems, classification problems, or any other task. The network structure of the trained model is also not particularly limited. The acquisition unit 11 also acquires monitored data. The monitored data is data to be determined as to whether or not it is unknown data. The type and format of the data are not particularly limited and may be any type and format as long as it can be input into the trained model. In this embodiment, unknown data means data that has properties that make it statistically indistinguishable from the data used to train the trained model.

[0013] The intermediate output calculation unit 12 calculates the intermediate output by applying the trained model acquired by the acquisition unit 11 to the monitored data acquired by the acquisition unit 11. The intermediate output is the output of the hidden layer preceding the output layer of the trained model.

[0014] The noise impact evaluation unit 13 calculates the degree of influence when noise is added to the intermediate output calculated by the intermediate output calculation unit 13, using the parameters that constitute the trained model acquired by the acquisition unit 11. More specifically, the noise impact evaluation unit 13 calculates the degree of change in the output of the subsequent hidden layer before and after adding a small amount of noise to the intermediate output calculated by the intermediate output calculation unit 13. The noise impact evaluation unit 13 in this embodiment has two forms: one that evaluates the degree of noise impact by adding noise, and another that evaluates the degree of noise impact by calculating the projection component.

[0015] Figure 2 shows an example configuration of the noise impact evaluation unit 13, which evaluates the degree of noise impact by adding noise. As shown in Figure 2, the noise impact evaluation unit 13 has a noise addition unit 111 and a change degree calculation unit 112. The noise addition unit 111 adds noise to the intermediate output of an intermediate hidden layer of the trained model. The change degree calculation unit 112 calculates a first intermediate output by applying a hidden layer later than the intermediate hidden layer to the intermediate output that has not had noise added. The change degree calculation unit 112 also calculates a second intermediate output by applying a later hidden layer to the intermediate output that has had noise added. The change degree calculation unit 112 then calculates the degree of change between the first intermediate output and the second intermediate output as the noise impact.

[0016] Figure 3 shows an example of the configuration of a noise impact evaluation unit 13 in which the degree of noise impact is evaluated by calculating projection components. As shown in Figure 3, the noise impact evaluation unit 13 has a projection matrix determination unit 121 and a projection component calculation unit 122. The projection matrix determination unit 121 determines a projection matrix based on the matrix decomposition of the parameters constituting the trained model. The projection component calculation unit 122 calculates the projection components of the intermediate output to the parameters constituting the trained model as the degree of noise impact. Specifically, the projection component calculation unit 122 calculates the projection components by applying the projection matrix determined by the projection matrix determination unit 121 to the intermediate output.

[0017] The discrimination unit 14 determines whether the monitored data acquired by the acquisition unit 11 is unknown data or not, based on the degree of noise influence calculated by the noise influence evaluation unit 13.

[0018] The output control unit 15 outputs the determination result of whether the monitored data by the discrimination unit 14 is unknown data or not. The determination result may be displayed on the display device 5, stored in the storage device 2, or transmitted to another computer via the communication device 4. The output control unit 15 may also display other arbitrary information on the display device 5 or the like.

[0019] Storage device 2 consists of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), integrated circuit storage devices, etc. Storage device 2 stores data to be monitored, trained models, unknown data detection programs, etc.

[0020] Input device 3 receives various commands from the user. Input device 3 can include a keyboard, mouse, various switches, touchpad, touch panel display, etc. Output signals from input device 3 are supplied to processing circuit 1. Input device 3 may also be an input device of a computer connected to processing circuit 1 via wired or wireless connection.

[0021] Communication device 4 is an interface for data communication with external devices connected to the unknown data detection device 100 via a network.

[0022] The display device 5 displays various types of information. For example, the display device 5 displays structural and performance-related data according to the control of the output control unit 16. The display device 5 can be a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art. The display device 5 may also be a projector.

[0023] The unknown data detection device 100 according to this embodiment will be described in detail below.

[0024] The unknown data detection device 100 processes the data during the evaluation phase by determining whether the monitored data is unknown data or not based on the degree of noise influence.

[0025] Figure 4 is a schematic diagram illustrating the processing of unknown data. Figure 5 is a diagram illustrating the processing procedure for unknown data. As shown in Figures 4 and 5, the acquisition unit 11 acquires the trained model 201 (step S301). The trained model 201 has training parameters based on multiple training data. The training parameters include weight parameters and biases related to the transformations between the layers that make up the trained model 201. In addition, the trained model 201 has activation functions assigned to each node or channel as hyperparameters. Other hyperparameters include the size of the convolutional kernel receptive field and batch normalization parameters.

[0026] The trained model 201 may be stored in the storage device 2 beforehand. In this case, the acquisition unit 11 reads the trained model 201 from the storage device 2. As another example, the acquisition unit 11 may receive the trained model 201 from another computer via the communication device 4.

[0027] When step S301 is performed, the acquisition unit 11 acquires the monitored data 202 (step S302). The monitored data 202 may be stored in the storage device 2 beforehand. In this case, the acquisition unit 11 reads the monitored data 202 from the storage device 2. As another example, the acquisition unit 11 may receive the monitored data 202 from another computer via the communication device 4.

[0028] When step S302 is performed, the intermediate output calculation unit 12 applies the monitored data 202 acquired in step S302 to the trained model 201 acquired in step S301 to calculate the intermediate output 203 (step S303).

[0029] Figure 6 shows the relationship between the network structure of a trained model and its intermediate output. As shown in Figure 6, the trained model has an input layer, multiple hidden layers, and an output layer. The number of hidden layers can be any number, as long as there is one or more layers. The input layer receives the data to be monitored, etc. The output layer outputs the final output. The data format of the final output differs depending on the task of the trained model. For example, if the task of the trained model is a classification problem, the final output will be the classification result, i.e., the probability of each class being correct.

[0030] Multiple hidden layers are provided between the input layer and the output layer. Various network layers such as convolutional layers, fully connected layers, batch normalization layers, and pooling layers can be used as each hidden layer. The output of each hidden layer is the intermediate output. The output of a hidden layer is also called a feature vector. There is no particular limitation on which intermediate output of a hidden layer is used as intermediate output 203, and it can be set arbitrarily. Furthermore, instead of the intermediate output of a single hidden layer, the intermediate output 203 may be a concatenation of multiple intermediate outputs from multiple hidden layers.

[0031] When step S303 is performed, the noise impact evaluation unit 13 calculates the degree of noise impact 205 on the intermediate output 203 calculated in step S303 (step S304).

[0032] Several methods can be considered for calculating the degree of noise impact. One is the method using the noise impact evaluation unit 13 shown in Figure 2, and the other is the method using the noise impact evaluation unit 13 shown in Figure 3.

[0033] FIG. 7 is a diagram showing a procedure for calculating the degree of influence of noise by the noise influence evaluation unit 13 shown in FIG. 2. As shown in FIG. 7, the intermediate output 203 calculated in step S303 has been calculated. The intermediate output 203 is output from an intermediate hidden layer of the learned model 201 as described above. The degree of change calculation unit 112 applies the hidden layer 201B after the intermediate hidden layer of the learned model 201 to the intermediate output 203 without noise applied thereto to calculate a first intermediate output 211 (step S312). In parallel with this, the noise application unit 111 directly applies minute noise to the intermediate output 203 (step S311). Next, the degree of change calculation unit 112 applies the hidden layer 201B after the above to the intermediate output 203 without noise applied thereto to calculate a second intermediate output 212 (step S312). Then, the degree of change calculation unit 112 calculates the degree of change between the first intermediate output 211 and the second intermediate output 212 as the degree of influence of the noise. As the degree of change, the difference between the first intermediate output 211 and the second intermediate output 212 may be calculated. Note that the degree of change is not limited to only the difference between the first intermediate output 211 and the second intermediate output 212, and may be the difference with an arbitrary numerical value or function added thereto, or may be the ratio of the second intermediate output 212 to the first intermediate output 211 or the like.

[0034] As an example, the degree of change ψ can be calculated according to the following formula. l and m represent hidden layers. The hidden layer l represents the hidden layer from which the intermediate output 203 is output. The hidden layer m represents the hidden layer from which the first intermediate output 211 and the second intermediate output 212 are output. x l represents the intermediate output 203 output from the hidden layer l before the noise η is applied. M lm (x l ) represents the first intermediate output 211 output from the hidden layer m by inputting the intermediate output x l to the hidden layer l. M lm (x l + η || x l ||) represents the second intermediate output 212 output from the hidden layer m by inputting the intermediate output x l + η || x l || with the noise η applied thereto to the hidden layer l.

[0035]

number

[0036] The number of hidden layers in the latter stage described above can be one or multiple. The noise to be added can be generated from an isotropic probability distribution, an anisotropic probability distribution, the intermediate output 204 provided by the training data, or the intermediate output 204 provided by the accessible public data. Furthermore, steps S311 and S312, which involve adding noise and calculating its impact, can be performed multiple times to obtain multiple intermediate outputs 212, and the difference between these outputs and the original intermediate output 211 can be calculated in step S313. Adding noise incurs no memory cost, making this a memory-efficient method. However, if noise is added multiple times, the computation time will increase.

[0037] Figure 8 shows the procedure for calculating the noise impact by the noise impact evaluation unit 13 shown in Figure 3. As shown in Figure 8, the noise impact on the intermediate output 203 may be expressed using a projection component corresponding to the alignment between the parameters of the trained model 201 and the intermediate output 203. The projection component may be calculated by calculating the projection component for a single layer, as in the case of directly calculating the noise impact, or by calculating the projection component across multiple layers.

[0038] The projection matrix determination unit 121 converts the parameters constituting the trained model 201 into a projection matrix 221 (step S321). In step S321, the projection matrix determination unit 121 calculates the projection matrix 221 by matrix decomposition of the weight parameters constituting the trained model 201. Next, the projection component calculation unit 122 applies the projection matrix 221 to the intermediate output 203 to calculate the projection component as a noise influence 204 (step S322).

[0039] Here, we will explain the significance of projecting the intermediate output 203 onto the parameters of the trained model 201. As a premise, if the input data x is equivalent to the training data, then the projection component of the intermediate output f onto the weight parameters W is large (see Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang, “Stronger Generalization Bounds for Deep Nets via a Compression Approach,” in Proceedings of the 35th International Conference on Machine Learning, PMLR, vol. 80, pp. 254-263, 2018). Therefore, if the projection component of the intermediate output f onto the weight parameters W is small, then the input data x is different from the training data, that is, the input data x is unknown data.

[0040] Furthermore, it has been demonstrated that stability against noise is achieved when the input data x is equivalent to the training data, and especially in situations with low noise, the evaluation by projection becomes equivalent to the evaluation of the impact of noise. Theoretically, strictly calculating the impact of noise requires an infinite number of noise additions, but evaluation by projection allows for a rigorous evaluation result without adding noise. In other words, it is a method that can obtain reliable results with little computation time. On the other hand, as will be discussed later, calculating the projection component requires a projection matrix, which may increase the required memory cost.

[0041] The projection components spanning multiple layers may also be calculated by linearly approximating the intermediate output obtained in the mth layer with the intermediate output obtained in the lth layer. Through linear approximation, the feature vector obtained in the mth layer is given in the form of applying a matrix to the feature vector obtained in the lth layer. This matrix is ​​uniquely determined given the parameters of the trained model 201 and the intermediate output 203. By performing matrix decomposition on the obtained matrix, the projection matrix 221 can be calculated. That is, by decomposing the matrix obtained by linear approximation (step S321), the projection matrix 221 can be calculated. By applying the projection matrix 221 to the intermediate output 203, the projection component corresponding to the noise impact, i.e., the noise impact 204, can be calculated. In this way, the noise impact can also be calculated based on projection.

[0042] The matrix decomposition method is not particularly limited, and methods such as singular value decomposition (SVD) and non-negative matrix factorization (NMF) can be used. In this embodiment, as an example, singular value decomposition is used as the matrix decomposition.

[0043] As mentioned above, projection components can be calculated across multiple layers, or a single projection component can be calculated with a noise influence of 204. In that case, only the weight parameters are used as the parameters to be transformed into the projection matrix 221. The weight parameters are transformed into a matrix by arranging the weight parameters for each layer according to a predetermined rule. For example, the weight parameters for a fully connected layer connecting the i-th and i+1-th layers are the number of channels C in the i-th layer. i , and the number of channels in the i+1 layer C i+1 Using this, the number of row components C i × number of column components C i+1 It is transformed into a matrix. Also, for example, the weight parameters of the convolutional layer connecting the i-th and i+1-th layers are, in addition to the above, the size F of the kernel receptive field of the i-th layer. i Using this, the number of row components (C i ×F i ) x number of column components C i+1is converted into a matrix.

[0044] When calculating the noise influence degree 204 using projection, the noise influence evaluation unit 13 calculates the noise influence degree 204 based on the projection matrix obtained by performing matrix decomposition on the weight parameters constituting the learned model 201. When using singular value decomposition, the weight parameter W is decomposed into USV T and converted. The weight parameter W is a matrix that defines the conversion rule of the intermediate output 204 over one or more layers configured by the above method. U is a matrix of left singular vectors, V is a matrix of right singular vectors, and S is a diagonal matrix of singular values. T represents transpose. The projection matrix 202 is V T and means a matrix representing the projection onto the weight parameter W.

[0045] As described above, the projection matrix 221 is the right singular vector V T when the weight parameter W is decomposed into USV T in conversion. More specifically, first, the layer number l before projection and the layer number m after projection are specified, and the weight parameter W (lm) is calculated. Here, l and m satisfy 1 ≤ l < m ≤ L. L is a natural number of 1 or more. Next, W (lm) is decomposed into U (lm) S (lm) V (lm)T in conversion. The projections matrices V (12)T V (13)T , ···, V (L-2,L―1)T and V (L―1,L)T obtained for various l and m can be used as the combined projection matrix V T . The projection component calculation unit 14 operates the projection matrix V T on the intermediate output f to calculate the projection component f p = V T f. Since the projection matrix V T is the orthonormal basis of the weight parameter W, the projection component f p means the projection of the intermediate output f in the principal component direction of the weight parameter W.

[0046] When step S304 is performed, the discrimination unit 14 determines whether the monitored data 202 acquired in step S302 is unknown data or not, based on the noise impact 204 calculated in step S304 (step S305). In step S305, the discrimination result 205 of whether the monitored data 202 is unknown data or not is output.

[0047] Various methods can be used to identify unknown data. For example, the intermediate output calculation unit 12 applies each training data to the trained model 201 to calculate an intermediate output, and the noise effect evaluation unit 13 calculates the effect of noise on each intermediate output. The discrimination unit 14 plots points corresponding to the noise effect of each training data (hereinafter referred to as training data points) in a space defined by the noise effect (hereinafter referred to as the noise effect space) and identifies clusters of training data points. The discrimination unit 14 then plots points corresponding to the noise effect 204 of the data to be monitored 202 (hereinafter referred to as monitored points) in the noise effect space and determines whether the monitored points belong to a cluster or not. For example, if the monitored points are not included in the cluster, the discrimination unit 14 determines that the data to be monitored 202 is unknown data, and if the monitored points are included in the cluster, it determines that the data to be monitored 202 is not unknown data. As another example, the discrimination unit 14 may determine a representative point of the cluster and determine that the monitored data 202 is unknown data if the distance between the representative point and the monitored point is longer than a threshold, and determine that the monitored data 202 is not unknown data if the distance is shorter than a threshold. The representative point may be set to the center point of the cluster, the point closest to the monitored point, or the average value of the training data points belonging to the cluster.

[0048] When step S305 is performed, the output control unit 15 outputs a determination result 205 indicating whether or not the data generated in step S305 is unknown data (step S306). The determination result 205 is displayed on the display device 5. For example, it is desirable that a message such as "The monitored data is unknown data" or "The monitored data is not unknown data" be displayed as the determination result 205.

[0049] With the above steps completed, the unknown data detection process related to the evaluation phase is finished. Note that the unknown data detection process according to this embodiment is not limited to the above example. For example, the order of acquiring the trained model (step S301) and acquiring the data to be monitored (step S302) may be reversed. Also, when evaluating the effect of noise using projection, the determination of the projection matrix 221 may be performed in any order as long as it is before the projection execution step S322. For example, the pre-processing phase in Figure 9 may be prepared before the evaluation phase in Figure 5, and the projection matrix 221 may be calculated within that phase. In this case, the projection matrix 221 may be determined by another dedicated computer, and the calculation result may be stored in the storage device 2. This makes it possible to realize the unknown data detection device 100 even in a computing environment with limited resources such as an edge device. Evaluation of the effect of noise addition can be performed even in a computing environment with more limited resources than evaluation using projection, because noise generation can be performed in a short time and with very little memory if only a small amount of noise is generated.

[0050] The unknown data detection process according to this embodiment can be applied in various ways as shown below.

[0051] (Application Example 1) The discrimination unit 14 in Application Example 1 performs discrimination of unknown data based on a discrimination variable rather than the degree of noise influence.

[0052] Figure 10 schematically illustrates the unknown data detection process related to Application Example 1. In the following explanation, elements that are substantially the same as those in Figure 4 are denoted by the same reference numerals, and are explained redundantly only when necessary.

[0053] As shown in Figure 10, when the noise impact level 204 is calculated from the intermediate output 203 of the monitored data 202, the discrimination unit 14 converts the noise impact level 204 into a one-dimensional discrimination variable 206 (step S601). The discrimination variable 206 can be any one-dimensional variable based on the noise impact level 204. For example, the norm of the difference before and after directly adding noise, the ratio of the difference norm to the norm before adding noise, the norm of the projected component after applying the projection matrix, or the ratio of the projected norm to the norm before projection may be used as the discrimination variable 206.

[0054] When step S601 is performed, the discrimination unit 14 determines whether the monitored data 202 is unknown data or not based on a comparison between the discrimination variable 206 obtained in step S206 and a preset threshold 207 (step S602). Specifically, the discrimination unit 14 compares the magnitude of the discrimination variable 206 and the threshold 207. If the discrimination variable 206 is greater than the threshold 207, it determines that the monitored data 202 is not unknown data. If the discrimination variable 206 is less than the threshold 207, it determines that the monitored data 202 is unknown data. The discrimination result 208 indicating whether the monitored data 202 is unknown data or not is output to the display device 5, etc., by the output control unit 15.

[0055] The threshold 207 is set to a value that can distinguish between the discriminant variable based on the training data and the discriminant variable based on the unknown data. The threshold 207 may be set, for example, in the preprocessing phase or evaluation phase through the following steps. First, the discriminant unit 14 allocates a portion of the collected training data to unknown data for setting the threshold, rather than for training the trained model. The criteria for allocation are not particularly limited, but for example, data that has properties that make it statistically distinguishable from the training data when converted into a discriminant variable may be allocated to the unknown data. For example, if the task of the trained model is multi-class classification, data belonging to a specific class may be set to the unknown data. The number of specific classes is not particularly limited, but a few, such as one or two, may suffice. After the trained model is trained based on the training data by the processing circuit 1, etc., the intermediate output calculation unit 12 calculates an intermediate output for each of the unknown data and the training data, the noise influence evaluation unit 13 calculates the degree of noise influence on the intermediate output, and the discriminant unit 14 converts the degree of noise influence into a discriminant variable. The discrimination unit 14 then searches for a value that can distinguish between the discrimination variable based on the unknown data and the discrimination variable based on the training data, and sets that value as the threshold 207. This makes it possible to set a threshold 207 that can detect the unknown data.

[0056] When setting the threshold, it is not necessary to classify the data into unknown data. In this case, the discrimination unit 14 sets the threshold 207 so that statistical outliers among the training data used to train the trained model 201 can be classified as unknown data. Specifically, first, the discrimination unit 14 identifies outliers from among multiple training data using an arbitrary detection algorithm. Next, the discrimination unit 14 searches for a value that can distinguish between a discrimination variable based on the outlier and a discrimination variable based on other training data, and sets that value as the threshold 207. This makes it possible to set a threshold 207 that can detect outliers.

[0057] (Application Example 2) In the above embodiment, when evaluating the influence of noise by projection, all components included in the projection matrix were used. However, this embodiment is not limited to this. The projection matrix determination unit 121 in Application Example 2 changes the position and / or number of matrix components included in the projection matrix. The position of a matrix component is defined by the row number and column number of the matrix component. The number of matrix components simply refers to the number of matrix components, regardless of their position.

[0058] As described above, projection matrix V T V (12)T , V (13)T , , , V (L-2,L-1)T and V (L-1,L)T This is a combination of V. For example, V from the 1st row to the Lth row in order. (12)T , V (13)T , , , V (L-2,L-1)T and V (L-1,L)T By arranging the projection matrix V T The projection matrix determination unit 12 reduces the position and / or number of matrix components in the projection matrix by deleting matrix components that have a low contribution to the task of the trained model. Matrix components with a low contribution are, for example, matrix components corresponding to singular values ​​smaller than the reference value. The reference value can be arbitrarily determined experimentally or empirically. Deletion of matrix components can be done by setting the value of the matrix component to zero, or by deleting the matrix component itself.

[0059] As another example, the projection matrix determination unit 121 may search for the positions and / or numbers of matrix components that satisfy predetermined conditions based on the change in the task performance of the trained model when the positions and / or numbers of matrix components included in the projection matrix are changed. Specifically, the projection matrix determination unit 121 evaluates the task performance of the trained model while changing the positions and / or numbers of matrix components included in the projection matrix. Performance can be evaluated using any performance index value such as AUROC (area under the receiver operating characteristic curve) or AUPR (area under the precision-recall curve). The projection matrix determination unit 121 determines the positions and / or numbers of matrix components that satisfy predetermined conditions. The predetermined conditions are not particularly limited, but as an example, they can be set to be optimal in terms of computational cost and / or performance. The projection matrix determination unit 121 identifies the matrix components corresponding to the positions and / or numbers that satisfy the predetermined conditions and deletes matrix components other than the identified matrix components. In addition, weight parameters corresponding to the deleted matrix components may be deleted from the trained model. This makes it possible to prune weight parameters that do not contribute to the task and compress the size of the trained model. For example, if the task of the trained model is a classification task, changing the position and number of weight parameters will change the class matching probability (final output) and classification performance. For example, it would be good to remove matrix components from the projection matrix that do not significantly reduce performance despite being removed, and then remove the weight parameters corresponding to those matrix components from the trained model.

[0060] As described above, by removing matrix components with low contribution from the projection matrix, it is possible to further reduce the memory usage of the projection matrix while maintaining the ability to detect unknown data. Alternatively, the unnecessary weight parameters mentioned above may be removed from the trained model. This makes it possible to obtain a trained model with lower memory usage while maintaining the same performance as before the removal.

[0061] The removal of unnecessary parameters from the pre-trained model described above may also be performed when evaluating the impact of noise by adding noise rather than projection. In this case as well, it is possible to obtain a pre-trained model with lower memory usage while maintaining performance equivalent to that before the removal.

[0062] (Application Example 3) In the above embodiment, there was one threshold for the discriminant variable. However, this embodiment is not limited to this. The discriminant unit 15 in Application Example 3 determines a threshold for each layer of the trained model and determines whether the monitored data is unknown data or not based on a comparison between the discriminant variable and the threshold for each layer. As an example, the discriminant unit 15 may determine the threshold based on the rank of the parameter matrix for each layer. More specifically, the discriminant unit 15 sets the threshold so that layers with a rank greater than the standard do not contribute to the discrimination of unknown data. Conversely, the discriminant unit 15 sets the threshold according to the above embodiment so that layers with a rank less than the standard contribute to the discrimination of unknown data. The rank is evaluated using the stability rank of the matrix, i.e., the Frobenius norm ||W|| of the matrix. F 2 and spectral norm ||W||2 2 The ratio of ||W|| F 2 / ||W||2 2 Alternatively, a stability rank normalized by the matrix size may be used.

[0063] Figure 11 shows the relationship between layer depth and normalized stability rank. In Figure 11, the vertical axis represents the normalized stability rank, and the horizontal axis represents the layer depth. The normalized stability rank takes values ​​from "1" to "0". A stability rank closer to "1" means a larger rank in the weight matrix, and a rank closer to "0" means a smaller rank. Layer depth refers to the number of layers from the input layer. The change in normalized stability rank with respect to layer depth tends to be convex downwards. Also, layers with high unknown data detection performance tend to have a rapid decrease in normalized stability rank. Therefore, the discrimination unit 14 first calculates the normalized stability rank for each layer based on the rank of the weight parameter matrix. Next, the discrimination unit 14 identifies the layer 70 in which the normalized stability rank decreases the most from among multiple layers. Next, the discrimination unit 14 sets a range 71 of a predetermined number of layers including the layer 70 in which the normalized stability rank decreases the most as the range used for detecting unknown data (hereinafter referred to as the usage range). The predetermined number of layers in the usage range 71 can be determined experimentally or empirically.

[0064] The discrimination unit 14 sets a threshold value to a relatively small value such as zero for layers outside the usage range (hereinafter referred to as the non-usage range) so as not to contribute to the discrimination of unknown data. On the other hand, the discrimination unit 14 sets a threshold value in accordance with the above embodiment so as to contribute to the discrimination of unknown data for layers within the usage range.

[0065] If thresholds are set for each layer, the discrimination unit 14 calculates discrimination variables for each layer and determines whether the monitored data is unknown data based on a comparison between the discrimination variables and thresholds for each layer. The discrimination unit 14 then makes a final determination as to whether the monitored data is unknown data based on the discrimination results for each layer. For example, the discrimination unit 14 may determine the final discrimination result by majority vote of the discrimination results for each layer within the usage range. Specifically, if the number of layers in which the monitored data is determined to be unknown data is greater than the number of layers in which it is determined not to be unknown data, the monitored data should be determined to be unknown data. If the number of layers in which the monitored data is determined not to be unknown data is less than the number of layers in which it is determined not to be unknown data, the monitored data should be determined not to be unknown data. By setting thresholds for each layer, it becomes possible to give more weight to the discrimination results of layers with high unknown data detection performance. This is expected to improve the overall unknown data detection performance.

[0066] The threshold values ​​are set to different values ​​for the usage range 71 and the non-usage range, but they may also be set to different values ​​for each layer within the usage range 71 or for each layer within the non-usage range. Furthermore, the discrimination unit 14 does not need to perform discrimination on all layers included in the trained model, and may limit its discrimination to the layers in the usage range 71. In other words, the discrimination unit 14 can determine the layer to which the parameter used for evaluating the noise effect of the intermediate output belongs, based on the rank of the parameters for each layer of the trained model. This makes it possible to reduce the computational cost of discrimination.

[0067] (effect) Figure 12 shows the performance index values ​​for each method of detecting unknown data (OOD). The detection methods are classified into methods using classification probability, methods using Mahalanobis distance, methods using noise addition, and methods using projection components. The method using classification probability determines whether data is unknown or not based on the classification probability, which is the final output when the data to be monitored is input to a trained model. The method using Mahalanobis distance is the method shown in Non-Patent Literature 1. The method shown in Non-Patent Literature 1 refers to the intermediate output of a convolutional neural network model trained on a classification task, approximates it with a multivariate normal distribution, and evaluates the degree of deviation from the training data distribution based on the Mahalanobis distance. The Mahalanobis distance is a distance index normalized by the variance of the Euclidean distance between the mean of the intermediate output obtained from the training data and the intermediate output of the data to be monitored. The method using projection components is the method according to this embodiment. AUROC and AUPR are indices that represent the performance of detecting unknown data.

[0068] As shown in Figure 12, the method described in Non-Patent Document 1, which uses the Mahalanobis distance, has higher detection performance than the method using classification probability, and the method using projection components according to this embodiment has detection performance that is approximately the same as the method described in Non-Patent Document 1. The method using noise addition has lower detection performance than the method described in Non-Patent Document 1 and the method using projection components, but as will be described later, it uses very little memory.

[0069] The method described in Non-Patent Document 1 requires that a covariance matrix of the intermediate output of the training data be maintained in order to calculate the Mahalanobis distance. In the case of a deep learning model for multi-class classification, a covariance matrix must be maintained for each class, resulting in enormous memory usage for the covariance matrix. The number of dimensions of the intermediate output input to the weight parameters is d. in If the number of classes is K, the computational cost of the Mahalanobis distance is O(Kd in 2It is expressed as follows. Furthermore, the computational cost can increase by tens of times or more due to secondary factors such as the number of classes, convolutional kernel receptive fields, and feature map size.Therefore, the method described in Non-Patent Literature 1 is difficult to use for detecting unknown data on devices with small memory capacity, such as edge devices.

[0070] On the other hand, the method according to this embodiment does not use the Mahalanobis distance and therefore does not use a covariance matrix. Instead, when evaluation is performed using projection, the projection matrix is ​​retained. When noise is added, it is not necessary to retain any matrix. The computational cost according to this embodiment using the projection matrix is ​​calculated by the number of dimensions of the intermediate output output by the weight parameters d оut Then, O(d in ·d оut ) is expressed as follows. Thus, in the method according to this embodiment, there is no increase in computational cost with respect to the number of classes, and the output dimension d оut The input dimension is d. in Combined with the fact that it is usually larger than the original, the reduction in computational cost is significant. When noise is added, no additional memory is required, resulting in even greater memory savings than the embodiment using projection matrices. In addition, it is possible to further reduce the output dimension and thus the computational cost by compressing the model.

[0071] Figure 13 shows the memory usage of the Mahalanobis distance method and the method according to this embodiment. VGG and ResNet are types of network structures for deep learning models. Numbers 13, 16, and 19 in VGG and 18 and 34 in ResNet represent the number of layers. The "covariance" shown in Figure 13 represents the covariance matrix used in the method described in Non-Patent Literature 1, and the memory usage [GB] of the covariance matrix in each network structure is shown. The "projection matrix" represents the projection matrix used in the method according to this embodiment.

[0072] As shown in Figure 13, the memory usage of the "projection matrix" is approximately the same as the network's memory usage, and significantly less than that of the covariance matrix. When noise is applied, no additional memory is required. Furthermore, the memory usage of the "projection matrix" after model compression is even lower.

[0073] Note that the number of dimensions d of the intermediate output is determined by the weight parameters after model compression. оut ' is 0.2d оut Optimal detection performance is obtained when the setting is adjusted to a certain degree. This corresponds to the region where a significant decrease in classification accuracy occurs due to model compression. In other words, even when model compression such as pruning, which removes nodes that do not contribute to classification from the trained model, is performed, the detection performance of unknown data according to this embodiment can be maintained.

[0074] Assuming that the detection of unknown data is performed on an edge device, the projection matrix determination unit 12 may determine the size of the projection matrix according to the memory capacity of the edge device. The size of the projection matrix can be adjusted by increasing or decreasing the number of matrix components of the projection matrix, as described in Application Example 2.

[0075] As shown in Figures 12 and 13, the method according to this embodiment can reduce memory usage while maintaining the same level of unknown data detection performance as the method described in Non-Patent Literature 1. Unlike the method described in Non-Patent Literature 1, the method according to this embodiment does not directly refer to the training data distribution, but only to the parameters of the trained model. Because the method according to this embodiment utilizes only the information about the training data distribution embedded in the trained model, its general applicability to deep learning models in general is enhanced. Furthermore, because the method according to this embodiment can perform more general unknown data detection that is independent of the task details of the trained model, it can reduce the increase in computational complexity and memory usage derived from secondary elements such as the number of classification classes and the convolutional kernel receptive fields. Due to its nature of referencing the parameters of the trained model itself, it is also compatible with deep learning model compression, and further memory reduction effects can be expected by using it in combination with model compression techniques.

[0076] (Summary) As described in some of the embodiments above, the unknown data detection device 100 includes an acquisition unit 11, an intermediate output calculation unit 12, a noise impact evaluation unit 13, and a discrimination unit 14. The acquisition unit 11 acquires the data to be monitored. The intermediate output calculation unit 12 applies a trained model to the data to be monitored and calculates an intermediate output. The noise impact evaluation unit 13 calculates the noise impact of the intermediate output on the parameters that constitute the trained model. The discrimination unit 14 determines whether the data to be monitored is unknown data or not based on the noise impact.

[0077] According to the above configuration, unknown data can be detected based on the influence of noise from the intermediate output on the parameters constituting the trained model. The method according to this embodiment makes it possible to achieve high detection performance of unknown data while reducing memory usage. Thus, it becomes possible to detect unknown data with a low memory capacity.

[0078] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0079] 1... Processing circuit, 2... Memory device, 3... Input device, 4... Communication device, 5... Display device, 11... Acquisition unit, 12... Intermediate output calculation unit, 13... Noise impact evaluation unit, 14... Discrimination unit, 15... Output control unit, 100... Unknown data detection device.

Claims

1. A data acquisition unit that acquires data to be monitored, An intermediate output calculation unit that applies a trained model to the monitored data and calculates an intermediate output, A noise impact evaluation unit calculates the noise impact of the intermediate output in any layer using only the parameters that have a high contribution to the task from among the parameters that constitute the trained model, A determination unit that determines whether the monitored data is unknown data based on the degree of influence of the noise, It is equipped with, The noise impact evaluation unit further comprises a projection component calculation unit that calculates the projection component of the intermediate output onto the parameters constituting the trained model as the noise impact, and a projection matrix determination unit that determines the projection matrix based on the matrix decomposition of the parameters. The projection component calculation unit calculates the projection component by applying the projection matrix to the intermediate output. Unknown data detection device.

2. The unknown data detection device according to claim 1, wherein the projection matrix determination unit calculates the projection matrix by singular value decomposition of the weight parameters constituting the trained model.

3. The unknown data detection device according to claim 2, wherein the projection matrix determination unit removes matrix components with small singular values ​​from among the matrix components included in the projection matrix as components with low contribution to the task.

4. The unknown data detection device according to claim 3, wherein the projection matrix determination unit searches for matrix components that satisfy predetermined conditions based on the change in the performance of the trained model on the task when the position and / or number of matrix components included in the projection matrix is ​​changed.

5. The aforementioned discrimination unit is The noise influence is converted into a one-dimensional discrimination variable, Based on the comparison between the aforementioned discrimination variable and the threshold, it is determined whether the monitored data is unknown data or not. The unknown data detection device according to claim 1.

6. The unknown data detection device according to claim 5, wherein the discrimination unit sets data from a plurality of training data that are not used to train the trained model as unknown data, and sets the threshold using the unknown data.

7. The unknown data detection device according to claim 6, wherein the discrimination unit sets the threshold so that outliers among the training data used to train the trained model can be classified as unknown data.

8. The unknown data detection device according to claim 5, wherein the discrimination unit calculates the norm of the noise influence, or the ratio of the norm of the intermediate output to the norm of the noise influence, as the discrimination variable.

9. The aforementioned discrimination unit is A threshold is determined for each layer of the aforementioned trained model. Based on a comparison between the discrimination variable for each layer and the threshold, it is determined whether the monitored data is unknown data or not. The unknown data detection device according to claim 5.

10. The unknown data detection device according to claim 9, wherein the discrimination unit determines the threshold value based on the number of levels of the parameters for each layer.

11. A computer, The acquisition process involves obtaining the data to be monitored, An intermediate output calculation step is performed by applying a trained model to the monitored data and calculating an intermediate output. A noise impact evaluation step, which calculates the noise impact of the intermediate output in any layer using only the parameters that have a high contribution to the task among the parameters that constitute the trained model, The system comprises a determination step of determining whether the monitored data is unknown data based on the noise impact, The noise impact evaluation step further comprises a projection component calculation step that calculates the projection component of the intermediate output onto the parameters constituting the trained model as the degree of noise impact, and a projection matrix determination step that determines the projection matrix based on the matrix decomposition of the parameters. The projection component calculation step involves applying the projection matrix to the intermediate output to calculate the projection component. Methods for detecting unknown data.

12. On the computer, A function to acquire data to be monitored, An intermediate output calculation function that applies a trained model to the monitored data and calculates an intermediate output, A noise impact evaluation function that calculates the noise impact of the intermediate output in any layer using only the parameters that have a high contribution to the task from among the parameters that constitute the trained model, A determination function that determines whether the monitored data is unknown data or not based on the degree of impact of the noise, To make it happen, The noise impact evaluation function further comprises a projection component calculation function that calculates the projection component of the intermediate output onto the parameters constituting the trained model as the noise impact degree, and a projection matrix determination function that determines the projection matrix based on the matrix decomposition of the parameters. The projection component calculation function calculates the projection component by applying the projection matrix to the intermediate output. Unknown data detection program.

Images