Information processing device, information processing method, and program

Abstract

To enable detecting variation of a data distribution using a simple method.SOLUTION: An information processing device 1 comprises: a model storage unit 121 that stores a unique identifier prediction model being machine-learned to output a unique identifier which is uniquely allocated to each of feature vectors constituting a teacher vector group when either of the teacher vector group is input; a reference histogram storage unit 122 that stores a reference histogram indicating a distribution of the unique identifier corresponding to a reference vector group; an acquisition unit 131 that acquires a test vector group; a test histogram generation unit 132 that generates a test histogram indicating a distribution of the unique identifier corresponding to the test vector group; a similarity calculation unit 133 that calculates similarity between respective distributions of the reference histogram and the test histogram; and a determination unit 134 that determines whether the distribution of the test vector group is out of the distribution of learning data on the basis of the similarity.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a need to detect changes in data distribution, which are a cause of performance degradation in machine learning models. As a method for detecting changes in data distribution, there is a method based on density ratio estimation of learning samples in feature space. However, since the method based on density ratio estimation has an issue with the amount of calculation, a method of comparing histograms is generally used. When creating a histogram, it is necessary to define bins that will be the units of aggregation. There are two methods for defining bins: a method based on the axis of the feature space (Non-Patent Documents 1 and 2) and a method based on clustering (Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] G. Ditzler and R. Polikar, “Hellinger distance based drift detection for nonstationary environments,” in 2011 IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments, 2011 [Non-Patent Document 2] A. Qahtan and S. Wang, “A PCA-Based Change Detection Framework for Multidimensional Data Streams Categories and Subject Descriptors,” KDD '15 21th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min, 2015 [Non-Patent Document 3] A. Liu, J. Lu, and G. Zhang, “Concept Drift Detection: Dealing With Missing Values ​​via Fuzzy Distance Estimations,” IEEE Trans. Fuzzy Syst., 2021 Summary of the Invention [Problem to be solved by the invention]

[0004] However, methods based on the axes of feature space have challenges in tuning. Specifically, to set the bin interval appropriately, measures such as taking statistics are required, and the number of bins needs to be set appropriately according to the batch size. In addition, methods based on clustering have low accuracy for high-dimensional data, and density-based clustering has the problem of increasing the amount of calculations required for high-dimensional data.

[0005] The present invention has been made in consideration of these points, and has an object to make it possible to detect changes in data distribution in a simple manner. [Means for solving the problem]

[0006] In the information processing device of the first aspect of the present invention, the information processing device includes a model storage unit that stores a unique identifier prediction model that has been machine-learned so as to output a unique identifier uniquely assigned to each of the feature vectors that constitute the teacher vector group when any one of N (N is an integer of 2 or more) feature vectors sampled as a teacher vector group from training data that is a set of feature vectors is input, and a unique identifier output by the unique identifier prediction model when a reference vector group that is a set of feature vectors sampled from the training data and includes feature vectors different from the feature vectors that constitute the teacher vector group is input to the unique identifier prediction model. an acquisition unit that acquires a test vector group which is a collection of feature vectors; a test histogram generation unit that generates a test histogram which is a histogram that indicates the distribution of unique identifiers output by the unique identifier prediction model when the test vector group is input to the unique identifier prediction model; a similarity calculation unit that calculates the similarity between the distribution of the reference histogram and the distribution of the test histogram; and a determination unit that determines whether the distribution of the test vector group deviates from the distribution of the feature vectors that constitute the learning data based on the calculated similarity.

[0007] The determination unit may compare the calculated similarity with a predetermined threshold value to determine whether or not a distribution of the test vector group deviates from a distribution of feature vectors constituting the training data.

[0008] The method further includes a calibration histogram generation unit that generates a calibration histogram which is a histogram showing a distribution of unique identifications output by the unique identifier prediction model when each calibration vector group is input to the unique identifier prediction model, the calibration vector group being a set of feature vectors sampled from the learning data, the feature vectors being different from the feature vectors constituting the teacher vector group and the reference vector group; and a threshold determination unit that calculates a similarity between each of the calibration histograms output by the calibration histogram generation unit based on a plurality of different calibration vector groups and the reference histogram, and determines a threshold based on a distribution of the similarities between each of the calculated calibration histograms and the reference histogram, and the determination unit may compare the similarity calculated by the similarity calculation unit with the threshold determined by the threshold determination unit to determine whether or not a distribution of the test vector group deviates from a distribution of the feature vectors constituting the learning data.

[0009] The similarity calculation unit may calculate the similarity to be higher when the ranking of each unique identifier in the reference histogram, which is determined by arranging the number of feature vectors distributed in each unique identifier in descending order, matches the ranking of each unique identifier in the test histogram.

[0010] The similarity calculation unit may calculate a higher similarity as the distribution of the feature vectors in the test histogram is more uniform, and may calculate a lower similarity as the distribution of the feature vectors in the test histogram is more biased.

[0011] The method may further include an output unit that outputs that the test vector group is out-of-distribution data when the determination unit detects that the distribution of the test vector group deviates from the distribution of the feature vectors constituting the training data a predetermined number of times or more.

[0012] The model storage unit may store the unique identifier prediction model that has been further machine-learned so that when a group of correction vectors that is a collection of feature vectors sampled from the learning data and that includes feature vectors different from the feature vectors that constitute the teacher vector group is input to the unique identifier prediction model, the unique identifier prediction model has been further machine-learned so that the distribution of unique identifiers output by the unique identifier prediction model is uniform.

[0013] In a second aspect of the information processing method of the present invention, a computer-executed step of acquiring a test vector group, which is a set of feature vectors; generating a test histogram, which is a histogram showing the distribution of unique identifiers output by the unique identifier prediction model when the test vector group is input to a unique identifier prediction model; calculating a similarity between the distributions of a reference histogram and the test histogram; and determining whether or not the distribution of the test vector group deviates from the distribution of feature vectors constituting training data based on the similarity, wherein the unique identifier prediction model is a unique identifier prediction model that has been machine-learned to output a unique identifier uniquely assigned to each of the feature vectors constituting the training vector group when any of N (N is an integer of 2 or more) feature vectors sampled as a teacher vector group from the training data, which is a set of feature vectors, is input to the unique identifier prediction model, and the reference histogram is a histogram showing the distribution of unique identifiers output by the unique identifier prediction model when a reference vector group, which is a set of feature vectors sampled from the training data and includes feature vectors different from the feature vectors constituting the teacher vector group, is input to the unique identifier prediction model.

[0014] A third aspect of the program of the present invention is a program that causes a computer to execute the steps of acquiring a test vector group, which is a set of feature vectors; generating a test histogram, which is a histogram showing the distribution of unique identifiers output by the unique identifier prediction model when the test vector group is input to a unique identifier prediction model; calculating a similarity between the distributions of a reference histogram and the test histogram; and determining, based on the similarity, whether or not the distribution of the test vector group deviates from the distribution of feature vectors constituting training data. The unique identifier prediction model is a unique identifier prediction model that has been machine-learned to output a unique identifier uniquely assigned to each of the feature vectors constituting the training vector group when any of N (N is an integer of 2 or more) feature vectors sampled as a teacher vector group from the training data, which is a set of feature vectors, is input to the unique identifier prediction model. The reference histogram is a histogram showing the distribution of unique identifiers output by the unique identifier prediction model when a reference vector group, which is a set of feature vectors sampled from the training data and includes feature vectors different from the feature vectors constituting the teacher vector group, is input to the unique identifier prediction model. Effect of the Invention

[0015] The present invention has an effect of making it possible to detect a change in data distribution using a simple method. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram showing an overview of out-of-distribution data detection processing in an information processing device 1. FIG. [Diagram 2] 1 is a block diagram showing a configuration of an information processing device 1. FIG. [Diagram 3] 11 is a diagram illustrating the learning process of a learning unit 138. FIG. [Figure 4] 11 is a diagram illustrating a process in which a threshold value determining unit 136 determines a threshold value. FIG. [Diagram 5]3 is a flowchart showing a process flow in the information processing device 1. [Figure 6] 3 is a flowchart showing a process flow in the information processing device 1. [Figure 7] FIG. 11 is a diagram illustrating a learning process of a learning unit 138 according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] <First embodiment> [Overview of processing in information processing device 1] 1 is a diagram showing an overview of out-of-distribution data detection processing in an information processing device 1. The information processing device 1 is a device for detecting that the distribution of data to be inferred by a machine learning model deviates from the distribution of training data used in training. The information processing device 1 is, for example, a server.

[0018] An overview of the processing in the information processing device 1 will be described. The information processing device 1 inputs a test vector group V1 to a unique identifier prediction model M ((1) in FIG. 1). The test vector group V1 is a group of feature vectors of data to be judged. The unique identifier prediction model M is a trained model trained to output a different unique identifier for each of the training feature vector groups, the number of which is the same as the unique identifiers. The unique identifier (also called a UUID (Universally Unique Identifier)) is an identifier for classifying feature vectors.

[0019] The information processing device 1 generates a test histogram H1 indicating the frequency of each unique identifier output by the unique identifier prediction model M based on the input test vector group V1 ((2) in FIG. 1).

[0020] The information processing device 1 calculates the similarity between the generated test histogram H1 and a reference histogram H2. Based on the calculated similarity, the information processing device 1 determines whether or not the test vector group V1 is out-of-distribution data ((3) in FIG. 1). The reference histogram H2 is a histogram generated by inputting a group of feature vectors extracted from the training data into a unique identifier prediction model M, and is a histogram that reflects the distribution of the training data. The training data is a collection of data or feature vectors collected for training.

[0021] By configuring the information processing device 1 in this way, it is possible to detect changes in data distribution in a simple manner.

[0022] [Configuration of information processing device 1] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The storage unit 12 has a model storage unit 121 and a reference histogram storage unit 122. The control unit 13 has an acquisition unit 131, a test histogram generation unit 132, a similarity calculation unit 133, a judgment unit 134, a calibration histogram generation unit 135, a threshold determination unit 136, an output unit 137, a learning unit 138, and a reference histogram generation unit 139.

[0023] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), a hard disk drive, etc. The storage unit 12 stores in advance a program to be executed by the control unit 13.

[0024] The model storage unit 121 stores the unique identifier prediction model M learned by the learning unit 138 as described below. The unique identifier prediction model M is a trained model that has been machine-learned to output a unique identifier uniquely assigned to each of the feature vectors constituting the teacher vector group V2 when any of N (N is an integer equal to or greater than 2) feature vectors sampled as a teacher vector group V2 from training data D1, which is a collection of feature vectors, is input.

[0025] The reference histogram storage unit 122 stores the reference histogram H2 generated by the reference histogram generation unit 139 as described below. The reference histogram H2 is a histogram indicating the distribution of unique identifiers output by the unique identifier prediction model M when the reference vector group V3 is input to the unique identifier prediction model M. The reference vector group V3 is a collection of feature vectors sampled from the learning data D1, and includes feature vectors different from the feature vectors constituting the teacher vector group V2.

[0026] The control unit 13 is a processor such as a CPU (Central Processing Unit), etc. The control unit 13 executes the programs stored in the storage unit 12 to function as an acquisition unit 131, a test histogram generation unit 132, a similarity calculation unit 133, a judgment unit 134, a calibration histogram generation unit 135, a threshold determination unit 136, an output unit 137, a learning unit 138, and a reference histogram generation unit 139.

[0027] [Learning process] The learning process in the information processing device 1 will be described. The learning unit 138 extracts feature vector groups from the learning data D1, the number of which is the same as the number of unique identifiers ((1) in FIG. 3). As an example, when there are 128 unique identifiers, the learning unit 138 extracts 128 feature vector groups to create a teacher vector group V2. The learning unit 138 generates correct answer data D2 in which the correct unique identifier is associated with each of the feature vectors included in the teacher vector group V2 ((2) in FIG. 3).

[0028] The learning unit 138 inputs the teacher vector group V2 to the unique identifier prediction model M ((3) in FIG. 3), and predicts a unique identifier corresponding to the teacher vector group V2 ((4) in FIG. 3). The learning unit 138 evaluates the accuracy of the unique identifier prediction model M based on the correct answer data D2. As an example, the learning unit 138 evaluates the accuracy of the unique identifier prediction model M using the output cross entropy error as a loss function, and updates the parameters of the unique identifier prediction model M based on the evaluation result ((5) in FIG. 3).

[0029] The learning unit 138 repeats the processes (3) to (5) in FIG. 3 until a learning termination condition is met, and trains the unique identifier prediction model M. The termination condition is, for example, that the average value of the loss function falls below a predetermined value or that the number of parameter updates reaches a predetermined number. The learning unit 138 stores the trained unique identifier prediction model M in the model storage unit 121.

[0030] As a preferred example, the unique identifier prediction model M may be configured as follows. The input layer of the unique identifier prediction model M has the same number of dimensions as the number of features of the input feature vector. The unique identifier prediction model M has three hidden layers. Each hidden layer has 256 dimensions and uses Leaky ReLU (Leaky Rectified Linear Unit) as an activation function. The output layer of the unique identifier prediction model M has the same number of dimensions as the number of digits of the unique identifier (for example, if the number of unique identifiers is 128, it is 7 digits in binary encoding), and uses a Sigmoid function as an activation function. The learning unit 138 evaluates the accuracy of the unique identifier prediction model M using the binary cross entropy error between the predicted value for each dimension of the vector output by the output layer and the binary value of the corresponding digit of the correct unique identifier as a loss function. The learning unit 138 updates the parameters of the unique identifier prediction model M by the error backpropagation method. As an example, the learning unit 138 uses Adam, which is one of the stochastic gradient descent algorithms, as a parameter update method for the unique identifier prediction model M, and updates the parameters based on the calculated binary cross entropy error.

[0031] Returning to FIG. 2, the reference histogram generating unit 139 extracts a reference vector group V3 from the learning data D1. The reference vector group V3 is a feature vector group including feature vectors different from the teacher vector group V2. The number of feature vectors included in the reference vector group V3 is arbitrary. The reference histogram generating unit 139 inputs the reference vector group V3 to the unique identifier prediction model M, and outputs unique identifiers corresponding to each of the feature vectors included in the reference vector group V3. The reference histogram generating unit 139 generates a reference histogram H2 based on the reference vector group V3 and the unique identifier output by the unique identifier prediction model M. The reference histogram H2 is a histogram showing the distribution of the number of feature vectors included in the reference vector group V3 assigned to each unique identifier, with the unique identifiers output by the unique identifier prediction model M as bins.

[0032] [Judgment process] The determination process in the information processing device 1 will be described. The acquisition unit 131 acquires a test vector group V1 which is a collection of feature vectors. The acquisition unit 131 may acquire the test vector group V1 from an external device (not shown) via the communication unit 11, or may acquire the test vector group V1 stored in the storage unit 12. The acquisition unit 131 may also acquire data to be determined and extract feature vectors from the acquired data to be inferred to acquire the test vector group V1.

[0033] When the test histogram generation unit 132 inputs the test vector group V1 to the unique identifier prediction model M, it generates a test histogram H1 which is a histogram showing the distribution of unique identifiers output by the unique identifier prediction model M. The test histogram generation unit 132 inputs the test vector group V1 to the unique identifier prediction model M, and causes it to output unique identifiers corresponding to each of the feature vectors included in the test vector group V1. The test histogram generation unit 132 generates a test histogram H1 based on the test vector group V1 and the unique identifiers output by the unique identifier prediction model M. The test histogram H1 is a histogram showing the distribution of the number of feature vectors included in the test vector group V1 assigned to each unique identifier, with the unique identifiers output by the unique identifier prediction model M as bins.

[0034] The similarity calculation unit 133 calculates the similarity between the distributions of the reference histogram H2 and the test histogram H1. For example, the similarity calculation unit 133 may calculate the similarity by a known method such as the Kullback-Leibler divergence, or may calculate the similarity by a method described later.

[0035] The determination unit 134 determines whether the distribution of the test vector group V1 deviates from the distribution of the feature vectors constituting the learning data D1 based on the calculated similarity. The determination unit 134 may compare the calculated similarity with a predetermined threshold value to determine whether the distribution of the test vector group V1 deviates from the distribution of the feature vectors constituting the learning data D1. The determination unit 134 may determine that the distribution of the test vector group V1 deviates from the distribution of the feature vectors when the similarity is less than the threshold value. As described later, the determination unit 134 may determine that the distribution of the test vector group V1 deviates from the distribution of the feature vectors when the similarity between the test vector group V1 and the reference vector group V3 is less than a predetermined threshold value for a predetermined number of times or more. When the determination unit 134 detects out-of-distribution data, the output unit 137 may output the detection of out-of-distribution data to an external device.

[0036] Incidentally, even if the determination unit 134 detects out-of-distribution data, the possibility cannot be denied that the test data to be determined deviates from the distribution by chance. Therefore, the information processing device 1 may be configured to determine whether or not the test data is out-of-distribution data based on the number of times out-of-distribution data is detected or the degree to which the test data deviates from the distribution of the learning data D1.

[0037] When the determination unit 134 detects that the distribution of the test vector group V1 deviates from the distribution of the feature vectors constituting the learning data a predetermined number of times or more, the output unit 137 may output that the test vector group is out-of-distribution data. The determination unit 134 counts the number of times that the out-of-distribution data is detected. Then, when the determination unit 134 detects the out-of-distribution data a predetermined number of times or more, the output unit 137 outputs that the test vector group V1 is out-of-distribution data.

[0038] The determination unit 134 counts the number of times out-of-distribution data is detected for each degree to which the similarity of the test data calculated by the similarity calculation unit 133 deviates from the distribution of similarity between the reference histogram H2 and the calibrated histogram H3 (for example, the difference between the calculated similarity and the average value of the similarity between the reference histogram H2 and the calibrated histogram H3 is less than 1σ, 1σ or more and less than 2σ, 2σ or more and less than 3σ, 3σ or more, etc.) based on the average value and variance (σ) of the similarity between the reference histogram H2 and the calibrated histogram H3 calculated by the threshold determination unit 136. When the counted number of times of detection exceeds a predetermined number determined for each degree to which the similarity is deviated from the distribution, the determination unit 134 may cause the output unit 137 to output that the test vector group V1 is out-of-distribution data. For example, the determination unit 134 may determine out-of-distribution data based on the known Western Electric rules.

[0039] By configuring the information processing device 1 in this way, it is possible to improve the accuracy of detecting out-of-distribution data.

[0040] [Threshold determination process] The process of determining a threshold for detecting out-of-distribution data will now be described. FIG. 4 is a diagram illustrating the process of threshold determination unit 136 determining a threshold. The processes (1) to (3) in FIG. 4 have already been described in the learning process, so they will not be described here. First, calibration histogram generation unit 135 extracts a group of calibration vectors V4 from learning data D1 ((4) in FIG. 4). Calibration vector group V4 is a collection of feature vectors sampled from learning data D1, and each of the calibration vector group V4 includes feature vectors different from the feature vectors constituting teacher vector group V2 and reference vector group V3.

[0041] The calibration histogram generation unit 135 inputs each of the multiple calibration vector groups V4 to the unique identifier prediction model M, and generates multiple calibration histograms H3, which are histograms showing the distribution of unique identifiers output by the unique identifier prediction model M. The calibration histogram generation unit 135 inputs the multiple extracted calibration vector groups V4 to the unique identifier prediction model M ((5) in FIG. 4), and outputs unique identifiers corresponding to each of the feature vectors included in the calibration vector group V4. The calibration histogram generation unit 135 generates multiple calibration histograms H3 based on the multiple calibration vector groups V4 and the unique identifiers output by the unique identifier prediction model M. The calibration histogram H3 is a histogram showing the distribution of the number of feature vectors included in the calibration vector group V4 assigned to each unique identifier, with the unique identifiers output by the unique identifier prediction model M as bins.

[0042] The threshold determination unit 136 calculates the similarity between each of the multiple calibration histograms H3 output by the calibration histogram generation unit 135 based on the multiple different calibration vector groups V4 and the reference histogram H2, and determines a threshold based on the distribution of the calculated similarities between each of the multiple calibration histograms H3 and the reference histogram H2. As an example, the threshold determination unit 136 may determine the threshold based on a statistical value (median, average, standard deviation) of the similarity between each of the multiple calibration histograms H3 and the reference histogram H2. As an example, the threshold determination unit 136 may determine the threshold by subtracting a predetermined value from the average value or median value of the calculated similarities, or may determine the threshold by subtracting a value obtained by multiplying the calculated variance by a predetermined coefficient from the average value of the calculated similarities. Then, the judgment unit 134 compares the similarity calculated by the similarity calculation unit 133 with the threshold determined by the threshold determination unit 136 to judge whether the distribution of the test vector group V1 deviates from the distribution of the feature vectors constituting the learning data D1.

[0043] [Similarity calculation process] The details of the process of calculating the similarity will be described. The similarity calculation unit 133 may calculate the similarity to be higher as the rank of each unique identifier, which is determined by arranging the number of feature vectors distributed in each unique identifier in the reference histogram H2 in descending order, matches the rank of each unique identifier in the test histogram H1. As an example, the similarity calculation unit 133 may calculate the similarity based on nDCG (normalized discounted cumulative gain). nDCGp, which indicates the similarity between the reference histogram H2 and the test histogram H1, in which the number of unique identifiers is p and p unique identifiers are bins, is expressed by the following formula.

number

[0044] IDCGp in Equation 1 is the gain in the reference histogram H2 and is expressed by the following equation.

number

[0045] In formula 2, i is the rank of each unique identifier when the reference histogram H2 is sorted in descending order of the number of feature vectors (hereinafter sometimes referred to as frequency) distributed in each unique identifier. In formula 2, reli is the frequency corresponding to the unique identifier in the i-th rank.

[0046] DCGp in Equation 1 is the gain in the test histogram H1 and is expressed by the following equation.

number

[0047] In Equation 3, j is the rank of each unique identifier when the test histogram H1 is sorted in descending order of the frequency corresponding to each unique identifier. In Equation 3, relj' is the frequency corresponding to the unique identifier in the jth rank in the reference histogram H2.

[0048] Incidentally, by taking into consideration the variation in distribution when calculating the similarity, the accuracy of detecting out-of-distribution data can be improved. Therefore, the similarity calculation unit 133 may calculate a higher similarity as the distribution of the feature vectors in the test histogram H1 is more uniform, and may calculate a lower similarity as the distribution of the feature vectors in the test histogram H1 is more biased. For example, the similarity calculation unit 133 may calculate the similarity by multiplying G and nDCGp expressed by the following formula.

number

[0049] In the formula 4, yk is the frequency corresponding to the kth unique identifier when the test histogram H1 is sorted in ascending order of the frequency corresponding to each unique identifier. By configuring the information processing device 1 in this way, it is possible to improve the accuracy of detecting out-of-distribution data.

[0050] [Processing flow in information processing device 1] Fig. 5 is a flowchart showing the flow of the learning process in the information processing device 1. The flowchart shown in Fig. 5 starts from the point in time when an instruction to start the learning process is received. The learning unit 138 acquires a teacher vector group V2 (S01). The learning unit 138 inputs the acquired teacher vector group V2 to the unique identifier prediction model M (S02). The learning unit 138 compares the unique identifier output from the unique identifier prediction model M with the correct answer data D2 to evaluate the accuracy of the unique identifier prediction model M (S03). The learning unit 138 updates the parameters of the unique identifier prediction model M based on the evaluation result (S04).

[0051] The learning unit 138 determines whether or not the learning is complete (S05). If the learning is not complete (NO in S05), the process returns to S02. If the learning is complete (YES in S05), the process proceeds to S06.

[0052] The reference histogram generating unit 139 acquires the reference vector group V3 (S06). The reference histogram generating unit 139 inputs the reference vector group V3 to the unique identifier prediction model M (S07). The reference histogram generating unit 139 generates a reference histogram H2 based on the unique identifier output from the unique identifier prediction model M (S08). Then, the information processing device 1 ends the learning process.

[0053] Fig. 6 is a flowchart showing the flow of the inference process in the information processing device 1. The flowchart shown in Fig. 6 starts from the point in time when a trigger for starting the inference process is pulled. The acquisition unit 131 acquires a test vector group V1 (S101). The test histogram generation unit 132 inputs the test vector group V1 to the unique identifier prediction model M (S102). The test histogram generation unit 132 generates a test histogram H1 based on the unique identifier output by the unique identifier prediction model M (S103).

[0054] The similarity calculation unit 133 acquires the reference histogram H2 (S104). The similarity calculation unit 133 calculates the similarity between the test histogram H1 and the reference histogram H2 (S105). The determination unit 134 determines whether the similarity calculated by the similarity calculation unit 133 is less than a threshold (S106). If the similarity is less than the threshold (YES in S106), the output unit 137 outputs that the data is out of distribution (S107), and the information processing device 1 ends the process. If the similarity is equal to or greater than the threshold (NO in S106), the information processing device 1 ends the process.

[0055] [Effects of the first embodiment] As described above, in the information processing device 1 according to the first embodiment, a change in data distribution can be detected in a simple manner.

[0056] <Second embodiment> However, when the teacher vector group V2 extracted from the learning data D1 is small compared to the data included in the learning data D1, a problem occurs in that the bias of the data of the extracted teacher vector group V2 itself is reflected in the learned model. Therefore, the learning unit 138 may be configured to input a vector group different from the teacher vector group V2 to the unique identifier prediction model M and further train the unique identifier prediction model M so that the distribution of the output unique identifier becomes uniform. The following describes the differences between the second embodiment and the first embodiment. In the second embodiment, the unique identifier may be expressed by one-hot encoding.

[0057] 7 is a diagram for explaining the learning process of the learning unit 138 according to the second embodiment. The processes from (1) to (4) in FIG. 7 correspond to the learning process in FIG.

[0058] The learning unit 138 extracts a correction vector group V5 from the learning data D1 ((5) in FIG. 7). The correction vector group V5 is a collection of feature vectors sampled from the learning data D1, and includes feature vectors different from the feature vectors constituting the teacher vector group V2. The number of feature vectors included in the correction vector group V5 is arbitrary, but it is desirable that the number of feature vectors is larger than that of the teacher vector group V2. As an example, when the number of feature vectors in the teacher vector group V2 is 64, the number of feature vectors included in the correction vector group V5 may be set to 1000.

[0059] The learning unit 138 inputs the correction vector group V5 to the unique identifier prediction model M ((6) in FIG. 7). The learning unit 138 calculates the entropy of the unique identifier corresponding to the correction vector group V5. As an example, the learning unit 138 may average the output unique identifier in the sample direction, normalize it in the output dimension direction, and calculate the entropy H of the normalized unique identifier. The learning unit 138 evaluates the accuracy of the unique identifier prediction model M using the loss function obtained by subtracting the entropy H calculated from the error between the correct answer data D2 and the prediction result D3. Then, the learning unit 138 updates the parameters of the unique identifier prediction model M based on the evaluation result ((8) in FIG. 7).

[0060] 7 is repeated until a learning end condition is satisfied, to train the unique identifier prediction model M. The learning unit 138 stores the trained unique identifier prediction model M in the model storage unit 121.

[0061] The model storage unit 121 stores the unique identifier prediction model M that has been further machine-learned so that when the correction vector group V5 is input to the unique identifier prediction model M, the distribution of unique identifiers output by the unique identifier prediction model M becomes uniform.

[0062] By configuring the information processing device 1 in this way, it becomes possible to generate a prediction model in which the bias of the data in the teacher vector group V2 is alleviated, and the accuracy of detecting out-of-distribution data can be improved.

[0063] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote industry, innovation and infrastructure."

[0064] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment. [Explanation of symbols]

[0065] 1. Information processing device 11 Communications Department 12 Storage section 13 Control section 121 Model Memory Section 122 Reference histogram memory unit 131 Acquisition Department 132 Test histogram generator 133 Similarity calculation unit 134 Judgment section 135 Calibrated histogram generator 136 Threshold determination unit 137 Output section 138 Learning Department 139 Reference histogram generator

Claims

1. a model storage unit that stores a unique identifier prediction model that has been machine-learned so as to output a unique identifier that is uniquely assigned to each of the feature vectors that constitute the teacher vector group when any one of N (N is an integer equal to or greater than 2) feature vectors sampled as a teacher vector group from training data that is a collection of feature vectors is input; a reference histogram storage unit that stores a reference histogram, which is a histogram indicating a distribution of unique identifiers output by the unique identifier prediction model when a reference vector group, which is a set of feature vectors sampled from the learning data and includes feature vectors different from the feature vectors constituting the teacher vector group, is input to the unique identifier prediction model; an acquisition unit that acquires a test vector group, which is a set of feature vectors; a test histogram generation unit that generates a test histogram that is a histogram indicating a distribution of unique identifiers output by the unique identifier prediction model when the test vector group is input to the unique identifier prediction model; a similarity calculation unit that calculates a similarity between the distributions of the reference histogram and the test histogram; a determination unit that determines whether or not a distribution of the test vector group deviates from a distribution of feature vectors constituting the learning data based on the calculated similarity; An information processing device having the above configuration.

2. the determination unit compares the calculated similarity with a predetermined threshold value to determine whether or not a distribution of the test vector group deviates from a distribution of feature vectors constituting the learning data. The information processing device according to claim 1 .

3. a calibration histogram generation unit that generates a calibration histogram, which is a histogram showing a distribution of unique identifiers output by the unique identifier prediction model, when each calibration vector group, which is a set of feature vectors sampled from the learning data and includes feature vectors different from the feature vectors constituting the teacher vector group and the reference vector group, is input to the unique identifier prediction model; a threshold determination unit that calculates a similarity between each of the plurality of calibration histograms output by the calibration histogram generation unit based on a plurality of different calibration vector groups and the reference histogram, and determines a threshold based on a distribution of the calculated similarities between each of the plurality of calibration histograms and the reference histogram, the determination unit compares the similarity calculated by the similarity calculation unit with the threshold determined by the threshold determination unit to determine whether or not a distribution of the test vector group deviates from a distribution of feature vectors constituting the learning data. The information processing device according to claim 1 .

4. the similarity calculation unit calculates a higher similarity as the rank of each unique identifier in the reference histogram, which is determined by arranging the number of feature vectors distributed in each unique identifier in descending order, matches the rank of each unique identifier in the test histogram; The information processing device according to claim 1 .

5. the similarity calculation unit calculates a higher similarity as the distribution of the feature vectors in the test histogram becomes more uniform, and calculates a lower similarity as the distribution of the feature vectors in the test histogram becomes more biased; The information processing device according to claim 1 .

6. and an output unit that outputs that the test vector group is out-of-distribution data when the determination unit detects that the distribution of the test vector group deviates from the distribution of the feature vectors constituting the learning data a predetermined number of times or more. The information processing device according to claim 1 .

7. the model storage unit stores the unique identifier prediction model that has been further machine-learned so that when a correction vector group including a set of feature vectors sampled from the learning data and different feature vectors from the feature vectors constituting the teacher vector group is input to the unique identifier prediction model, the unique identifier prediction model is outputted with a uniform distribution of unique identifiers. The information processing device according to claim 1 .

8. The computer executes obtaining a test vector group which is a set of feature vectors; generating a test histogram which is a histogram showing a distribution of unique identifiers output by a unique identifier prediction model when the test vector group is input to the unique identifier prediction model; Calculating a similarity between the distributions of a reference histogram and the test histogram; and determining whether or not a distribution of the test vector group deviates from a distribution of feature vectors constituting training data based on the similarity, the unique identifier prediction model is a unique identifier prediction model that has been machine-trained so that, when any one of N (N is an integer of 2 or more) feature vectors sampled as a teacher vector group from the learning data, which is a set of feature vectors, is input, the unique identifier prediction model outputs a unique identifier that is uniquely assigned to each of the feature vectors that constitute the teacher vector group; the reference histogram is a histogram showing a distribution of unique identifications output by the unique identifier prediction model when a reference vector group, which is a set of feature vectors sampled from the learning data and includes feature vectors different from the feature vectors constituting the teacher vector group, is input to the unique identifier prediction model; Information processing methods.

9. On the computer, obtaining a test vector group which is a set of feature vectors; generating a test histogram which is a histogram showing a distribution of unique identifiers output by a unique identifier prediction model when the test vector group is input to the unique identifier prediction model; Calculating a similarity between the distributions of a reference histogram and the test histogram; and determining whether or not a distribution of the test vector group deviates from a distribution of feature vectors constituting training data based on the similarity, the unique identifier prediction model is a unique identifier prediction model that has been machine-trained so that, when any one of N (N is an integer of 2 or more) feature vectors sampled as a teacher vector group from the learning data, which is a set of feature vectors, is input, the unique identifier prediction model outputs a unique identifier that is uniquely assigned to each of the feature vectors that constitute the teacher vector group; the reference histogram is a histogram showing a distribution of unique identifications output by the unique identifier prediction model when a reference vector group, which is a set of feature vectors sampled from the learning data and includes feature vectors different from the feature vectors constituting the teacher vector group, is input to the unique identifier prediction model; program.

Images