Information processing apparatus, information processing method, and information processing program

Abstract

To enhance accuracy of determination processing using a JIT model.SOLUTION: An information processing apparatus 100 performs first determination processing based on predicted values of objective variables corresponding to combinations of values of respective explanatory variables indicated by a first query 101, calculated by a first JIT model 110. When a result of the first determination processing is an error, the information processing apparatus 100 identifies, based on the first JIT model 110, a first explanatory variable having a relatively large degree of impact on the error. Upon acquiring a second query 102, the information processing apparatus 100 learns a second JIT model 120 that emphasizes explanatory variables other than the first explanatory variable over the first explanatory variable. The information processing apparatus 100 performs second determination processing based on predicted values of objective variables corresponding to combinations of values of respective explanatory variables indicated by the acquired second query 102, calculated using the learned second JIT model 120.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, there is a technique for learning a JIT (Just In Time) model that enables calculation of a predicted value of a current target variable corresponding to a combination of values of each of a plurality of explanatory variables at the current time. For example, among a group of learning data including learning data that associates a value of a target variable with a combination of values of each of the explanatory variables, the JIT model is learned based on the learning data showing a combination similar to the combination of values of each of the current explanatory variables. Further, it is conceivable to perform a determination process for determining whether or not the actually measured value of the current target variable is normal based on the predicted value of the target variable based on the JIT model. For example, it is conceivable to perform a determination process for setting a range of normal values based on the predicted value of the target variable based on the JIT model and determining whether or not the actually measured value of the current target variable is included in the range of normal values.

[0003] As prior art, for example, a plurality of candidate models for detecting abnormalities in sensor data are generated by a plurality of methods, the determination accuracy of the plurality of candidate models is calculated, and one or more candidate models are selected from among the plurality of candidate models based on the determination accuracy of the plurality of candidate models. Further, for example, there is a technique for updating the weights in a model so that abnormal sound data is determined to be abnormal and the probability that normal sound data is determined to be abnormal is minimized.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art, there are cases where the determination process using the JIT model cannot be accurately performed. For example, among a plurality of explanatory variables, if there is an explanatory variable that causes the predicted value of the target variable based on the JIT model to deviate far from the normal value, the determination process using the JIT model cannot be accurately performed.

[0006] In one aspect, the present invention aims to improve the accuracy of the determination process using the JIT model.

Means for Solving the Problems

[0007] According to one embodiment, among a group of learning data including learning data that associates the value of an objective variable with combinations of the values of a plurality of explanatory variables, when the result of a predetermined determination process based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the first query, calculated by the first JIT model learned using the learning data indicating a combination similar to the combination of the values of the explanatory variables indicated by the first query, is incorrect, based on the first JIT model, among the plurality of explanatory variables, a first explanatory variable having a greater degree of influence on the error than other explanatory variables, or a degree of influence on the error being equal to or greater than a threshold value is specified, a second query indicating the combination of the values of the explanatory variables is obtained, and among the group of learning data including learning data that associates the value of the objective variable with the combination of the values of the explanatory variables, using the learning data indicating a combination similar to the combination of the values of the explanatory variables indicated by the obtained second query, a second JIT model that emphasizes other explanatory variables other than the specified first explanatory variable more than the specified first explanatory variable is learned, and based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the obtained second query, calculated using the learned second JIT model, an information processing apparatus, an information processing method, and an information processing program for performing the determination process are proposed.

Effect of the Invention

[0008] According to one aspect, it becomes possible to improve the accuracy of the determination process using the JIT model.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Embodiments for Carrying Out the Invention

[0010] Hereinafter, with reference to the drawings, embodiments of an information processing apparatus, an information processing method, and an information processing program according to the present invention will be described in detail.

[0011] (An Example of the Information Processing Method According to the Embodiment) FIG. 1 is an explanatory diagram showing an example of the information processing method according to the embodiment. The information processing apparatus 100 is a computer for improving the accuracy of a predetermined determination process.

[0012] The predetermined determination process is performed, for example, using the feature value at the target time point. The predetermined determination process corresponds to, for example, determining whether the feature value at the target time point satisfies a predetermined condition. The predetermined condition may be, for example, a condition for determining whether the feature value at the target time point is normal or abnormal. For example, if the feature value is abnormal, an alert indicating that the feature value is abnormal may be output.

[0013] For example, a predetermined determination process is used in the field of anomaly detection. For example, it is conceivable that the characteristic value is an index value related to communication. Specifically, it is conceivable that the index value is a packet loss rate or the like. And, by performing a predetermined determination process for determining whether the packet loss rate satisfies a predetermined condition, it is conceivable to determine whether the packet loss rate is abnormal, and based on the determined result, to detect an abnormality in communication. For example, when an abnormality in communication is detected, an alert addressed to the user is output. The user is, for example, an administrator of a system related to communication or the like.

[0014] Conventionally, there has been a tendency to use a fixed threshold value for a predetermined condition. For example, it is conceivable to determine whether the characteristic value at the target time point is normal or abnormal by performing a predetermined determination process for determining whether the characteristic value at the target time point is included in the range of normal values defined by a fixed threshold value. The threshold value is set by, for example, the user.

[0015] However, in this case, it is difficult to accurately determine whether the characteristic value at the target time point is normal or abnormal. For example, it is difficult for the user to appropriately define the range of normal values using a fixed threshold value. Specifically, the user may define the range of normal values to be wider or narrower than the appropriate range. For example, the wider the defined range of normal values is than the appropriate range, the more the probability of misjudging an actually abnormal characteristic value as normal increases. On the other hand, for example, the narrower the defined range of normal values is than the appropriate range, the more the probability of misjudging an actually normal characteristic value as abnormal increases.

[0016] Therefore, based on the result of the determination process, it becomes difficult to appropriately output an alert indicating that the feature value is abnormal. For example, for a feature value that is actually normal, an alert indicating that the feature value is abnormal may be erroneously output, leading to an increase in the workload of the user. For example, for a feature value that is actually abnormal, an alert indicating that the feature value is abnormal may not be output, and the user may overlook the fact that there is an abnormal feature value, making it difficult to grasp the cause of the abnormal feature value. For this reason, it is desirable to appropriately determine whether the feature value is normal or abnormal and output an alert at an appropriate timing. Specifically, it is desirable to dynamically define the range of normal values using a dynamic threshold value.

[0017] On the other hand, a method of performing the determination process using the JIT model can be considered. Specifically, the feature value at the target time point is used as the measured value of the target variable, and using the JIT model, the predicted value of the target variable corresponding to the combination of the values of each explanatory variable of the plurality of explanatory variables at the target time point is calculated, and based on the calculated predicted value, it is considered possible to define the range of normal values. And specifically, it is considered possible to determine whether the feature value at the target time point, which is the measured value of the target variable, is included in the defined range of normal values.

[0018] The JIT model is learned based on a learning data group including learning data that associates the value of the target variable with the combination of the values of each explanatory variable of the plurality of explanatory variables. The JIT model is learned, for example, based on learning data showing a combination similar to the combination of the values of each explanatory variable of the plurality of explanatory variables at the target time point among the learning data group. The range of normal values can be defined, for example, as the range of ±3σ of the predicted value of the target variable at the target time point using the standard deviation σ in the JIT model.

[0019] Even with this method, it may be difficult to accurately determine whether the feature value at the target time point is normal or abnormal. For example, the worse the accuracy of the predicted value of the target variable at the target time point calculated by the JIT model, the worse the accuracy of the range of normal values defined based on the predicted value. Therefore, for example, it is likely to deteriorate the accuracy of determining whether the feature value at the target time point, which is the measured value of the target variable, is normal or abnormal.

[0020] Specifically, when there is a bias in the distribution of combinations of values of each explanatory variable shown by the learning data included in the learning data group and used when learning the JIT model, there may be a region where it is difficult to accurately calculate the predicted value of the target variable by the JIT model. Specifically, among the spaces of combinations of values of each explanatory variable, the predicted value of the target variable corresponding to the region where the combination shown by the learning data does not exist tends to be difficult to calculate accurately.

[0021] At this time, among the plurality of explanatory variables, there may be an explanatory variable that is a factor for moving the predicted value of the target variable based on the JIT model away from the actual normal value, and it becomes difficult to accurately perform the determination process using the JIT model. Therefore, the accuracy of the range of normal values defined based on the predicted value of the target variable corresponding to the region where the combination shown by the learning data does not exist tends to deteriorate, and the accuracy of determining whether the feature value at the target time point, which is the measured value of the target variable, is normal or abnormal, tends to deteriorate.

[0022] In addition, a method of determining whether the feature value at the target time point among a plurality of time points is normal or abnormal by analyzing the feature values at each of the plurality of time points in time series can be considered, but there is a problem that it is likely to increase the processing burden.

[0023] Therefore, in the present embodiment, an information processing method capable of improving the accuracy of the determination process using the JIT model will be described.

[0024] In FIG. 1, the information processing apparatus 100 stores a learning data group including learning data that associates a combination of the value of the target variable and the values of the respective explanatory variables. The value of the target variable indicated by the learning data represents the past value of the value to be obtained by prediction. The information processing apparatus 100 holds items of explanatory variables used for predicting the target variable. If the information processing apparatus 100 acquires new learning data, it updates the learning data group by adding the acquired new learning data to the learning data group.

[0025] (1-1) The information processing apparatus 100 acquires a first query 101. The first query 101 includes, for example, a combination of the values of the respective explanatory variables. Specifically, the first query 101 includes a combination of the values of the respective explanatory variables at the first time point. The first query 101 may further include, for example, the measured value of the target variable. Specifically, the first query 101 may include the measured value of the target variable at the first time point. The first query 101 requests, for example, to determine whether the measured value of the target variable is included in the range of normal values.

[0026] The information processing apparatus 100 learns a first JIT model 110. The information processing apparatus 100 learns and creates the first JIT model 110, for example, using the learning data indicating a combination similar to the combination of the values of the respective explanatory variables indicated by the acquired first query 101 among the learning data group at the first time point. Similarity means that the distance between information vectors indicating combinations of the values of the respective explanatory variables is equal to or less than a certain value. The first JIT model 110 is represented by, for example, a mathematical formula including the items of the respective explanatory variables.

[0027] (1-2) The information processing apparatus 100 performs a first determination process based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the first query 101, calculated by the learned first JIT model 110. The first determination process is, for example, a process of determining whether the measured value of the target variable is included in the set range of normal values.

[0028] The information processing apparatus 100 calculates, for example, a predicted value and a standard deviation of an objective variable corresponding to a combination of values of respective explanatory variables indicated by a first query 101 based on the learned first JIT model 110. The information processing apparatus 100 sets, for example, a range of ±(3×standard deviation) based on the calculated predicted value as a range of normal values. The information processing apparatus 100 performs, for example, a first determination process of determining whether or not the actually measured value of the objective variable at the first time point is included in the set range of normal values.

[0029] (1-3) When the result of the first determination process is incorrect, the information processing apparatus 100 identifies, based on the first JIT model 110, a first explanatory variable among a plurality of explanatory variables that has a relatively large degree of influence on the error. The first explanatory variable is an explanatory variable among the plurality of explanatory variables that has a greater degree of influence on the error than other explanatory variables or that has a degree of influence on the error equal to or greater than a threshold value. The first explanatory variable may be one or a plurality.

[0030] The information processing apparatus 100 receives, for example, an input of feedback information indicating whether or not the result of the first determination process is incorrect based on a user's operation input by outputting the first determination result so that the user can refer to it. When the result of the first determination process is incorrect, the information processing apparatus 100 identifies, in the mathematical formula of the first JIT model 110, an explanatory variable that causes the predicted value of the objective variable to change most greatly in the direction in which the result of the determination process becomes incorrect among the plurality of explanatory variables as the first explanatory variable.

[0031] (1-4) The information processing apparatus 100 acquires a second query 102. The second query 102 indicates a combination of values of respective explanatory variables. Specifically, the second query 102 includes a combination of values of respective explanatory variables at a second point in time. The second query 102 may further include, for example, an actually measured value of an objective variable. Specifically, the second query 102 may include an actually measured value of the objective variable at the second point in time. For example, the second query 102 requests to determine whether the actually measured value of the objective variable is included in a normal value range. The second query 102 is a query different from the first query 101. For example, the second query 102 is a new query acquired after the first query 101.

[0032] (1-5) The information processing apparatus 100 learns a second JIT model 120. For example, the information processing apparatus 100 uses learning data indicating a combination similar to the combination of values of respective explanatory variables indicated by the acquired second query 102 among the learning data group at the second point in time to learn the second JIT model 120. The second JIT model 120 is learned by emphasizing other explanatory variables other than the specified first explanatory variable among a plurality of explanatory variables more than the specified first explanatory variable. Similarity indicates that the distance between information vectors indicating combinations of values of respective explanatory variables is equal to or less than a certain value. The second JIT model 120 is represented by, for example, a mathematical formula including terms of respective explanatory variables.

[0033] Specifically, the information processing apparatus 100 learns a second JIT model 120 that uses other explanatory variables other than the specified first explanatory variable without using the specified first explanatory variable among a plurality of explanatory variables. Specifically, the information processing apparatus 100 may learn a second JIT model 120 in which weights are assigned to respective explanatory variables so as to emphasize other explanatory variables more than the specified first explanatory variable.

[0034] The information processing apparatus 100 performs a second determination process based on the predicted values of the objective variables corresponding to the combinations of the values of the explanatory variables indicated by the acquired second query 102, which are calculated using the learned second JIT model 120. The second determination process is, for example, a process of determining whether the measured value of the objective variable is included in the set normal value range.

[0035] The information processing apparatus 100 calculates, for example, the predicted value and the standard deviation of the objective variable corresponding to each combination of the values of the explanatory variables indicated by the acquired second query 102 by means of the learned second JIT model 120. The information processing apparatus 100 sets, for example, the range of ±(3 × standard deviation) based on the calculated predicted value as the normal value range. The information processing apparatus 100 performs, for example, a second determination process of determining whether the measured value of the objective variable at the second time point is included in the set normal value range.

[0036] Thereby, the information processing apparatus 100 can improve the accuracy of the determination process using the JIT model. The information processing apparatus 100 can appropriately vary, for example, the range to be treated as the normal value range using the JIT model, and can accurately perform the determination process of determining whether the measured value of the objective variable is included in the normal value range. Therefore, the information processing apparatus 100 can appropriately output an alert indicating that the measured value of the objective variable is abnormal based on the result of the determination process. The user can, for example, not overlook the fact that there is an abnormal measured value of the objective variable and can grasp the cause of the abnormal measured value of the objective variable. In addition, the information processing apparatus 100 can reduce the processing load.

[0037] Here, the case where the information processing apparatus 100 learns the first JIT model 110 and performs the first determination process using the first JIT model 110 has been described, but it is not limited to this. For example, there may be other computers that learn the first JIT model 110 and perform the first determination process using the first JIT model 110. In this case, the information processing apparatus 100 acquires the first JIT model 110 and the result of the first determination process from another computer.

[0038] Here, the case where the second query 102 is a query different from the first query 101 has been described, but it is not limited to this. For example, the second query 102 may be the same query as the first query 101. For example, the second query 102 may be an old query acquired before the first query 101.

[0039] (An example of the information processing system 200) Next, an example of the information processing system 200 to which the information processing apparatus 100 shown in FIG. 1 is applied will be described with reference to FIG. 2.

[0040] FIG. 2 is an explanatory diagram showing an example of the information processing system 200. In FIG. 2, the information processing system 200 includes an information processing apparatus 100, a measurement apparatus 201, and a client apparatus 202.

[0041] In the information processing system 200, the information processing apparatus 100 and the measurement apparatus 201 are connected via a wired or wireless network 210. The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like. In the information processing system 200, the information processing apparatus 100 and the client apparatus 202 are connected via a wired or wireless network 210.

[0042] The information processing apparatus 100 receives, from the measuring apparatus 201, a measured value that is the value of the target variable or measured values that are the values of the respective explanatory variables of a plurality of explanatory variables. The information processing apparatus 100 generates data that associates and shows a combination of the value of the target variable and the values of the respective explanatory variables based on the received measured values. The information processing apparatus 100 accumulates the generated data as learning data. The information processing apparatus 100 may receive the generated data as a query.

[0043] When the information processing apparatus 100 receives a query, it learns a JIT model based on the accumulated learning data and performs a determination process using the learned JIT model. The determination process is a process of determining whether the value of the target variable is included in the range of normal values. When the information processing apparatus 100 determines that the value of the target variable is not included in the range of normal values, it transmits an alert to the client apparatus 202. The alert indicates that the value of the target variable is abnormal.

[0044] If the result of the determination process is incorrect, the information processing apparatus 100 identifies an explanatory variable having a relatively large degree of influence on the error. For example, when the information processing apparatus 100 receives a notification from the client apparatus 202 that the result of the determination process is incorrect, it identifies an explanatory variable having a relatively large degree of influence on the error. When the information processing apparatus 100 receives subsequent queries, it may learn a JIT model based on explanatory variables other than the identified explanatory variable and perform a determination process using the learned JIT model. The information processing apparatus 100 is, for example, a server or a PC (Personal Computer).

[0045] The measuring apparatus 201 is a computer that acquires measured values. The measuring apparatus 201 transmits the acquired measured values to the information processing apparatus 100. The measuring apparatus 201 is, for example, a PC, a tablet terminal, a smartphone, a wearable terminal, or a sensor device.

[0046] The client device 202 is a computer that receives alerts from the information processing device 100. The client device 202 may send a notification to the information processing device 100 that the result of the determination process is incorrect based on the user's operation input. The client device 202 is, for example, a PC, a tablet terminal, a smartphone, or a wearable terminal, etc.

[0047] Here, the case where the information processing device 100 is a device different from the measuring device 201 has been described, but it is not limited to this. For example, the information processing device 100 may have the function of the measuring device 201 and may also operate as the measuring device 201.

[0048] Here, the case where the information processing device 100 is a device different from the client device 202 has been described, but it is not limited to this. For example, the information processing device 100 may have the function of the client device 202 and may also operate as the client device 202.

[0049] (Hardware configuration example of the information processing device 100) Next, a hardware configuration example of the information processing device 100 will be described with reference to FIG. 3.

[0050] FIG. 3 is a block diagram showing a hardware configuration example of the information processing device 100. In FIG. 3, the information processing device 100 includes a CPU (Central Processing Unit) 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, and a recording medium 305. Also, each component is connected by a bus 300.

[0051] Here, the CPU 301 controls the overall operation of the information processing apparatus 100. The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash ROM. Specifically, for example, the flash ROM or ROM stores various programs, and the RAM is used as the work area of the CPU 301. The programs stored in the memory 302 are loaded into the CPU 301 to cause the CPU 301 to execute the coded processes.

[0052] The network I / F 303 is connected to the network 210 through a communication line and is connected to other computers via the network 210. Then, the network I / F 303 manages the interface between the network 210 and the internal components and controls the input / output of data from / to other computers. The network I / F 303 is, for example, a modem or a LAN adapter.

[0053] The recording medium I / F 304 controls the read / write of data to / from the recording medium 305 according to the control of the CPU 301. The recording medium I / F 304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, etc. The recording medium 305 is a non-volatile memory that stores the data written under the control of the recording medium I / F 304. The recording medium 305 is, for example, a disk, a semiconductor memory, a USB memory, etc. The recording medium 305 may be detachable from the information processing apparatus 100.

[0054] In addition to the components described above, the information processing apparatus 100 may have, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, etc. Also, the information processing apparatus 100 may have multiple recording medium I / Fs 304 and recording media 305. Further, the information processing apparatus 100 may not have the recording medium I / F 304 and the recording medium 305.

[0055] (Example of the hardware configuration of the measuring device 201) The hardware configuration example of the measurement device 201 is the same as the hardware configuration example of the information processing device 100 shown in FIG. 3, and thus the description thereof is omitted.

[0056] (Hardware Configuration Example of Client Device 202) The hardware configuration example of the client device 202 is the same as the hardware configuration example of the information processing device 100 shown in FIG. 3, and thus the description thereof is omitted.

[0057] (Functional Configuration Example of Information Processing Device 100) Next, a functional configuration example of the information processing device 100 will be described with reference to FIG. 4.

[0058] FIG. 4 is a block diagram showing a functional configuration example of the information processing device 100. The information processing device 100 includes a storage unit 400, an acquisition unit 401, a learning unit 402, a determination unit 403, a specification unit 404, and an output unit 405.

[0059] The storage unit 400 is realized by a storage area such as the memory 302 and the recording medium 305 shown in FIG. 3, for example. Hereinafter, the case where the storage unit 400 is included in the information processing device 100 will be described, but it is not limited thereto. For example, the storage unit 400 may be included in a device different from the information processing device 100, and the stored content of the storage unit 400 may be referable from the information processing device 100.

[0060] The acquisition unit 401 to the output unit 405 function as an example of a control unit. Specifically, the acquisition unit 401 to the output unit 405 realize their functions by causing the CPU 301 to execute a program stored in a storage area such as the memory 302 and the recording medium 305 shown in FIG. 3, or by the network I / F 303. The processing results of each functional unit are stored in a storage area such as the memory 302 and the recording medium 305 shown in FIG. 3, for example.

[0061] The storage unit 400 stores various information that is referenced or updated in the processing of each functional unit. The storage unit 400 stores a learning data group. The learning data shows the association between the value of the target variable and the combination of the values of a plurality of explanatory variables. The value of the explanatory variable is, for example, a measured value. The value of the explanatory variable is, for example, the number of packets. The value of the target variable is, for example, a measured value. The value of the target variable is, for example, the packet loss rate. The learning data is, for example, acquired by the acquisition unit 401.

[0062] The storage unit 400 stores a first explanatory variable among the plurality of explanatory variables, which has a relatively large degree of influence on causing an error in the result of a predetermined determination process using the JIT model. The predetermined determination process is, for example, to determine whether the value of the target variable is abnormal. Specifically, the predetermined determination process is to determine whether the value of the target variable is included in the range of normal values. The first explanatory variable is, for example, specified by the specifying unit 404.

[0063] The storage unit 400 stores the JIT model. The storage unit 400 stores, for example, a first JIT model using each explanatory variable. The first JIT model is represented by, for example, a mathematical formula including terms of each explanatory variable. The first JIT model is, for example, learned by the learning unit 402. The storage unit 400 may store, for example, a second JIT model that emphasizes other explanatory variables other than the first explanatory variable more than the first explanatory variable. The second JIT model is represented by, for example, a mathematical formula that does not include the term of the first explanatory variable and includes terms of other explanatory variables. The second JIT model may be represented by, for example, a mathematical formula in which weights are assigned to the terms of each explanatory variable so as to emphasize other explanatory variables more than the first explanatory variable. The second JIT model is, for example, learned by the learning unit 402.

[0064] The storage unit 400 stores conditions for performing a new predetermined determination process in consideration of the first explanatory variable. The conditions relate to, for example, a query. A query requests to perform a predetermined determination process. A query is data that associates a value of a target variable with a combination of values of each of a plurality of explanatory variables. The conditions are generated, for example, by the specifying unit 404.

[0065] The acquisition unit 401 acquires various types of information used in the processing of each functional unit. The acquisition unit 401 stores the acquired various types of information in the storage unit 400 or outputs them to each functional unit. Further, the acquisition unit 401 may output the various types of information stored in the storage unit 400 to each functional unit. The acquisition unit 401 acquires various types of information, for example, based on an operation input by a user. The acquisition unit 401 may receive various types of information from a device different from the information processing apparatus 100, for example.

[0066] The acquisition unit 401 acquires data that associates a value of a target variable with a combination of values of each of a plurality of explanatory variables. The acquisition unit 401 acquires, for example, data that associates a value of a target variable with a combination of values of each of a plurality of explanatory variables by receiving the data from another computer. The other computer is, for example, the measurement device 201 or the like. The acquisition unit 401 may acquire data by receiving an input of data that associates a value of a target variable with a combination of values of each of a plurality of explanatory variables based on an operation input by a user, for example.

[0067] The acquisition unit 401 outputs the acquired data to the learning unit 402 as a query, for example. Specifically, the acquisition unit 401 acquires the data acquired at the first time point as the first query at the first time point and outputs it to the learning unit 402. Specifically, the acquisition unit 401 acquires the data acquired at the second time point as the second query at the second time point and outputs it to the learning unit 402. The second time point is, for example, a time point after the first time point. The acquisition unit 401 may store the acquired data in the storage unit 400 as learning data, for example.

[0068] The acquisition unit 401 acquires a notification that the result of a predetermined determination process is incorrect. For example, the acquisition unit 401 acquires a notification that the result of a predetermined determination process is incorrect by receiving it from another computer. The other computer is, for example, the client device 202 or the like. The acquisition unit 401 may acquire a notification that the result of a predetermined determination process is incorrect, for example, by accepting an input of a notification that the result of a predetermined determination process is incorrect based on a user's operation input.

[0069] The acquisition unit 401 may accept a start trigger for starting the processing of any functional unit. The start trigger is, for example, that there has been a predetermined operation input by the user. The start trigger may be, for example, that predetermined information has been received from another computer. The start trigger may be, for example, that any functional unit has output predetermined information. For example, the acquisition unit 401 accepts, as a start trigger for starting the processing of the learning unit 402, that query data has been acquired.

[0070] The learning unit 402 learns a JIT model based on a learning data group. For example, the learning unit 402 extracts learning data indicating combinations similar to the combinations of the values of the respective explanatory variables indicated by the first query from among the learning data group. The learning unit 402 learns the first JIT model using, for example, the extracted learning data. Thereby, the learning unit 402 can calculate a predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables.

[0071] For example, the learning unit 402 extracts learning data indicating combinations similar to the combinations of the values of the respective explanatory variables indicated by the second query from among the learning data group. The learning unit 402 learns the second JIT model using, for example, the extracted learning data. The second JIT model is learned, for example, to prioritize other explanatory variables other than the specified first explanatory variable among the plurality of explanatory variables over the specified first explanatory variable.

[0072] Specifically, the learning unit 402 extracts learning data that shows a combination similar to the combination of the values of explanatory variables other than the first explanatory variable indicated by the acquired second query from among the learning data group. Then, specifically, the learning unit 402 learns the second JIT model based on the combination of the value of the target variable and the values of other explanatory variables indicated by the extracted learning data. Thereby, the learning unit 402 can learn a JIT model that can accurately calculate the predicted value of the target variable in consideration of the degree of influence on the error of the predetermined determination process for each explanatory variable.

[0073] For example, the learning unit 402 may determine whether the acquired second query satisfies a predetermined condition. The predetermined condition is set based on the first query, for example. Specifically, the predetermined condition is that the combination of the values of the explanatory variables indicated by the first query is similar to the combination of the values of the explanatory variables indicated by the second query. More specifically, the predetermined condition is that the distance between the information vector indicating the combination of the values of the explanatory variables indicated by the first query and the information vector indicating the combination of the values of the explanatory variables indicated by the second query is equal to or less than a threshold value. The predetermined condition is set by the specifying unit 404, for example.

[0074] For example, when the predetermined condition is satisfied, the learning unit 402 learns the second JIT model. On the other hand, for example, when the predetermined condition is not satisfied, the learning unit 402 learns the first JIT model. Thereby, the learning unit 402 can learn a JIT model that can accurately calculate the predicted value of the target variable in consideration of the degree of influence on the error of the predetermined determination process for each explanatory variable. The learning unit 402 can learn an appropriate JIT model for the second query based on the predetermined condition, and can accurately perform the predetermined determination process.

[0075] The determination unit 403 performs a predetermined determination process. For example, the determination unit 403 calculates the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the first query using the learned first JIT model. Then, the determination unit 403 performs a first determination process based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the calculated first query. The first determination process is, for example, to determine whether the value of the objective variable indicated by the first query is included in the normal value range.

[0076] Specifically, the determination unit 403 performs a second determination process to determine whether the value of the objective variable indicated by the first query is included in the normal value range set based on the calculated predicted value. Thereby, the determination unit 403 can perform the first determination process and can determine whether the value of the objective variable indicated by the first query is abnormal.

[0077] For example, the determination unit 403 calculates the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the acquired second query using the learned second JIT model. Then, the determination unit 403 performs a second determination process based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the calculated second query. The second determination process is, for example, to determine whether the value of the objective variable indicated by the second query is included in the normal value range.

[0078] Specifically, the determination unit 403 performs a second determination process to determine whether the value of the objective variable indicated by the second query is included in the normal value range set based on the calculated predicted value. Thereby, the determination unit 403 can accurately perform the second determination process and can determine whether the value of the objective variable indicated by the second query is abnormal.

[0079] When the acquired second query satisfies a predetermined condition, for example, the determination unit 403 calculates predicted values of the objective variables corresponding to the combinations of the values of the explanatory variables indicated by the acquired second query using the learned second JIT model. On the other hand, when the acquired second query does not satisfy the predetermined condition, for example, the determination unit 403 calculates predicted values of the objective variables corresponding to the combinations of the values of the explanatory variables indicated by the acquired second query using the learned first JIT model.

[0080] Then, the determination unit 403 performs a second determination process based on, for example, the predicted values of the objective variables corresponding to the combinations of the values of the explanatory variables indicated by the calculated second query. Specifically, the determination unit 403 performs a second determination process of determining whether the value of the objective variable indicated by the second query is included in the range of normal values set based on the calculated predicted values. Thereby, the determination unit 403 can accurately perform the second determination process and can determine whether the value of the objective variable indicated by the second query is abnormal. In the second determination process, the determination unit 403 can appropriately distinguish between the first JIT model and the second JIT model, making it easier to accurately perform the second determination process.

[0081] The specifying unit 404 specifies a first explanatory variable among the plurality of explanatory variables that has a relatively large degree of influence on causing an error in a predetermined determination process. For example, when the result of the first determination process is incorrect, the specifying unit 404 specifies, based on the first JIT model, a first explanatory variable among the plurality of explanatory variables that has a relatively large degree of influence on the error.

[0082] Specific part 404, specifically, when obtaining a notification that the result of a predetermined determination process is incorrect, based on the first JIT model, among a plurality of explanatory variables, identifies a first explanatory variable with a relatively large degree of influence on the error. More specifically, in the mathematical formula representing the first JIT model, among the plurality of explanatory variables, the explanatory variable that causes the predicted value of the target variable to change most greatly in the direction in which the result of the first determination process becomes incorrect is identified as the first explanatory variable. Thereby, specific part 404 can obtain a guideline for learning the second JIT model.

[0083] Specific part 404 may set conditions based on the first query. The conditions are conditions for using a second JIT model that is learned to prioritize other explanatory variables other than the identified first explanatory variable among the plurality of explanatory variables over the identified first explanatory variable. For example, when the result of the first determination process is incorrect and the first explanatory variable is identified, specific part 404 sets conditions based on the first query.

[0084] Specific part 404 specifically sets a condition indicating that the distance between the information vector indicating the combination of the values of each explanatory variable indicated by the first query and the information vector indicating the combination of the values of each explanatory variable indicated by the second query is equal to or less than a threshold value. Thereby, specific part 404 can appropriately distinguish between using the first JIT model and the second JIT model, and can improve the accuracy of a predetermined determination process.

[0085] Output part 405 outputs the processing result of at least any one of the functional parts. The output format is, for example, display on a display, print output to a printer, transmission to an external device via network I / F 303, or storage in a storage area such as memory 302 or recording medium 305. Thereby, output part 405 can notify the user of the processing result of at least any one of the functional parts, and can improve the convenience of information processing apparatus 100.

[0086] The output unit 405 outputs, for example, the result of a predetermined determination process. Specifically, the output unit 405 outputs the result of the predetermined determination process so that it can be referred to by the user. More specifically, the output unit 405 displays the result of the predetermined determination process on a display. More specifically, the output unit 405 may also transmit the result of the predetermined determination process to another computer. The other computer is, for example, the client device 202. Thereby, the output unit 405 can make the result of the predetermined determination process available for the user to refer to. The output unit 405 can enable the user to determine whether the result of the predetermined determination process is incorrect.

[0087] The output unit 405 outputs, for example, an alert based on the result of a predetermined determination process. For example, when the result of the predetermined determination process indicates that the value of the target variable is not within the normal value range and is abnormal, the output unit 405 outputs an alert indicating that the value of the target variable is abnormal so that it can be referred to by the user. Specifically, the output unit 405 displays the alert on a display. Specifically, the output unit 405 may also transmit the alert to another computer. The other computer is, for example, the client device 202. Thereby, the output unit 405 can enable the user to grasp that the value of the target variable is not within the normal value range and is abnormal.

[0088] The output unit 405 outputs, for example, the first JIT model. For example, the output unit 405 outputs the first JIT model so that it can be referred to by the user. Specifically, the output unit 405 displays the first JIT model on a display. Specifically, the output unit 405 transmits the first JIT model to another computer. The other computer is, for example, the client device 202. Thereby, the output unit 405 can make the first JIT model available for the user to use.

[0089] The output unit 405 outputs, for example, the specified first explanatory variable. Specifically, the output unit 405 displays the specified first explanatory variable on a display. Specifically, the output unit 405 transmits the specified first explanatory variable to another computer. The other computer is, for example, the client device 202. Thereby, the output unit 405 enables the user to grasp which explanatory variables have a relatively large adverse effect on a predetermined determination process.

[0090] The output unit 405 outputs, for example, the second JIT model. For example, the output unit 405 outputs the second JIT model so that the user can refer to it. Specifically, the output unit 405 displays the second JIT model on a display. Specifically, the output unit 405 transmits the second JIT model to another computer. The other computer is, for example, the client device 202. Thereby, the output unit 405 enables the user to use the second JIT model.

[0091] Here, the case where the information processing apparatus 100 includes each functional unit has been described, but it is not limited thereto. For example, the information processing apparatus 100 may not include any of the functional units. Specifically, the information processing apparatus 100 may not include the learning unit 402 and may cooperate with another computer including the learning unit 402.

[0092] Also, here, the case where the information processing apparatus 100 performs the first determination process and the second determination process has been described, but it is not limited thereto. For example, the information processing apparatus 100 may not perform the first determination process and may cooperate with another computer that performs the first determination process.

[0093] (Specific functional configuration example of the information processing apparatus 100) Next, a specific functional configuration example of the information processing apparatus 100 will be described with reference to FIG. 5.

[0094] FIG. 5 is a block diagram showing a specific functional configuration example of the information processing apparatus 100. In FIG. 5, the measuring device has a measuring function 501. The measuring device has a measurement DB (DataBase) 511.

[0095] The measuring function 501 measures the value of each of a plurality of items at each predetermined measurement timing, and generates measurement data indicating a combination of the measured values of the respective items. The predetermined measurement timing is, for example, a timing at regular intervals. The plurality of items includes, for example, an item that becomes a target variable. The plurality of items includes, for example, an item that becomes an explanatory variable. The measuring function 501 stores the generated measurement data in the measurement DB 511 in association with the current measurement timing. The measurement DB 511 accumulates the measurement data.

[0096] The information processing apparatus 100 has an aggregating function 502. The information processing apparatus 100 has an aggregation DB 512.

[0097] The aggregating function 502 acquires, from the measuring device, a group of measurement data accumulated in the measurement DB 511 after the previous creation timing at each predetermined creation timing. The predetermined creation timing is, for example, a timing at regular intervals. The aggregating function 502 extracts, based on each of one or more aggregation conditions, the measurement data that satisfies the aggregation condition from among the acquired group of measurement data. The aggregation condition is, for example, a condition related to any one of the plurality of items. The aggregating function 502 generates aggregated data obtained by aggregating the extracted measurement data. The aggregating function 502 stores the generated aggregated data in the aggregation DB 512 in association with the time point indicating the current creation timing. The aggregation DB 512 accumulates the aggregated data.

[0098] The information processing apparatus 100 has a reading function 521, a creation function 522, an adjustment function 523, a prediction function 524, a determination function 525, a notification function 526, an input function 527, and an analysis function 528. The information processing apparatus 100 has a learning DB 531 and a pattern DB 532.

[0099] The loading function 521 acquires the aggregated data accumulated in the aggregation DB 512 since the previous determination timing at each predetermined determination timing. The creation function 522 stores the aggregated data as learning data in the learning DB 531. The learning DB 531 stores the learning data.

[0100] The adjustment function 523 extracts the aggregated data corresponding to the item to be determined from the acquired aggregated data. For each of the extracted aggregated data, the adjustment function 523 sets the aggregated data as a query, and refers to the pattern DB 532 to determine whether the query corresponds to any of the false determination query patterns. The pattern DB 532 stores the false determination query patterns. The false determination query pattern indicates, for example, the criteria for properly using the JIT models. The false determination query pattern is generated, for example, by the analysis function 528.

[0101] If the set query does not correspond to any of the false determination query patterns, the adjustment function 523 sets each of the plurality of explanatory variables as an explanatory variable to be used in the determination process. If the set query corresponds to any of the false determination query patterns, the adjustment function 523 sets one or more explanatory variables associated with the any of the false determination query patterns among the plurality of explanatory variables as the target explanatory variables to be used in the determination process. Thereby, the adjustment function 523 can appropriately set the target explanatory variables to be used when learning the JIT model.

[0102] The prediction function 524 extracts the learning data indicating a combination similar to the combination of the values of the target explanatory variables indicated by the query from the learning data group in the learning DB 531. Based on the extracted learning data, the prediction function 524 learns a JIT model including the item of the target explanatory variable. Based on the learned JIT model, the prediction function 524 calculates the predicted value and the standard deviation of the target variable corresponding to the combination of the values of the target explanatory variables indicated by the query. The prediction function 524 sets the range of ±(3×standard deviation) based on the predicted value as the range of normal values.

[0103] The determination function 525 determines whether the value of the target variable indicated by the query is abnormal by determining whether the value of the target variable indicated by the query is within the range of the set normal values. The notification function 526 outputs an alert indicating that the value of the target variable indicated by the query is abnormal to the user when it is determined that the value of the target variable indicated by the query is abnormal. The input function 527 accepts feedback on whether the result of the determination process indicating that the value of the target variable indicated by the query is abnormal is incorrect based on the user's operation input.

[0104] When the result of the determination process is incorrect, the analysis function 528 generates an incorrect determination query pattern based on the query. The incorrect determination query pattern can identify, for example, at least an unfavorable explanatory variable as the target explanatory variable. The incorrect determination query pattern can identify, for example, for which queries there are unfavorable explanatory variables as the target explanatory variables.

[0105] For example, based on the JIT model, the analysis function 528 identifies an explanatory variable with a relatively large degree of influence causing the determination result to be incorrect among the target explanatory variables as an unfavorable explanatory variable as the target explanatory variable. The analysis function 528 identifies it on the condition that the combination of the values of the target explanatory variables indicated by subsequent queries exists within a certain distance range from the combination of the values of the target explanatory variables indicated by the current query. The analysis function 528 stores in the pattern DB 532 an incorrect determination query pattern indicating the association between the identified explanatory variable and the identified condition.

[0106] Thereby, the information processing apparatus 100 can accurately perform the determination process of determining whether the value of the target variable indicated by the query is abnormal by determining whether the value of the target variable indicated by the query is within the range of the normal values set based on the JIT model. The information processing apparatus 100 can appropriately use the explanatory variables used when learning the JIT model for the determination process among the plurality of explanatory variables according to the query, and can enable the determination process to be accurately performed.

[0107] (Flow of Operations of Information Processing Apparatus 100) Next, with reference to FIGS. 6 and 7, the flow of operations of information processing apparatus 100 will be described.

[0108] FIGS. 6 and 7 are explanatory diagrams showing the flow of operations of information processing apparatus 100. In FIG. 6, information processing apparatus 100 stores a plurality of explanatory variables 601. Information processing apparatus 100 stores a learning data group including learning data that associates and shows a combination of the value of the target variable and the value of each explanatory variable. The value of the target variable indicated by the learning data indicates the correct value.

[0109] If information processing apparatus 100 acquires new learning data, it updates the learning data group by adding the acquired new learning data to the learning data group. The learning data group includes, for example, past normal-time data 602 corresponding to the case where the value of the target variable is included in the normal value range and is normal.

[0110] (6-1) Based on the past normal-time data 602, information processing apparatus 100 identifies a plurality of determination-use explanatory variables 603 among the plurality of explanatory variables 601. For example, information processing apparatus 100 identifies, as the plurality of determination-use explanatory variables 603, a plurality of explanatory variables determined to be easy to represent the value of the target variable indicated by the past normal-time data 602.

[0111] (6-2) Information processing apparatus 100 acquires determination target data 604 that associates and shows a combination of the value of the target variable and the value of each explanatory variable as a first query.

[0112] When information processing apparatus 100 acquires the first query, it extracts learning data showing an information vector that is relatively close to the information vector indicated by the determination target data 604 that is the first query from among the learning data group. The information vector indicates, for example, a combination of the values of the respective explanatory variables 603 of the plurality of explanatory variables 603. The distance is the distance between vectors. Based on the extracted learning data, information processing apparatus 100 learns a first JIT model represented by a mathematical formula including terms of the respective explanatory variables 603.

[0113] (6-3) The information processing device 100 calculates a predicted value of the target variable corresponding to the information vector indicated by the first query and the standard deviation σ according to the learned first JIT model. The information processing device 100 sets the range of the calculated predicted value ± 3σ as the range of normal values. The information processing device 100 determines whether the value of the target variable indicated by the first query is abnormal by determining whether the value of the target variable indicated by the first query is included in the set range of normal values.

[0114] (6-4) The information processing device 100 outputs the determination result so that the user can refer to it. The information processing device 100 determines whether the determination result is incorrect based on the user's operation input. If the determination result is incorrect, the information processing device 100 identifies the explanatory variable 603 that causes the determination result to be incorrect among the plurality of explanatory variables 603. The information processing device 100 excludes the identified explanatory variable 603 from the plurality of explanatory variables 603 and identifies a new plurality of explanatory variables 605 for determination.

[0115] (6-5) The information processing device 100 extracts learning data indicating an information vector whose distance from the information vector indicated by the determination target data 604 serving as the first query among the learning data group is relatively close. The information vector indicates, for example, a combination of the values of each of the plurality of explanatory variables 605. The information processing device 100 learns a second JIT model represented by a mathematical formula including terms of each of the explanatory variables 605 based on the extracted learning data.

[0116] (6-6) The information processing device 100 calculates a predicted value of the target variable corresponding to the information vector indicated by the first query and the standard deviation σ according to the learned second JIT model. The information processing device 100 sets the range of the calculated predicted value ± 3σ as the range of normal values. The information processing device 100 determines whether the value of the target variable indicated by the first query is abnormal by determining whether the value of the target variable indicated by the first query is included in the set range of normal values.

[0117] (6-7) The information processing apparatus 100 outputs the determination result so that the user can refer to it. The information processing apparatus 100 determines whether the determination result is incorrect based on the user's operation input. If the determination result is not incorrect, the information processing apparatus 100 generates a condition for determining that it is preferable to use a plurality of explanatory variables 605 instead of a plurality of explanatory variables 603 when learning the JIT model based on the first query.

[0118] For example, the information processing apparatus 100 generates, as a condition, that subsequent queries indicate information vectors within a certain distance range from the information vector indicated by the first query. The information processing apparatus 100 generates an incorrect determination query pattern that associates the generated condition with a plurality of explanatory variables 605 and stores it in the pattern DB 532.

[0119] Thereby, the information processing apparatus 100 can detect subsequent queries that are likely to cause a decrease in determination accuracy when using a plurality of explanatory variables 603 when learning the JIT model. For example, if subsequent queries are queries similar to the current query having the same characteristics as the current query, it is likely to cause a decrease in determination accuracy. Specifically, it is conceivable that the information vector indicated by a subsequent query is relatively close to the information vector indicated by the current query.

[0120] In other words, the information processing apparatus 100 can detect subsequent queries similar to the current query that are determined to be likely to improve the determination accuracy when using a plurality of explanatory variables 605 instead of a plurality of explanatory variables 603 when learning the JIT model. Also, when the information processing apparatus 100 acquires subsequent queries similar to the current query, it can make a plurality of explanatory variables 605 identifiable. Next, we will move on to the description of FIG. 7.

[0121] In FIG. 7, (7-1) Based on the past normal data 602, the information processing apparatus 100 assumes that a plurality of determination explanatory variables 603 among the plurality of explanatory variables 601 have been specified. The information processing apparatus 100 may re-specify a new plurality of determination explanatory variables 603.

[0122] (7-2) The information processing apparatus 100 acquires determination target data 701 that associates and shows a combination of the value of the target variable and the values of the respective explanatory variables as a second query. The information processing apparatus 100 determines whether the second query satisfies the conditions included in the misjudgment query pattern stored in the pattern DB 532.

[0123] For example, the information processing apparatus 100 determines that the condition is satisfied when the information vector indicated by the second query exists within a certain distance range from the information vector indicated by the first query included in the misjudgment query pattern. For example, the information processing apparatus 100 determines that the condition is not satisfied when the information vector indicated by the second query does not exist within a certain distance range from the information vector indicated by the first query included in the misjudgment query pattern.

[0124] Here, when the information processing apparatus 100 determines that the condition included in the misjudgment query pattern is not satisfied, it proceeds to the processing of (7-3) and (7-4). On the other hand, when the information processing apparatus 100 determines that the condition included in the misjudgment query pattern is satisfied, it proceeds to the processing of (7-5) and (7-6).

[0125] (7-3) When the information processing apparatus 100 determines that the condition is not satisfied, it extracts learning data that shows an information vector that is relatively close to the information vector indicated by the determination target data 701 that becomes the second query from among the learning data group. The information vector indicates, for example, a combination of the values of the respective explanatory variables 603 of the plurality of explanatory variables 603. The distance is the distance between vectors. The information processing apparatus 100 learns a third JIT model represented by a mathematical formula including terms of the respective explanatory variables 603 based on the extracted learning data.

[0126] (7-4) The information processing apparatus 100 calculates a predicted value of the target variable corresponding to the information vector indicated by the second query and the standard deviation σ according to the learned third JIT model. The information processing apparatus 100 sets the range of the calculated predicted value ±3σ as the range of normal values. The information processing apparatus 100 determines whether the value of the target variable indicated by the second query is abnormal by determining whether the value of the target variable indicated by the second query is included in the set range of normal values. The information processing apparatus 100 outputs the determination result so that the user can refer to it.

[0127] (7-5) When it is determined that the conditions are satisfied, the information processing apparatus 100 extracts learning data indicating an information vector that is relatively close to the information vector indicated by the determination target data 701 that becomes the second query from among the learning data group. The information vector indicates, for example, a combination of the values of the respective explanatory variables 605 of the plurality of explanatory variables 605. The information processing apparatus 100 learns a fourth JIT model represented by a mathematical formula including terms of the respective explanatory variables 605 based on the extracted learning data.

[0128] (7-6) The information processing apparatus 100 calculates a predicted value of the target variable corresponding to the information vector indicated by the second query and the standard deviation σ according to the learned fourth JIT model. The information processing apparatus 100 sets the range of the calculated predicted value ±3σ as the range of normal values. The information processing apparatus 100 determines whether the value of the target variable indicated by the second query is abnormal by determining whether the value of the target variable indicated by the second query is included in the set range of normal values. The information processing apparatus 100 outputs the determination result so that the user can refer to it.

[0129] As a result, the information processing apparatus 100 can improve the accuracy of the determination process using the JIT model. For example, the information processing apparatus 100 can appropriately vary the range to be treated as the normal value range using the JIT model, and can accurately perform a determination process for determining whether the value of the target variable is included in the normal value range. Therefore, the information processing apparatus 100 can appropriately output an alert indicating that the value of the target variable is abnormal based on the result of the determination process. For example, the user can avoid overlooking the fact that there is an abnormal value of the target variable and can grasp the cause of the abnormal value of the target variable.

[0130] (An example of the operation of the information processing apparatus 100) Next, an example of the operation of the information processing apparatus 100 will be described with reference to FIGS. 8 to 15.

[0131] FIGS. 8 to 15 are explanatory diagrams showing an example of the operation of the information processing apparatus 100. In FIG. 8, the information processing apparatus 100 is applied to the server system 800. For example, the information processing apparatus 100 treats the characteristic value of each item regarding the communication between the subnet of site A and server B as the value of each explanatory variable of a plurality of explanatory variables, or as the value of the target variable. Server B corresponds to, for example, a measuring device.

[0132] Specifically, the information processing apparatus 100 measures, for each time point, the packet loss rate, the number of packets, the number of bytes, etc. from the subnet of site A to server B based on the result of capturing communication packets on the communication path. Specifically, the information processing apparatus 100 measures, for each time point, the number of packets, the number of bytes, etc. from server B to the subnet of site A based on the result of capturing communication packets on the communication path.

[0133] In the following description, the packet loss rate from the subnet of Site A to Server B may be referred to as the "AtoB loss rate". In the following description, the number of packets from the subnet of Site A to Server B may be referred to as the "AtoB packet count". In the following description, the number of bytes from the subnet of Site A to Server B may be referred to as the "AtoB byte count".

[0134] In the following description, the number of packets from Server B to the subnet of Site A may be referred to as the "BtoA packet count". In the following description, the number of bytes from Server B to the subnet of Site A may be referred to as the "BtoA byte count". The information processing apparatus 100 acquires, at each point in time, data indicating a combination of measured values of the AtoB packet count, the AtoB byte count, the BtoA packet count, the BtoA byte count, and the AtoB loss rate as a query and learning data. The information processing apparatus 100 accumulates the acquired learning data.

[0135] The information processing apparatus 100 sets the AtoB packet count, the AtoB byte count, the BtoA packet count, the BtoA byte count, etc. as a plurality of explanatory variables for determination. The information processing apparatus 100 sets the AtoB loss rate as the target variable. And the information processing apparatus 100 aims to perform a predetermined determination process for determining whether or not the measured value of the AtoB loss rate at the current point in time included in the query is abnormal at each point in time.

[0136] The predetermined determination process determines whether or not the AtoB loss rate at the current point in time is abnormal by, for example, determining whether or not the measured value of the AtoB loss rate at the current point in time is within the range of normal values at each point in time. Next, the description will proceed to FIGS. 9 and 10.

[0137] In FIGS. 9 and 10, a case where the information processing apparatus 100 learns a JIT model based on a learning data group using a plurality of explanatory variables for determination and performs a predetermined determination process using the learned JIT model will be described. Here, the plurality of explanatory variables for determination are the number of A-to-B packets, the number of A-to-B bytes, the number of B-to-A packets, and the number of B-to-A bytes.

[0138] It is assumed that the information processing apparatus 100 learns a JIT model based on a learning data group at each time point, using the number of A-to-B packets, the number of A-to-B bytes, the number of B-to-A packets, and the number of B-to-A bytes as explanatory variables, and performs a predetermined determination process using the JIT model.

[0139] For example, at each time point, the information processing apparatus 100 extracts, as local data, learning data similar to the current query from the learning data group, and learns a JIT model based on the local data. The local data is, for example, the top n pieces of learning data in the learning data group that have a relatively short vector distance from the current query. n is, for example, 20. The vector distance is, for example, the distance in the space of the information vector indicating the combination of the values of the explanatory variables.

[0140] For example, the information processing apparatus 100 calculates a predicted value and a standard deviation of the current A-to-B loss rate using the learned JIT model, sets an upper limit of the normal value range, and determines whether the measured value of the current A-to-B loss rate is equal to or less than the upper limit of the set normal value range. For example, if the measured value of the current A-to-B loss rate is greater than the upper limit of the set normal value range, the information processing apparatus 100 determines that the measured value of the current A-to-B loss rate is abnormal.

[0141] The graph 900 in FIG. 9 shows the predicted value of the A-to-B loss rate (AtoB_LossRate), the upper limit of the normal value range, and the measured value at each time point. The dashed line in the graph 900 indicates the predicted value. The thin line in the graph 900 indicates the normal range upper limit, which is the upper limit value of the normal value range. The thick line in the graph 900 indicates the measured value. The predicted value is calculated, for example, by the JIT model.

[0142] Region 901 is a region where the measured value of the AtoB loss rate, which is the value of the target variable, has increased compared to normal due to access concentration. Therefore, it is preferable for the information processing apparatus 100 to determine that the measured value of the AtoB loss rate, which is the value of the target variable in region 901, is abnormal. Next, we will move on to the description of FIG. 10.

[0143] Graph 1000 in FIG. 10 is an enlarged view of region 901. As shown in FIG. 10, in the JIT model that learned the AtoB packet count, the AtoB byte count, the BtoA packet count, and the BtoA byte count as explanatory variables, the predicted value increases in region 901, and the upper limit of the normal value range tends to increase.

[0144] Therefore, when the information processing apparatus 100 uses the JIT model that learned the AtoB packet count, the AtoB byte count, the BtoA packet count, and the BtoA byte count as explanatory variables, it may be difficult to accurately perform a predetermined determination process. Next, using FIGS. 11 and 12, an example of a situation where the predicted value increases, the upper limit of the normal value range increases, and it becomes difficult to accurately perform a predetermined determination process will be described.

[0145] Graph 1100 in FIG. 11 is a scatter diagram showing the relationship between the current query and the learning data group including past learning data.

[0146] The horizontal axis of graph 1100 indicates the vector distance (norm) between the current query and past learning data. The vertical axis of graph 1100 indicates the AtoB loss rate of the current query or past learning data. The small white circles indicate past learning data. The small black circles indicate the learning data adopted as local data among the past learning data. The large black circle indicates the current query. The large white circle indicates the predicted value of the AtoB loss rate.

[0147] As shown in FIG. 11, there is local data with a relatively short vector distance from the current query, and the learned JIT model tends to accurately calculate the predicted value of the current A-to-B loss rate corresponding to the current query. In this case, it is considered that the predicted value is less likely to increase and the upper limit of the normal value range is less likely to increase, and a predetermined determination process can be accurately performed.

[0148] Graph 1200 in FIG. 12 is a scatter diagram showing the relationship between the current query and a group of learning data including past learning data.

[0149] The horizontal axis of graph 1200 indicates the vector distance (norm) between the current query and past learning data. The vertical axis of graph 1200 indicates the A-to-B loss rate of the current query or past learning data. Small white circles indicate past learning data. Small black circles indicate learning data adopted as local data among past learning data. A large black circle indicates the current query. A large white circle indicates the predicted value of the A-to-B loss rate.

[0150] As shown in FIG. 12, when there is no local data with a relatively short vector distance from the current query, the learned JIT model tends to have difficulty accurately calculating the predicted value of the current A-to-B loss rate corresponding to the current query. In this case, it is considered that the predicted value is likely to increase and the upper limit of the normal value range is likely to increase, making it difficult to accurately perform a predetermined determination process. Next, the description of FIG. 13 will be given.

[0151] In FIG. 13, the information processing apparatus 100 has learned a JIT model based on a learning data group and has already performed a predetermined determination process using the JIT model. The JIT model is represented by, for example, a mathematical formula including terms such as the number of A-to-B packets, the number of A-to-B bytes, the number of B-to-A packets, and the number of B-to-A bytes. As such a model creation method, for example, there is a learning method such as local linear regression. Specifically, the JIT model is represented by a predicted value = a × number of A-to-B packets + b × number of A-to-B bytes + c × number of B-to-A packets + d × number of B-to-A bytes + intercept. Here, the information processing apparatus 100 identifies an explanatory variable that causes the predetermined determination process to be incorrect based on the JIT model.

[0152] The information processing apparatus 100 detects that the result of a predetermined determination process using the learned JIT model is incorrect. For example, assume that the information processing apparatus 100 has received a notification from the user that the measured value of the target variable is abnormal, even though the result of the predetermined determination process using the JIT model learned during an abnormality indicates that the measured value of the target variable is normal. Thereby, the information processing apparatus 100 can detect, for example, that the result of the predetermined determination process is incorrect.

[0153] When the result of the predetermined determination process is incorrect, the information processing apparatus 100 identifies an explanatory variable that causes the predetermined determination process to be incorrect based on the JIT model used in the predetermined determination process. In the example of FIG. 13, since it becomes difficult to accurately perform the predetermined determination process as the predicted value increases, the information processing apparatus 100 identifies an explanatory variable with a large degree of increase in the predicted value as an explanatory variable that causes the predetermined determination process to be incorrect.

[0154] Specifically, the information processing apparatus 100 identifies the number of A-to-B bytes based on the value of each explanatory variable term shown in the graph 1300 corresponding to the case where the predetermined determination result is incorrect. When the predetermined determination result is correct, as shown in the graph 1300, it may be difficult to determine whether the value of each explanatory variable term causes the predetermined determination process to be incorrect. Next, the description will proceed to FIGS. 14 and 15.

[0155] In FIG. 14, the information processing apparatus 100 adopts explanatory variables other than the specified A-to-B byte count as explanatory variables used when learning the JIT model. The information processing apparatus 100 learns the JIT model based on the learning data group using the adopted explanatory variables, and performs a predetermined determination process using the learned JIT model. Here, the adopted explanatory variables are the A-to-B packet count, the B-to-A packet count, and the B-to-A byte count.

[0156] The information processing apparatus 100 learns the JIT model based on the learning data group at each point in time, using the A-to-B packet count, the B-to-A packet count, and the B-to-A byte count as explanatory variables, and performs a predetermined determination process using the JIT model.

[0157] For example, at each point in time, the information processing apparatus 100 extracts, as local data, learning data similar to the current query regarding the adopted explanatory variables from among the learning data group, and learns the JIT model based on the local data. The local data is, for example, the top n pieces of learning data in the learning data group that have a relatively short vector distance from the current query regarding the adopted explanatory variables. n is, for example, 20. The vector distance is, for example, the distance in the space of the information vector indicating the combination of the values of the adopted explanatory variables.

[0158] For example, the information processing apparatus 100 calculates a predicted value and a standard deviation of the current A-to-B loss rate using the learned JIT model, sets an upper limit of the normal value range, and determines whether the measured value of the current A-to-B loss rate is less than or equal to the upper limit of the set normal value range. For example, if the measured value of the current A-to-B loss rate is greater than the upper limit of the set normal value range, the information processing apparatus 100 determines that the measured value of the current A-to-B loss rate is abnormal.

[0159] Graph 1400 in FIG. 14 shows, at each point in time, the predicted value of the A-to-B loss rate (AtoB_LossRate), the upper limit of the normal value range, and the measured value. The dashed line in graph 1400 indicates the predicted value. The thin line in graph 1400 indicates the upper limit of the normal range, which is the upper limit value of the normal value range. The thick line in graph 1400 indicates the measured value. The predicted value is calculated, for example, by a JIT model.

[0160] Region 1401 is a region where, due to access concentration, the measured value of the A-to-B loss rate, which is the value of the target variable, has increased compared to normal times. Therefore, it is preferable for the information processing apparatus 100 to determine that the measured value of the A-to-B loss rate, which is the value of the target variable in region 1401, is abnormal. Next, we will move on to the description of FIG. 15.

[0161] Graph 1500 in FIG. 15 is an enlarged view of region 1401. As shown in FIG. 15, in a JIT model that learns using the number of A-to-B packets, the number of B-to-A packets, and the number of B-to-A bytes as explanatory variables without using the number of A-to-B bytes as an explanatory variable, the upper limit of the normal value range in region 1401 has a tendency not to increase significantly.

[0162] Therefore, when the information processing apparatus 100 uses a JIT model that learns using the number of A-to-B packets, the number of B-to-A packets, and the number of B-to-A bytes as explanatory variables, it can more easily perform a predetermined determination process with high accuracy.

[0163] Thereby, the information processing apparatus 100 can improve the accuracy of the determination process using the JIT model. The information processing apparatus 100 can, for example, identify explanatory variables that are not preferably used in the determination process, learn a JIT model using other explanatory variables other than the identified explanatory variables, and enable the determination process to be performed.

[0164] (Specific Example of the Operation of the Information Processing Apparatus 100) Next, a specific example of the operation of the information processing apparatus 100 will be described with reference to FIGS. 16 to 22.

[0165] Figures 16 to 22 are explanatory diagrams showing specific examples of the operation of the information processing apparatus 100. In Figure 16, the information processing apparatus 100 acquires a measurement data group shown in Table 1600 based on the result of capturing communication packets on the communication path between Site A and Server B. Table 1600 is realized by a storage area such as the memory 302 or the recording medium 305 of the information processing apparatus 100 shown in Figure 3, for example.

[0166] Table 1600 has fields for L4 analysis time, source IP (Internet Protocol), destination IP, protocol number, source Port, and destination Port. Table 1600 further has fields for the number of A-to-B packets, the number of bytes of A-to-B, the loss rate of A-to-B, the number of B-to-A packets, the number of bytes of B-to-A, the loss rate of B-to-A, RTT (round trip time), and server processing time. Table 1600 shows measurement data as records by setting information in each field.

[0167] In the field of L4 analysis time, the L4 analysis time indicating the time when L4 analysis is performed on the communication packet is set. In the field of source IP, the IP address indicating the source of the communication packet is set. In the field of destination IP, the IP address indicating the destination of the communication packet is set. In the field of protocol number, the protocol number related to the communication packet is set. In the field of source Port, the port number at the source related to the communication packet is set. In the field of destination Port, the port number at the destination related to the communication packet is set.

[0168] In the field of the number of A-to-B packets, the number of packets from Site A to Server B at the L4 analysis time is set. In the field of the number of bytes of A-to-B, the number of bytes from Site A to Server B at the L4 analysis time is set. In the field of the number of lost A-to-B packets, the number of packet losses from Site A to Server B at the L4 analysis time is set.

[0169] In the field of the number of BtoA packets, the number of packets from server B to site A at the L4 analysis time is set. In the field of the number of BtoA bytes, the number of bytes from server B to site A at the L4 analysis time is set. In the field of the number of BtoA losses, the number of packet losses from server B to site A at the L4 analysis time is set. In the field of RTT, the RTT between site A and server B at the L4 analysis time is set. In the field of server processing time, the server processing time between site A and server B at the L4 analysis time is set.

[0170] The information processing apparatus 100 refers to the table 1600, and for each L4 analysis time, aggregates the measurement data that satisfies the preset aggregation conditions for each preset aggregation condition, and generates aggregated data. For example, the information processing apparatus 100 aggregates the values of each item in the measurement data that satisfies the preset aggregation condition for each preset aggregation condition for each L4 analysis time, and generates aggregated data.

[0171] At this time, the information processing apparatus 100 may calculate the number of AtoB losses and the number of BtoA losses based on the number of AtoB losses and the number of BtoA losses, and include them in the aggregated data. The information processing apparatus 100 may calculate the number of connections, etc., and include them in the aggregated data. The aggregation conditions are, for example, condition-1 and condition-2, etc. Aggregation is, for example, calculating statistical values for each item. The statistical values are the average value, the mode, the median, the maximum value, or the minimum value, etc.

[0172] Condition-1 is, for example, that the protocol number is "6", the IP address indicating the source is "10.20.0.0 / 16", and the IP address indicating the destination is "10.20.30.50". Condition-2 is, for example, that the protocol number is "6", the IP address indicating the source is "20.30.0.0 / 16", and the IP address indicating the destination is "10.20.30.50".

[0173] The information processing apparatus 100 generates aggregated data indicated by reference numeral 1601 by aggregating measurement data that satisfies, for example, Condition-1. The information processing apparatus 100 generates aggregated data indicated by reference numeral 1602 by aggregating measurement data that satisfies, for example, Condition-2. The information processing apparatus 100 accumulates the generated aggregated data, for example. The aggregated data is used, for example, as a query or learning data. Next, the description will proceed to FIGS. 17 and 18.

[0174] In FIG. 17, the information processing apparatus 100 selects, for each aggregation condition, an item to be an explanatory variable of each of one or more explanatory variables that form an information vector based on a user's operation input. The information processing apparatus 100 may select, for each aggregation condition, an item to be an explanatory variable of each of one or more explanatory variables that form an information vector based on the accumulated past aggregated data group 1700 by the stepwise method. The aggregated data group 1700 includes, for example, aggregated data for the past three weeks.

[0175] Here, the aggregated data included in the aggregated data group 1700 tends to be aggregated data corresponding to the normal state where the value of the target variable is normal. For this reason, it is not always possible to accurately predict the value of the target variable regarding the abnormal state where the value of the target variable becomes abnormal, nor is it always possible to correctly determine that the value of the target variable is abnormal, based on the items that are explanatory variables selected based on the past aggregated data group 1700. The items that are explanatory variables selected based on the past aggregated data group 1700 are not necessarily indicators that easily represent the value of the target variable in the normal state and are not necessarily indicators that easily represent the value of the target variable in the abnormal state.

[0176] The information processing apparatus 100 selects an item to be the target variable based on a user's operation input. In the example of FIG. 17, the information processing apparatus 100 selects the A-to-B loss rate as the target variable. Next, the description will proceed to FIG. 18.

[0177] In FIG. 18, specifically, as shown in Table 1800 of FIG. 18, the information processing apparatus 100 stores the A-to-B loss rate selected as the target variable in association with Condition-1. Specifically, as shown in Table 1800 of FIG. 18, the information processing apparatus 100 selects the A-to-B packet number, the A-to-B byte number, the B-to-A packet number, and the B-to-A byte number as items each having an explanatory variable of one or more explanatory variables in association with Condition-1.

[0178] Returning to the description of FIG. 17, the information processing apparatus 100 extracts the values of the selected explanatory variables and the value of the target variable from each of the aggregated data included in the accumulated past aggregated data group 1700. The information processing apparatus 100 generates learning data that associates and shows the combination of the extracted values of the explanatory variables and the value of the target variable. The information processing apparatus 100 may generate the learning data, for example, after normalizing the extracted values of the explanatory variables. The learning data corresponds to, for example, each white dot on the graph 1710.

[0179] In the example of FIG. 17, for simplicity of explanation, the graph 1710 shows the relationship between the combination of the explanatory variables of the A-to-B packet number and the A-to-B byte number and the target variable of the A-to-B loss rate. For simplicity of explanation, in the graph 1710, the description of the B-to-A packet number and the B-to-A byte number is omitted.

[0180] At each point in time, the information processing apparatus 100 extracts the values of the selected explanatory variables and the value of the target variable from the current aggregated data 1701, and generates a query that associates and shows the combination of the extracted values of the explanatory variables and the value of the target variable. The query corresponds to, for example, a black dot on the graph 1710.

[0181] Each time the information processing apparatus 100 generates a query, it extracts learning data similar to the query from the generated learning data, and learns a JIT model based on the extracted learning data. In the following description, the learning data similar to the query may be referred to as "local data". For example, the information processing apparatus 100 extracts, as local data, learning data whose information vector is within a certain distance range from the information vector indicated by the query from the generated learning data, and learns a JIT model based on the extracted local data.

[0182] For example, the information processing apparatus 100 may extract, as local data, up to a predetermined number of learning data whose information vectors are within a certain distance range from the information vector indicated by the query from the generated learning data. For example, the information processing apparatus 100 may extract a predetermined number of local data in order from the learning data whose information vector is relatively close to the information vector indicated by the query from the generated learning data.

[0183] The information processing apparatus 100 calculates a predicted value and a standard deviation of the target variable by the learned JIT model, and sets a range of normal values of the target variable. The information processing apparatus 100 determines whether the value of the target variable indicated by the query is abnormal based on whether the value of the target variable indicated by the query is included in the set range of normal values. Next, the description will move to FIG. 19.

[0184] In FIG. 19, as shown in table 1900, the information processing apparatus 100 learns a JIT model at each time point, and stores the determination result of whether the value of the target variable indicated by the query at that time point is abnormal. The table 1900 is realized by a storage area such as the memory 302 or the recording medium 305 of the information processing apparatus 100 shown in FIG. 3, for example.

[0185] The table 1900 has fields for time, aggregation condition number, determination item, measured value, predicted value, upper limit of predicted normal range, lower limit of predicted normal range, and determination. The table 1900 shows the determination result as a record by setting information in each field.

[0186] In the time field, the determination time is set. In the aggregation condition number field, a number indicating the aggregation condition is set. In the determination item field, the target variable indicated by the query at the above determination time is set. In the measured value field, the measured value of the above target variable indicated by the above query is set. In the predicted value field, the predicted value of the above target variable is set.

[0187] In the predicted normal range upper limit field, the upper limit of the normal value range based on the above predicted value is set. In the predicted normal range lower limit field, the lower limit of the normal value range based on the above predicted value is set. In the determination field, the determination result is set. The information processing apparatus 100 receives an input as to whether the determination result is correct based on the user's operation input. Next, the description will move on to FIG. 20.

[0188] In FIG. 20, it is assumed that the information processing apparatus 100 has received an input that the determination result that the A-to-B loss rate is normal at the time "2019 / 02 / 10 00:01:00" is incorrect. The information processing apparatus 100 identifies, based on the JIT model used when the determination result is incorrect, among a plurality of explanatory variables, the explanatory variable that caused the determination result to be incorrect.

[0189] The information processing apparatus 100 determines that the determination result is incorrect because, although the measured value of the A-to-B loss rate tends to be a small value during normal times, the predicted value of the A-to-B loss rate is calculated to be a relatively large value. Therefore, the information processing apparatus 100 identifies, as the explanatory variable that caused the determination result to be incorrect, the explanatory variable that is the factor for the predicted value of the A-to-B loss rate to become a relatively large value.

[0190] Specifically, as shown in Table 2001, for each explanatory variable, the information processing apparatus 100 identifies the slope × measured value in the JIT model. The slope is a coefficient. Specifically, the information processing apparatus 100 identifies the AtoB byte count at which the slope × measured value is the largest as the explanatory variable that causes the determination result to be incorrect. Specifically, the information processing apparatus 100 may identify the explanatory variable whose slope × measured value is equal to or greater than the threshold value as the explanatory variable that causes the error.

[0191] For example, the information processing apparatus 100 may identify how the predicted value of the AtoB loss rate changes due to the user's operation input and cause the determination result to be easily incorrect, and identify the explanatory variable that causes the determination result to be incorrect. Further, for example, when the AtoB loss rate is normal and there is no error in the determination result, the information processing apparatus 100 may identify the explanatory variable that causes the determination result to be incorrect based on the slope × measured value of each explanatory variable. Specifically, the information processing apparatus 100 may identify, among a plurality of explanatory variables, the explanatory variable whose slope × measured value is a certain amount or more away from the normal time as the explanatory variable that causes the determination result to be incorrect.

[0192] As shown in Table 2001, when there is no error in the determination result, the slope × measured value of each explanatory variable is relatively small, and there is a tendency that it is difficult to cause an error in the determination result. Specifically, Table 2001 shows the statistical values of the slope × measured value of each explanatory variable at normal times.

[0193] The information processing apparatus 100 learns a JIT model using the remaining explanatory variables excluding the identified explanatory variable that causes the error, and determines whether the value of the target variable indicated by the query when the determination result is incorrect is abnormal. When the information processing apparatus 100 correctly determines that the value of the target variable indicated by the query when the determination result is incorrect is abnormal, the information processing apparatus 100 identifies the remaining explanatory variables excluding the identified explanatory variable that causes the error as the explanatory variables suitable for the determination based on the query.

[0194] As shown in Table 2002, since the predicted value of the information processing apparatus 100 is relatively small, it is assumed that when the determination result is incorrect, the value of the target variable indicated by the query is correctly determined to be abnormal. Next, we will move on to the explanation of FIG. 21.

[0195] In FIG. 21, as shown in Table 2100, since the information processing apparatus 100 has correctly determined that the value of the target variable is abnormal, it stores the remaining explanatory variables excluding the explanatory variable that causes the problem, in association with the target variable and Condition-1. Next, we will move on to the explanation of FIG. 22.

[0196] In FIG. 22, the information processing apparatus 100 generates a misjudgment query pattern regarding the query when the determination result is incorrect. The misjudgment query pattern indicates the properties of a query that is judged to be likely to result in an incorrect determination result by the JIT model when using a plurality of explanatory variables without excluding the explanatory variable that causes the problem. The misjudgment query pattern is a condition for narrowing down the queries for which it is preferable to perform the determination by the JIT model using the remaining explanatory variables after excluding the explanatory variable that causes the problem.

[0197] For example, as a misjudgment query pattern, the information processing apparatus 100 specifies that the minimum value of the distance between the information vector indicated by the query and the information vector indicated by the learning data in a plurality of learning data is greater than the threshold value L. For example, as shown in Table 2200, the information processing apparatus 100 stores the specified misjudgment query pattern in association with the information that enables the identification of the remaining explanatory variables shown in Table 2100.

[0198] After that, if there is a query that satisfies the misjudgment query pattern, the information processing apparatus 100 learns the JIT model using the remaining explanatory variables shown in Table 2100 and performs the determination process by the JIT model. Thereby, the information processing apparatus 100 can improve the accuracy of the determination process by the JIT model. For example, after that, based on the misjudgment query pattern, the information processing apparatus 100 can use the explanatory variables used when learning the JIT model appropriately according to the properties of the query, and can improve the accuracy of the determination process.

[0199] (Generation processing procedure) Next, with reference to FIG. 23, an example of the generation processing procedure executed by the information processing apparatus 100 will be described. The generation processing is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0200] FIG. 23 is a flowchart showing an example of the generation processing procedure. In FIG. 23, the information processing apparatus 100 determines whether the current time is the timing for creating aggregated data (step S2301). The creation timing is, for example, each of a plurality of timings at regular intervals. The creation timing may be, for example, a predetermined timing specified in advance by the user.

[0201] Here, if it is not the creation timing (step S2301: No), the information processing apparatus 100 returns to the process of step S2301. On the other hand, if it is the creation timing (step S2301: Yes), the information processing apparatus 100 proceeds to the process of step S2302.

[0202] In step S2302, the information processing apparatus 100 reads the measurement data from the accumulated measurement data group from the previous creation timing to the current time which is the current creation timing (step S2302). Next, the information processing apparatus 100 selects one aggregation condition that has not been selected yet and is the target of abnormality determination among the plurality of aggregation conditions (step S2303). Then, the information processing apparatus 100 extracts the measurement data corresponding to the selected aggregation condition from the read measurement data (step S2304).

[0203] Next, the information processing apparatus 100 aggregates the extracted measurement data to generate aggregated data (step S2305). Then, the information processing apparatus 100 determines whether all the aggregation conditions have been selected (step S2306).

[0204] Here, when all aggregation conditions are selected (step S2306: Yes), the information processing apparatus 100 returns to the process of step S2301. On the other hand, when there are still unselected aggregation conditions remaining (step S2306: No), the information processing apparatus 100 returns to the process of step S2303. Thereby, the information processing apparatus 100 can accumulate the aggregated data.

[0205] (Determination processing procedure) Next, with reference to FIG. 24, an example of a determination processing procedure executed by the information processing apparatus 100 will be described. The determination processing is realized by, for example, the CPU 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0206] FIG. 24 is a flowchart showing an example of a determination processing procedure. In FIG. 24, the information processing apparatus 100 determines whether the current time is the determination timing of an abnormality (step S2401). The determination timing is, for example, each of a plurality of timings at regular intervals. The determination timing may be, for example, a predetermined timing specified in advance by the user. The determination timing may be the same as the creation timing.

[0207] The information processing apparatus 100 reads the current aggregated data (step S2402). The current aggregated data may be, for example, the aggregated data at the creation timing closest to the determination timing. Next, the information processing apparatus 100 selects an item to be the target of abnormality determination that has not yet been selected (step S2403). The item is, for example, an item indicating a target variable. Then, the information processing apparatus 100 generates a query that summarizes the values of the selected items from the read aggregated data (step S2404).

[0208] Next, the information processing apparatus 100 generates learning data that aggregates the values of the selected items from the past aggregated data, and extracts, based on the generated query, the learning data used for anomaly determination from among the plurality of learning data (step S2405). Then, the information processing apparatus 100 selects explanatory variables to be used for determination based on the extracted learning data (step S2406).

[0209] Next, the information processing apparatus 100 determines whether the query corresponds to an incorrect determination query pattern (step S2407). Here, if it corresponds to an incorrect determination query pattern (step S2407: Yes), the information processing apparatus 100 proceeds to the process of step S2408. On the other hand, if it does not correspond to an incorrect determination query pattern (step S2407: No), the information processing apparatus 100 proceeds to the process of step S2409.

[0210] In step S2408, the information processing apparatus 100 deletes the explanatory variables corresponding to the incorrect determination query pattern among the selected explanatory variables, and re-selects the remaining explanatory variables (step S2408). Then, the information processing apparatus 100 proceeds to the process of step S2409.

[0211] In step S2409, the information processing apparatus 100 performs JIT analysis processing based on the selected explanatory variables (step S2409). The JIT analysis processing is a process of learning a JIT model using the selected explanatory variables based on the query and setting the range of normal values. Next, the information processing apparatus 100 performs anomaly determination processing to determine whether the value of the selected item is abnormal based on the result of the JIT analysis processing (step S2410).

[0212] Then, the information processing apparatus 100 determines whether all items have been selected (step S2411). Here, if there are still unselected items remaining (step S2411: No), the information processing apparatus 100 returns to the process of step S2403. On the other hand, if all items have been selected (step S2411: Yes), the information processing apparatus 100 proceeds to the process of step S2412.

[0213] In step S2412, if the information processing apparatus 100 meets the alarm condition for the item determined to be abnormal, it outputs an alarm (step S2412). The alarm condition indicates, for example, a predetermined item. Then, the information processing apparatus 100 returns to the process of step S2401. Thereby, the information processing apparatus 100 can accurately determine whether the value of the item is abnormal.

[0214] (Reflection processing procedure) Next, with reference to FIG. 25, an example of the reflection processing procedure executed by the information processing apparatus 100 will be described. The reflection processing is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

[0215] FIG. 25 is a flowchart showing an example of the reflection processing procedure. In FIG. 25, the information processing apparatus 100 receives a designation of the result of an abnormal determination that is an error based on the operation input of the user (step S2501). The information processing apparatus 100 receives, for example, a designation of the result of an abnormal determination that erroneously indicates that the value of the target variable is normal. Here, it is assumed that the result of the abnormal determination correctly indicates that the value of the target variable is abnormal.

[0216] Next, the information processing apparatus 100 extracts a query and learning data corresponding to the designated result of the abnormal determination (step S2502). Then, the information processing apparatus 100 performs JIT analysis processing using a plurality of explanatory variables (step S2503).

[0217] Thereafter, the information processing apparatus 100 searches for an explanatory variable that causes the error based on the result of the JIT analysis processing (step S2504). The information processing apparatus 100 searches, for example, for the explanatory variable having the greatest influence on the error as the explanatory variable that causes the error based on the JIT model learned by the JIT analysis processing.

[0218] Next, the information processing apparatus 100 determines whether there is an explanatory variable that causes an error (step S2505). Here, if there is no explanatory variable that causes an error (step S2505: No), the information processing apparatus 100 proceeds to the process of step S2506. On the other hand, if there is an explanatory variable that causes an error (step S2505: Yes), the information processing apparatus 100 proceeds to the process of step S2507.

[0219] In step S2506, the information processing apparatus 100 adjusts the setting of the normal range (step S2506). The information processing apparatus 100, for example, widens the normal range. Then, the information processing apparatus 100 ends the reflection process.

[0220] In step S2507, the information processing apparatus 100 excludes the explanatory variable that causes an error from the explanatory variables used in the JIT analysis, and selects the remaining explanatory variables (step S2507). Next, the information processing apparatus 100 performs a JIT analysis process using the remaining explanatory variables (step S2508). Then, the information processing apparatus 100 performs an abnormality determination process based on the result of the JIT analysis process (step S2509).

[0221] Next, the information processing apparatus 100 determines whether the result of the abnormality determination is correct (step S2510). Here, if the result of the abnormality determination is incorrect (step S2510: No), the information processing apparatus 100 proceeds to the process of step S2506. On the other hand, if the result of the abnormality determination is correct (step S2510: Yes), the information processing apparatus 100 proceeds to the process of step S2511.

[0222] In step S2511, the information processing apparatus 100 identifies an error determination query pattern (step S2511). Next, the information processing apparatus 100 associates and registers the error determination query pattern with the excluded explanatory variable (step S2512). Then, the information processing apparatus 100 ends the reflection process.

[0223] As a result, the information processing apparatus 100 can selectively use explanatory variables used when learning the JIT model for each property of the query, and can improve the accuracy of the determination process. Here, the case where the result of the abnormality determination indicates that the value of the target variable is abnormal has been described, but it is not limited to this. For example, there may be a case where the result of the abnormality determination indicates that the value of the target variable is normal.

[0224] As described above, according to the information processing apparatus 100, when the result of a predetermined determination process using the first JIT model is incorrect, based on the first JIT model, among a plurality of explanatory variables, a first explanatory variable having a relatively large influence on the error can be specified. According to the information processing apparatus 100, a second query indicating a combination of values of each explanatory variable can be acquired. According to the information processing apparatus 100, among the learning data group, learning data indicating a combination similar to the combination of values of each explanatory variable indicated by the acquired second query can be extracted. According to the information processing apparatus 100, using the extracted learning data, among a plurality of explanatory variables, a second JIT model that emphasizes other explanatory variables other than the specified first explanatory variable more than the specified first explanatory variable can be learned. According to the information processing apparatus 100, based on the predicted value of the target variable corresponding to the combination of values of each explanatory variable indicated by the acquired second query calculated using the learned second JIT model, a predetermined determination process can be performed. Thereby, the information processing apparatus 100 can improve the accuracy of the predetermined determination process.

[0225] According to the information processing apparatus 100, among the learning data group, learning data indicating a combination similar to the combination of values of other explanatory variables indicated by the acquired second query can be extracted. According to the information processing apparatus 100, based on the value of the target variable indicated by the extracted learning data and the combination of values of other explanatory variables, the second JIT model can be learned. Thereby, the information processing apparatus 100 can learn a second JIT model that enables improvement in the accuracy of a predetermined determination process.

[0226] According to the information processing apparatus 100, among a plurality of explanatory variables, a second JIT model can be learned in which each explanatory variable is weighted so as to attach more importance to other explanatory variables than to the specified first explanatory variable. Thereby, the information processing apparatus 100 can learn a second JIT model that enables improvement in the accuracy of a predetermined determination process.

[0227] According to the information processing apparatus 100, conditions based on a first query can be set. According to the information processing apparatus 100, when the acquired second query satisfies the set conditions, a predetermined determination process can be performed using the learned second JIT model. According to the information processing apparatus 100, when the acquired second query does not satisfy the set conditions, a predetermined determination process can be performed using the learned first JIT model. Thereby, the information processing apparatus 100 can appropriately distinguish between the first JIT model and the second JIT model, and can improve the accuracy of a predetermined determination process.

[0228] According to the information processing apparatus 100, a determination process can be performed to determine whether or not the actually measured value of the target variable is included in a range set based on the predicted value of the target variable. Thereby, the information processing apparatus 100 can determine whether or not the actually measured value of the target variable is abnormal.

[0229] According to the information processing apparatus 100, it is possible to extract learning data indicating combinations similar to the combinations of the values of the explanatory variables indicated by the first query from the learning data group. According to the information processing apparatus 100, the first JIT model can be learned using the extracted learning data. According to the information processing apparatus 100, the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the first query can be calculated by the learned first JIT model. According to the information processing apparatus 100, a determination process can be performed based on the calculated predicted value of the objective variable. According to the information processing apparatus 100, when the result of the performed determination process is incorrect, based on the first JIT model, among the plurality of explanatory variables, the first explanatory variable having a relatively large influence degree on the error can be specified. Thereby, the information processing apparatus 100 can perform a predetermined determination process using the first JIT model by itself.

[0230] The information processing apparatus 100 can represent the first JIT model by a mathematical formula including terms of the respective explanatory variables. According to the information processing apparatus 100, when the result of the determination process is incorrect, in the mathematical formula representing the first JIT model, among the plurality of explanatory variables, the explanatory variable that causes the predicted value of the objective variable to change most greatly in the direction in which the result of the determination process becomes incorrect is specified as the first explanatory variable. Thereby, the information processing apparatus 100 can accurately specify the explanatory variable that causes the determination result to be incorrect.

[0231] Note that the information processing method described in this embodiment can be realized by executing a pre-prepared program on a computer such as a PC or a workstation. The information processing program described in this embodiment is recorded on a computer-readable recording medium and is executed by being read from the recording medium by a computer. The recording medium is a hard disk, a flexible disk, a CD (Compact Disc)-ROM, an MO (Magneto Optical disc), a DVD (Digital Versatile Disc), or the like. Further, the information processing program described in this embodiment may be distributed via a network such as the Internet.

[0232] Regarding the above-described embodiment, the following additional remarks are disclosed.

[0233] (Supplementary Note 1) Among a group of learning data including learning data that associates a value of an objective variable with a combination of values of each of a plurality of explanatory variables, when the result of a predetermined determination process based on a predicted value of the objective variable corresponding to the combination of values of each of the explanatory variables indicated by a first query, calculated by a first JIT model learned using learning data indicating a combination similar to the combination of values of each of the explanatory variables indicated by the first query, is incorrect, based on the first JIT model, among the plurality of explanatory variables, a first explanatory variable whose degree of influence on the error is greater than that of other explanatory variables or whose degree of influence on the error is equal to or greater than a threshold value is specified. A second query indicating the combination of values of each of the explanatory variables is obtained. Among a group of learning data including learning data that associates a value of an objective variable with a combination of values of each of the explanatory variables, using learning data indicating a combination similar to the combination of values of each of the explanatory variables indicated by the obtained second query, a second JIT model that emphasizes other explanatory variables other than the specified first explanatory variable more than the specified first explanatory variable is learned among the plurality of explanatory variables. Based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the acquired second query, which is calculated using the learned second JIT model, perform the determination process. An information processing apparatus characterized by having a control unit.

[0234] (Appendix 2) The control unit Among the learning data groups including the learning data showing the association between the value of the target variable and the combination of the values of the respective explanatory variables, extract the learning data showing a combination similar to the combination of the values of the other explanatory variables indicated by the acquired second query, and based on the value of the target variable and the combination of the values of the other explanatory variables shown by the extracted learning data, learn the second JIT model. The information processing apparatus according to Appendix 1, characterized by this.

[0235] (Appendix 3) The control unit Learn the second JIT model in which weights are assigned to the respective explanatory variables so as to emphasize the other explanatory variables among the plurality of explanatory variables more than the specified first explanatory variable. The information processing apparatus according to Appendix 1 or 2, characterized by this.

[0236] (Appendix 4) The control unit Set the conditions based on the first query. When the acquired second query satisfies the set conditions, perform the determination process based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the acquired second query, which is calculated using the learned second JIT model. When the acquired second query does not satisfy the set conditions, perform the determination process based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the acquired second query, which is calculated using the learned first JIT model. The information processing apparatus according to any one of Appendices 1 to 3, characterized by this.

[0237] (Appendix 5) The first query includes the measured value of the target variable. The second query includes the measured value of the target variable. The determination process determines whether the measured value of the target variable is included in a range set based on the predicted value of the target variable. The information processing apparatus according to any one of Appendices 1 to 4, characterized by this.

[0238] (Appendix 6) The control unit Among the learning data groups including learning data showing the association between the value of the target variable and the combination of the values of the respective explanatory variables, extracts the learning data showing a combination similar to the combination of the values of the respective explanatory variables indicated by the first query. Using the extracted learning data, learns the first JIT model. Based on the learned first JIT model, calculates the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the first query. Based on the calculated predicted value of the target variable, performs the determination process. If the result of the performed determination process is incorrect, based on the first JIT model, among the plurality of explanatory variables, identifies a first explanatory variable whose degree of influence on the error is greater than that of other explanatory variables, or whose degree of influence on the error is equal to or greater than a threshold value. The information processing apparatus according to any one of Appendices 1 to 5, characterized by this.

[0239] (Appendix 7) The first JIT model is represented by a mathematical formula including terms of the respective explanatory variables. The control unit If the result of the determination process is incorrect, in the mathematical formula representing the first JIT model, among the plurality of explanatory variables, identifies as the first explanatory variable the explanatory variable that causes the predicted value of the target variable to change most significantly in the direction in which the result of the determination process becomes incorrect. The information processing apparatus according to any one of Appendices 1 to 6, characterized by this.

[0240] When the result of a predetermined determination process based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the first query, calculated by the first JIT model learned using the learning data indicating a combination similar to the combination of the values of the respective explanatory variables indicated by the first query among the learning data groups including the learning data associating the value of the target variable with the combination of the values of the respective explanatory variables, is incorrect, based on the first JIT model, among the plurality of explanatory variables, identify a first explanatory variable whose degree of influence on the error is greater than that of other explanatory variables or whose degree of influence on the error is equal to or greater than a threshold value. Obtain a second query indicating the combination of the values of the respective explanatory variables. Among the learning data groups including the learning data associating the value of the target variable with the combination of the values of the respective explanatory variables, learn a second JIT model that emphasizes other explanatory variables other than the identified first explanatory variable more than the identified first explanatory variable, using the learning data indicating a combination similar to the combination of the values of the respective explanatory variables indicated by the obtained second query. Perform the determination process based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the obtained second query, calculated using the learned second JIT model. An information processing method characterized in that a computer executes the process.

[0241] When the result of a predetermined determination process based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the first query, calculated by the first JIT model learned using the learning data indicating the combination of the values of the explanatory variables similar to the combination of the values of the explanatory variables indicated by the first query among the learning data groups including the learning data associating the value of the objective variable with the combination of the values of each of the plurality of explanatory variables is incorrect, based on the first JIT model, among the plurality of explanatory variables, identify a first explanatory variable whose degree of influence on the error is greater than that of other explanatory variables, or whose degree of influence on the error is equal to or greater than a threshold value. Obtain a second query indicating the combination of the values of the respective explanatory variables. Among the learning data groups including the learning data associating the value of the objective variable with the combination of the values of the respective explanatory variables, use the learning data indicating the combination of the values of the explanatory variables similar to the combination of the values of the explanatory variables indicated by the obtained second query to learn a second JIT model that emphasizes other explanatory variables other than the identified first explanatory variable more than the identified first explanatory variable among the plurality of explanatory variables. Perform the determination process based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the obtained second query, calculated using the learned second JIT model. An information processing program characterized by causing a computer to execute the process.

Explanation of Signs

[0242] 100 Information processing apparatus 101 First query 102 Second query 110 First JIT model 120 Second JIT model 200 Information processing system 201 Measuring device 202 Client device 210 Network 300 Bus 301 CPU 302 Memory 303 Network I / F 304 Recording Medium I / F 305 Recording Medium 400 Memory Unit 401 Acquisition Unit 402 Learning Unit 403 Judgment Unit 404 Identification Unit 405 Output Unit 501 Measurement Function 502 Aggregation Function 511 Measurement DB 512 Aggregation DB 521 Reading Function 522 Creation Function 523 Adjustment Function 524 Prediction Function 525 Judgment Function 526 Notification Function 527 Input Function 528 Analysis Function 531 Learning DB 532 Pattern DB 601, 603, 605 Explanatory Variables 602 Past Normal Data 604, 701 Data to be Judged 800 Server System 900, 1000, 1100, 1200, 1300, 1400, 1500, 1710 Graphs 901, 1401 Areas 1600, 1900 Tables 1601, 1602 Signs 1700 Aggregated Data Group 1701 Aggregated Data 1800, 2001, 2002, 2100, 2200 Tables

Claims

1. Among a group of learning data including learning data that associates the value of an objective variable with combinations of the values of a plurality of explanatory variables, when the result of a predetermined determination process based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the first query, calculated by a first JIT model learned using learning data indicating a combination similar to the combination of the values of the explanatory variables indicated by the first query, is incorrect, based on the first JIT model, among the plurality of explanatory variables, identify a first explanatory variable for which the degree of influence on the error is greater than that of other explanatory variables or for which the degree of influence on the error is equal to or greater than a threshold value. Obtain a second query indicating the combination of the values of the explanatory variables. Among a group of learning data including learning data that associates the value of an objective variable with combinations of the values of the explanatory variables, use learning data indicating a combination similar to the combination of the values of the explanatory variables indicated by the obtained second query to learn a second JIT model that emphasizes other explanatory variables other than the identified first explanatory variable more than the identified first explanatory variable among the plurality of explanatory variables. Perform the determination process based on the predicted value of the objective variable corresponding to the combination of the values of the explanatory variables indicated by the obtained second query, calculated using the learned second JIT model. An information processing apparatus characterized by having a control unit.

2. The control unit extracts learning data indicating a combination similar to the combination of the values of the other explanatory variables indicated by the obtained second query from a group of learning data including learning data that associates the value of an objective variable with combinations of the values of the explanatory variables, and learns the second JIT model based on the value of the objective variable and the combination of the values of the other explanatory variables indicated by the extracted learning data. The information processing apparatus according to claim 1, characterized by this.

3. The control unit learns the second JIT model that weights each of the explanatory variables so as to emphasize the other explanatory variables more than the identified first explanatory variable among the plurality of explanatory variables. The information processing apparatus according to claim 1 or 2, characterized by this.

4. The control unit sets conditions based on the first query. When the obtained second query satisfies the set condition, the determination process is performed based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the obtained second query, calculated using the learned second JIT model. When the obtained second query does not satisfy the set condition, the determination process is performed based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the obtained second query, calculated using the learned first JIT model. The information processing apparatus according to any one of claims 1 to 3, characterized in that.

5. Among the learning data groups including learning data showing the association between the value of the target variable and the combination of the values of the respective explanatory variables, when the result of the predetermined determination process based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the first query, calculated by the first JIT model learned using the learning data showing a combination similar to the combination of the values of the respective explanatory variables indicated by the first query, is incorrect, based on the first JIT model, among the plurality of explanatory variables, identify a first explanatory variable whose degree of influence on the error is greater than that of other explanatory variables, or whose degree of influence on the error is equal to or greater than a threshold value. Obtain a second query indicating the combination of the values of the respective explanatory variables. Among the learning data groups including learning data showing the association between the value of the target variable and the combination of the values of the respective explanatory variables, using the learning data showing a combination similar to the combination of the values of the respective explanatory variables indicated by the obtained second query, learn a second JIT model that emphasizes other explanatory variables other than the identified first explanatory variable more than the identified first explanatory variable among the plurality of explanatory variables. The determination process is performed based on the predicted value of the target variable corresponding to the combination of the values of the respective explanatory variables indicated by the obtained second query, calculated using the learned second JIT model. An information processing method, characterized in that a computer executes the process.

6. Among a group of learning data including learning data that associates the value of a target variable with combinations of the values of a plurality of explanatory variables, if the result of a predetermined determination process based on the predicted value of the target variable corresponding to the combination of the values of the explanatory variables indicated by a first query, calculated by a first JIT model learned using learning data indicating combinations of the values of the explanatory variables similar to the combination of the values of the explanatory variables indicated by the first query, is incorrect, based on the first JIT model, among the plurality of explanatory variables, identify a first explanatory variable for which the degree of influence on the error is greater than that of other explanatory variables, or for which the degree of influence on the error is equal to or greater than a threshold value. Obtain a second query indicating the combination of the values of the explanatory variables. Among a group of learning data including learning data that associates the value of a target variable with combinations of the values of the explanatory variables, use learning data indicating combinations of the values of the explanatory variables similar to the combination of the values of the explanatory variables indicated by the obtained second query to learn a second JIT model that places more importance on other explanatory variables other than the identified first explanatory variable than on the identified first explanatory variable among the plurality of explanatory variables. Perform the determination process based on the predicted value of the target variable corresponding to the combination of the values of the explanatory variables indicated by the obtained second query, calculated using the learned second JIT model. An information processing program characterized by causing a computer to execute the process.

Images