Index calculation apparatus, method and program

Abstract

To provide an index calculation apparatus, a method, and a program that assist in detecting deviations between a learning data constellation and an operational data constellation.SOLUTION: An index calculation apparatus according to an embodiment comprises acquisition means, creation means, calculation means, determination means, and output means. The acquiring means acquires a learning data constellation and an operational data constellation. The creating means creates a first histogram of the learning data constellation and a second histogram of the operational data constellation according to the first order of magnitude. The calculation means calculates an index based on the first and second histograms. The output means outputs the index when the index is determined to have converged by the determination means. If the determination means determines that the index has not converged, the first and second histograms are re-created according to the second factorial, and an index is re-calculated based on the re-created first and second histograms.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] An embodiment of the present invention relates to an index calculation device, a method, and a program. [Background technology]

[0002] In recent years, it has become known to operate systems equipped with artificial intelligence (AI).

[0003] Such a system has a machine learning model generated by learning multiple pieces of data prepared in advance based on techniques such as machine learning (deep learning), and the machine learning model can output a predicted value based on each of the multiple pieces of data obtained during operation of the system by inputting the multiple pieces of data.

[0004] However, if a phenomenon occurs in which the distribution of multiple data sets (hereinafter referred to as the operational data group) input into the machine learning model during system operation deviates from the distribution of multiple data sets (hereinafter referred to as the training data group) used to generate the machine learning model over time, for example (hereinafter referred to as data drift), the accuracy of the predicted values ​​output from the machine learning model may decrease.

[0005] It is possible to suppress such a decrease in the accuracy of predicted values ​​by having the machine learning model learn a new set of training data. However, in order to suppress such a decrease in the accuracy of predicted values, a mechanism is required that can properly detect the occurrence of the above-mentioned data drift (i.e., the deviation between the training data set and the operational data set). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2021-193497 A Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, an object of the present invention is to provide an index calculation device, method, and program used to detect a deviation between a training data group and an operational data group. [Means for solving the problem]

[0008] An index calculation device according to an embodiment includes an acquisition means, a creation means, a calculation means, a determination means, and an output means. The acquisition means acquires a learning data group including a plurality of data used to generate a machine learning model, and an operational data group including a plurality of data input to the machine learning model to obtain a predicted value. The creation means creates a first histogram representing a distribution of a plurality of data included in the acquired learning data group, and a second histogram representing a distribution of a plurality of data included in the acquired operational data group, according to a first number of classes. The calculation means calculates an index representing a similarity between the learning data group and the operational data group, based on the created first and second histograms. The determination means determines whether the calculated index has converged. When it is determined that the calculated index has converged, the output means outputs the index. When it is determined that the calculated index has not converged, the first and second histograms are created again according to a second number of classes that is an increase from the first number of classes, and the index is recalculated based on the re-created first and second histograms. [Brief description of the drawings]

[0009] [Figure 1] A diagram showing an overview of a system equipped with AI. [Diagram 2] A diagram to explain the overview of JSD. [Diagram 3] FIG. 13 is a diagram showing an example of a histogram representing the distribution of a plurality of data included in a learning data group created according to different numbers of classes. [Figure 4]11 is a diagram showing an example of a histogram representing the distribution of a plurality of data included in an operational data group created according to different numbers of classes. [Diagram 5] A diagram to explain the JSD calculated according to the number of classes. [Figure 6] FIG. 1 is a diagram for explaining an overview of an index calculation device according to a first embodiment. [Figure 7] FIG. 2 is a block diagram showing an example of a functional configuration of an index calculation device. [Figure 8] FIG. 2 is a diagram showing an example of a hardware configuration of an index calculation device. [Figure 9] 11 is a flowchart showing an example of a processing procedure of an index calculation device. [Figure 10] FIG. 4 is a diagram showing an example of a learning data group and an operation data group. [Figure 11] FIG. 13 is a diagram showing an example of a graph showing an index calculated for each class number. [Figure 12] FIG. 11 is a block diagram showing an example of the functional configuration of an index calculation device according to a second embodiment. [Figure 13] 11 is a flowchart showing an example of a processing procedure of an index calculation device. [Figure 14] FIG. 4 is a diagram showing an example of a learning data group and an operation data group. [Figure 15] FIG. 13 is a diagram showing an example of a graph showing first and second indexes calculated for each class number. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Each embodiment will be described below with reference to the drawings. (First embodiment) First, a first embodiment will be described below. Here, Fig. 1 shows an overview of a system equipped with AI (artificial intelligence).

[0011] The system shown in FIG. 1 has a machine learning model that is generated by learning a group of training data based on a technique such as machine learning, and the machine learning model outputs a predicted value based on each of a group of operational data obtained during operation of the system when the operational data is input.

[0012] The learning data group and the operational data group described above include, for example, a plurality of data (time-series numerical data) continuously measured by a sensor.

[0013] In such a system, for example, a group of operational data equivalent to the multiple data included in the training data group is input into the machine learning model, making it possible to output highly accurate predicted values.

[0014] On the other hand, if a phenomenon occurs in which the distribution of multiple data included in the operational data group shifts from the distribution of multiple data included in the training data group over time (data drift), the accuracy of the predicted values ​​output from the machine learning model (the predictive accuracy of the machine learning model) may decrease.

[0015] In other words, data drift over time (fluctuations in the distribution of data) can cause good predictive accuracy in machine learning models to become poor.

[0016] Therefore, it is conceivable to calculate an index representing the similarity between the above-mentioned training data group and the operational data group, and detect the occurrence of data drift (deviation between the training data group and the operational data group) based on the index. When the occurrence of data drift is detected in this way, the prediction accuracy of the machine learning model can be improved by retraining the machine learning model with a new training data group whose tendency is similar to that of the operational data group.

[0017] In the present embodiment, an index is assumed that numerically expresses the similarity between a training data group and an operational data group, and the index may be, for example, JSD (Jensen-Shannon Divergence). JSD is an index calculated based on the difference (distance) between histograms that represent the distribution of two sets of data.

[0018] Here, an overview of the JSD will be described with reference to Fig. 2. Fig. 2 shows the relationship between two histograms and the JSD calculated based on the two histograms.

[0019] As shown in Figure 2, the JSD varies depending on the area where two histograms overlap, for example. In other words, the JSD is an index that takes a small value when the distributions of multiple data are similar and takes a large value when the distributions of the multiple data are different.

[0020] By using such a JSD (value) as an index representing the similarity between the training data group and the operational data group, it is believed that the occurrence of data drift can be detected, for example, when the index exceeds a threshold value.

[0021] Here, the above-mentioned JSD is an index calculated based on two histograms, and the histograms are created by setting multiple classes, each of which is assigned a numerical range, on the horizontal axis and plotting the number of data (frequency) that falls into each class on the vertical axis. In other words, different histograms are created by changing the numerical range (i.e., the number of classes) set on the horizontal axis.

[0022] Note that Fig. 3 shows an example of a histogram representing the distribution of a plurality of data included in a learning data group created according to different numbers of classes. Specifically, the left side of Fig. 3 shows a histogram created by dividing the range from the minimum value to the maximum value of a plurality of data included in a learning data group into five ranges of values ​​(i.e., setting the number of classes = 5). The right side of Fig. 3 shows a histogram created by dividing the range from the minimum value to the maximum value of a plurality of data included in a learning data group into 50 ranges of values ​​(i.e., setting the number of classes = 50). As shown in Fig. 3, the distribution of a plurality of data represented by a histogram differs depending on the number of classes that is set.

[0023] Also, Fig. 4 shows an example of a histogram representing the distribution of multiple data included in the operational data group, which is created according to different numbers of classes. Specifically, the left side of Fig. 4 shows a histogram created by dividing the range from the minimum value to the maximum value of multiple data included in the operational data group into five ranges of values ​​(i.e., setting the number of classes = 5). The right side of Fig. 4 shows a histogram created by dividing the range from the minimum value to the maximum value of multiple data included in the operational data group into 50 ranges of values ​​(i.e., setting the number of classes = 50).

[0024] As shown in Figures 3 and 4, different histograms are created by setting different numbers of bins, and for example, as shown in Figure 5, the JSD calculated based on two histograms created according to the number of bins = 5 shown on the left side of Figures 3 and 4 is 0.23, and the JSD calculated based on two histograms created according to the number of bins = 50 shown on the right side of Figures 3 and 4 is 0.33. In other words, an index representing the similarity between a training data group and an operational data group, such as the above-mentioned JSD, depends on the number of bins set to create a histogram (i.e., the value differs depending on the number of bins).

[0025] In general, indices such as the above-mentioned JSD are often calculated based on a histogram created according to a preset number of bins, but there is no number of bins suitable for all data. In other words, a histogram created according to a preset number of bins may not properly represent the distribution of multiple data contained in the training data group and the operational data group. Therefore, an index calculated based on such a histogram may not properly reflect the similarity between the training data group and the operational data group, and may not be suitable for detecting the occurrence of the above-mentioned data drift.

[0026] Therefore, in this embodiment, an index calculation device is provided that optimizes a calculation method of an index that indicates the similarity between a training data group and an operational data group in accordance with the characteristics of the training data group and the operational data group.

[0027] Hereinafter, an overview of the index calculation device according to this embodiment will be described with reference to Fig. 6. As shown in Fig. 6, in this embodiment, an index (e.g., JSD) based on a histogram representing the distribution of a plurality of data included in a learning data group and a histogram representing the distribution of a plurality of data included in an operational data group is calculated while changing the number of classes set when creating the histogram. Fig. 6 shows histograms created while increasing the number of classes stepwise from 5 to 50, and indices calculated based on the histograms.

[0028] In this embodiment, attention is focused on the fact that the calculated index increases as the number of classes increases as described above, and the convergence value of the index is used to detect the occurrence of data drift.

[0029] 7 is a block diagram showing an example of the functional configuration of an index calculation device according to this embodiment. As shown in FIG 7, the index calculation device 10 includes an acquisition unit 11, an outlier processing unit 12, an index calculation unit 13, an index evaluation unit 14, a class number setting unit 15, and a display processing unit 16.

[0030] The acquiring unit 11 acquires the above-mentioned learning data group and operation data group. In addition, the acquiring unit 11 acquires a condition (hereinafter, referred to as a convergence condition) used to determine whether or not an index has converged, as described later.

[0031] The learning data group and the operation data group acquired by the acquisition unit 11 include a plurality of data, and the outlier processing unit 12 processes outliers from the plurality of data. Note that the processing executed by the outlier processing unit 12 includes a process called clipping in which a value (data) that does not fall within a predetermined range of numerical values ​​is regarded as an outlier, and the outlier is corrected to be data that falls within the range of numerical values.

[0032] The index calculation unit 13 creates a histogram (hereinafter referred to as a histogram of the learning data group) representing the distribution of multiple data included in the learning data group whose outliers have been processed by the outlier processing unit 12. The index calculation unit 13 also creates a histogram (hereinafter referred to as a histogram of the operation data group) representing the distribution of multiple data included in the operation data group whose outliers have been processed by the outlier processing unit 12. The histograms of the learning data group and the operation data group are created according to, for example, the same number of classes.

[0033] Furthermore, the index calculation unit 13 calculates an index (for example, JSD) representing the similarity between the learning data group and the operation data group, based on the created histograms of the learning data group and the operation data group.

[0034] The index evaluation unit 14 evaluates the index calculated by the index calculation unit 13, and determines whether or not the index has converged based on the convergence condition acquired by the acquisition unit 11 described above.

[0035] If it is determined that the index has not converged, the bin number setting unit 15 sets a new bin number that is greater than the bin number when the histogram was created by the index calculation unit 13. When a new bin number is set by the bin number setting unit 15 in this manner, the index calculation unit 13 creates histograms of the training data group and the operation data group again in accordance with the new bin number, and recalculates the index based on the created histograms of the training data group and the operation data group.

[0036] When it is determined that the index has converged, the index evaluation unit 14 outputs the index (convergence value) to the display processing unit 16. The display processing unit 16 displays the index (convergence value) output from the index evaluation unit 14.

[0037] Fig. 8 shows an example of a hardware configuration of the index calculation device 10 shown in Fig. 7. As shown in Fig. 8, the index calculation device 10 includes a CPU 101, a non-volatile memory 102, a main memory 103, a communication device 104, an input device 105, a display device 106, and the like.

[0038] The CPU 101 is a processor that controls the operation of each component in the index calculation device 10. The CPU 101 executes various programs loaded from the non-volatile memory 102, which is a storage device, to the main memory 103. These programs include an operating system and a program for calculating the above-mentioned indexes (hereinafter, referred to as an index calculation program), etc.

[0039] The communication device 104 is configured to execute communication with, for example, an external device different from the index calculation device 10. The input device 105 is configured to input various information and includes, for example, a mouse and a keyboard. The display device 106 is configured to display various information and includes, for example, a display.

[0040] 8, only the non-volatile memory 102 and the main memory 103 are shown, but the index calculation device 10 may further include other storage devices such as a hard disk drive (HDD) and a solid state drive (SSD). The index calculation device 10 may also include other devices not shown in FIG.

[0041] In this embodiment, some or all of the units 11 to 16 shown in Fig. 7 are realized by causing the CPU 101 shown in Fig. 8 to execute the index calculation program described above, that is, by software. This index calculation program may be downloaded to the index calculation device 10 via a network, or may be stored in a storage medium and distributed. In addition, some or all of the units 11 to 16 may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a configuration that combines software and hardware.

[0042] Next, an example of a processing procedure of the index calculation device 10 according to this embodiment will be described with reference to the flowchart in Fig. 9. Note that the processing shown in Fig. 9 is periodically executed to detect (monitor) the occurrence of data drift during operation of the above-mentioned system, for example.

[0043] First, the acquisition unit 11 acquires (inputs) a learning data group, an operation data group, and a convergence condition (step S1).

[0044] Here, assume that a system is operated that has a machine learning model generated by learning a learning data group that includes a plurality of data (time-series numerical data collected from the sensor) continuously measured by a sensor attached to a facility (instrument) operating in a plant. Note that in such a system, the machine learning model is constructed so that, by inputting each of a plurality of data included in an operation data group similar to the learning data group, the machine learning model outputs a predicted value such as an abnormality level of the facility to which the sensor is attached.

[0045] In this case, in step S1, a learning data group and an operation data group are acquired, for example, as shown in Fig. 10. As shown in Fig. 10, the learning data group and the operation data group each include, for example, numerical data measured by a sensor for each date and time. Note that the learning data group and the operation data group may be acquired (received), for example, from a server device or the like provided outside the index calculation device 10, or may be acquired from a storage unit (not shown) such as a database provided in the index calculation device 10.

[0046] In step S1, a convergence condition such as "the change in JSD with an increase in the number of classes is less than 0.001" is acquired. That is, the convergence condition acquired in step S1 includes a threshold value (here, 0.001). Note that the convergence condition (the threshold value included in the convergence condition) may be stored in advance in the index calculation device 10, or may be specified (input) by the user operating the input device 105, for example.

[0047] Next, the outlier processing unit 12 processes outliers among the multiple data included in the learning data group and the operational data group acquired in step S1 (step S2). In this case, the outlier processing unit 12 identifies data that does not fall within a predetermined numerical range among the multiple data (numerical data) included in the learning data group and the operational data group as an outlier, and corrects the data so that it falls within the numerical range. Specifically, the outlier is corrected to be, for example, a value closer to the maximum value or the minimum value of the predetermined numerical range. Here, the process of correcting the outlier has been described as being executed, but in step S2, a process of excluding the data identified as the outlier from the learning data group and the operational data group may be executed.

[0048] In the following description, the training data group and the operational data group from which outliers have been processed in step S2 will be simply referred to as the training data group and the operational data group.

[0049] Here, for example, a class number corresponding to an initial value is set (stored) in advance in the index calculation unit 13, and the index calculation unit 13 creates histograms of the learning data group and the operation data group according to the class number (initial value) (step S3). In step S3, as described above, each of a plurality of numerical ranges determined by the class number is set on the horizontal axis, and the number of data corresponding to each of the plurality of numerical ranges is represented on the vertical axis, thereby creating histograms of the learning data group and the operation data group.

[0050] Next, the index calculation unit 13 calculates an index representing the similarity (difference) between the learning data group and the operation data group based on the histograms of the learning data group and the operation data group created in step S3 (step S4).

[0051] In addition, the following formulas (1) to (3) are used for calculating the index in step S4.

number

[0052] When the process of step S4 is executed, the bin number setting unit 15 sets a new bin number by adding a predetermined value (increment) to the bin number (initial value) when the histogram was created in the above-mentioned step S3 (step S5). Note that the value added to the bin number (initial value) in step S5 increases according to the number of times the process of step S5 is executed (the number of trials of index calculation).

[0053] In this case, the index calculation unit 13 creates histograms of the learning data group and the operational data group according to the number of classes set in step S5 (step S6).

[0054] Next, the index calculation unit 13 calculates an index based on the histograms of the learning data group and the operation data group created in step S6 (step S7).

[0055] The processes in steps S6 and S7 are similar to those in steps S3 and S4 described above, except that the number of classes is different.

[0056] When the process of step S7 is executed, the index evaluation unit 14 judges whether the index satisfies the convergence condition acquired in step S1 based on the index calculated in step S4 and the index calculated in step S7 (step S8). In this case, the index evaluation unit 14 calculates the difference between the index calculated in step S4 and the index calculated in step S7 (i.e., the amount of change in the index), and judges that the index satisfies the convergence condition if the difference is less than the threshold included in the convergence condition. On the other hand, the index evaluation unit 14 judges that the index does not satisfy the convergence condition if the calculated difference is not less than the threshold included in the convergence condition.

[0057] If it is determined that the index does not satisfy the convergence condition (NO in step S8), the index is deemed not to have converged, and the process returns to step S5 and is repeated. In step S5 which is repeated in this manner, a number of classes is set that is at least greater than the number of classes set in the previous step S5.

[0058] On the other hand, if it is determined that the index satisfies the convergence condition (YES in step S8), the index evaluation unit 14 assumes that the index has converged, and outputs the index calculated in step S7 as a converged value (step S9).

[0059] When the process of step S9 is executed, for example, the display processing unit 16 can display on the display device 106 the index (convergence value) output from the index evaluation unit 14 in step S9.

[0060] Note that, although it has been described here that the index evaluation unit 14 outputs an index that is a convergence value in step S9, the index evaluation unit 14 may be configured to output all the indexes calculated in steps S4 and S7 (i.e., the indexes calculated for each class number). In this case, the display processing unit 16 can create a graph showing the indexes calculated for each class number (indexes corresponding to each class number) output from the index evaluation unit 14 as shown in Fig. 11, and display the created graph. Here, it has been described that a graph showing the indexes corresponding to each class number is displayed (i.e., the indexes corresponding to each class number are displayed in a graph format), but the indexes corresponding to each class number may be displayed in other formats such as a table.

[0061] Furthermore, the display processing unit 16 may be configured to include a detection unit (not shown) that detects the occurrence of data drift (deviation between the learning data group and the operational data group) based on, for example, the index (convergence value) output from the index evaluation unit 14, and to display the detection result by the detection unit. Specifically, if the index output from the index evaluation unit 14 is, for example, the JSD, the display processing unit 16 can detect the occurrence of data drift when the JSD exceeds a threshold value (i.e., the JSD is a value high enough to consider that data drift has occurred).

[0062] In addition, the index output from the index evaluation unit 14 in the above-mentioned step S9 may be transmitted to, for example, an external device of the index calculation device 10 and displayed on the external device, or may be used in processing, etc., executed on the external device.

[0063] As described above, in this embodiment, a learning data group including a plurality of data used to generate a machine learning model and an operation data group including a plurality of data input to the machine learning model to obtain a predicted value are acquired, a histogram (first histogram) of the acquired learning data group and a histogram (second histogram) of the acquired operation data group are created according to the first class number, and an index representing the similarity between the learning data group and the operation data group is calculated based on the created histograms of the learning data group and the operation data group. In this embodiment, if it is determined that the index calculated in this way has converged, the index (convergence value) is output. Also, in this embodiment, if it is determined that the index calculated in this way has not converged, histograms of the learning data group and the operation data group are created again according to the second class number that is increased from the first class number, and the index is calculated again based on the histograms of the learning data group and the operation data group that have been created again. In other words, the index calculation device 10 according to this embodiment operates to calculate an index while increasing the class number until convergence.

[0064] In this embodiment, it is possible to detect a deviation (occurrence of data drift) between the learning data group and the operational data group by using an appropriate index calculated by the above-mentioned configuration.

[0065] Here, as described above, when the index in this embodiment is the JSD, the JSD increases with an increase in the number of classes as described in Fig. 6. In this embodiment, a convergence value of the JSD that increases according to the number of classes is output, and the convergence value corresponds to the maximum value of the JSD that indicates the deviation between the training data group and the operational data group, so that it is possible to reduce the possibility of missing detection of the occurrence of data drift.

[0066] 6, for example, assume that the occurrence of data drift is detected when the JSD calculated according to a predetermined number of classes is 0.30 or more. In this case, for example, the JSD calculated according to a predetermined number of classes=11 is 0.29, and the occurrence of data drift is not detected based on this JSD. In contrast, this embodiment is configured to calculate the JSD while increasing the number of classes, thereby outputting 0.33, which is the convergence value of the JSD calculated according to the number of classes=50, and therefore the occurrence of data drift can be detected based on the JSD.

[0067] In other words, in a configuration in which an index is simply calculated according to a predetermined number of classes as described above, depending on the training data group and the operational data group, the number of classes may not be able to adequately express the distribution of multiple data, and the user cannot determine whether the index calculated in this manner correctly reflects the deviation between the training data group and the operational data group (i.e., the validity of the index).

[0068] However, in this embodiment, the convergence value of the index calculated while increasing the number of classes (the index calculated according to the number of classes that is assumed to adequately represent the distribution of multiple data) is output, so the user does not need to judge the validity of the index.

[0069] In this embodiment, the index output as described above can be displayed on the display device 106. With this configuration, the user can detect (determine) the occurrence of data drift by checking the index displayed on the display device 106.

[0070] In addition, in this embodiment, for example, the index calculated for each class number may be displayed in a graph format on the display device 106. With such a configuration, it is possible to visualize the progress of the index calculated for each class number, which may enable the user to more appropriately detect the occurrence of data drift.

[0071] Furthermore, this embodiment may be configured such that the occurrence of data drift (deviation between the learning data group and the operational data group) is automatically detected based on the above-mentioned index, and the detection result is displayed on the display device 106. With such a configuration, even if the user does not have the knowledge to determine the occurrence of data drift based on the index, the user can grasp (recognize) the occurrence of data drift.

[0072] In addition, in this embodiment, a convergence condition including a threshold value designated by a user is acquired, and when the calculated change amount (difference) of an index is less than the threshold value included in the convergence condition, it is determined that the index has converged. With this configuration, in this embodiment, it is possible to appropriately determine the convergence of an index and output the convergence value of the index.

[0073] Furthermore, in this embodiment, the index is calculated based on the histograms of the learning data group and the operation data group, but outliers among the multiple data included in the learning data group and the multiple data included in the operation data group may affect the histograms. In other words, for example, if data corresponding to an outlier is mixed into the learning data group and the operation data group, the histograms of such learning data group and operation data group may not be appropriate for calculating the index. For this reason, in this embodiment, outliers among the multiple data included in the learning data group and the multiple data included in the operation data group are processed, and histograms of the learning data group and the operation data group are created based on the processing results of the outliers. In this embodiment, such a configuration can mitigate the influence of outliers, and create appropriate histograms and calculate highly accurate indexes.

[0074] In this embodiment, the outliers are described as being processed, but in a situation where it is expected that there is a low possibility that outliers will be mixed into the training data group and the operational data group, the processing for the outliers (i.e., the processing of step S2 shown in FIG. 9) may be omitted.

[0075] In addition, in this embodiment, the JSD has been mainly described as an index representing the similarity between the training data group and the operational data group, but the index may be calculated based on the histogram of the training data group (distribution of a plurality of data included in the training data group) and the histogram of the operational data group (distribution of a plurality of data included in the operational data group), and may be a value equivalent to the degree of deviation or similarity between the distributions. Specifically, the index may be KLD (Kullback-Leibler Divergence) or the like.

[0076] Second embodiment Next, a second embodiment will be described. In the first embodiment described above, it is assumed that the number of types of data included in the learning data group and the operational data group is one, but this embodiment differs from the first embodiment in that the number of types of data included in the learning data group and the operational data group is two.

[0077] Fig. 12 is a block diagram showing an example of the functional configuration of an index calculation device according to this embodiment. In Fig. 12, the same reference numerals are given to the same parts as those in Fig. 7 described above, and detailed explanations thereof will be omitted. Here, the parts different from those in Fig. 7 described above will be mainly described.

[0078] 12, the index calculation device 10 includes an index totalizing unit 17. Like each of the units 11 to 16, the index totalizing unit 17 may be realized by software, may be realized by hardware, or may be realized by a combination of software and hardware.

[0079] In this embodiment, the learning data group and the operation data group acquired by the acquisition unit 11 include, for example, a plurality of first type data and a plurality of second type data. Note that the first type data is, for example, data measured by a first sensor, and the second type data is, for example, data measured by a second sensor different from the first sensor.

[0080] In addition, in this embodiment, the histogram of the learning data group created by the index calculation unit 13 includes a histogram representing the distribution of a plurality of first type data included in the learning data group acquired by the acquisition unit 11 (the learning data group whose outliers have been processed by the outlier processing unit 12) and a histogram representing the distribution of a plurality of second type data included in the learning data group.

[0081] Similarly, in this embodiment, the histogram of the operation data group created by the index calculation unit 13 includes a histogram representing the distribution of multiple first type data included in the operation data group acquired by the acquisition unit 11 (the operation data group whose outliers have been processed by the outlier processing unit 12) and a histogram representing the distribution of multiple second type data included in the operation data group.

[0082] Furthermore, in this embodiment, the indices calculated by the index calculation unit 13 include indices for each type of data calculated based on the histogram of the learning data group and the histogram of the operational data group.

[0083] The index totalizer 17 totals the indexes for each data type calculated by the index calculator 13 .

[0084] In this embodiment, the index evaluation unit 14 applies a convergence condition to the result of the index totalization unit 17 totaling the indexes for each data type, and determines whether the index has converged.

[0085] In this embodiment, when it is determined that the indexes have converged, the index evaluation unit 14 outputs the index (convergence value) for each type of data to the display processing unit 16.

[0086] Here, the functional configuration of the index calculation device 10 according to this embodiment has been described. However, since the hardware configuration of the index calculation device 10 in this embodiment is similar to that of the first embodiment described above, the hardware configuration will be described appropriately with reference to FIG. 8.

[0087] Next, an example of the processing procedure of the index calculation device 10 according to this embodiment will be described with reference to the flowchart in Fig. 13. Note that the processing shown in Fig. 13 is periodically executed to detect (monitor) the occurrence of data drift during operation of the above-mentioned system, for example, similar to the processing shown in Fig. 9 described above.

[0088] First, the acquiring unit 11 acquires a learning data group, an operation data group, and a convergence condition (step S11).

[0089] Here, for example, a machine learning model is generated by learning a group of training data including a plurality of first type data continuously measured by a first sensor attached to equipment (device) operating in a plant and a plurality of second type data continuously measured by a second sensor, and the machine learning model is constructed to output a predicted value of the degree of abnormality, etc. of the equipment by inputting each of the plurality of first type data and each of the plurality of second type data (i.e., corresponding sets of first and second type data) included in an operation data group similar to the training data group.

[0090] In this case, in step S11, a learning data group and an operation data group are acquired, for example, as shown in Fig. 14. Note that in Fig. 14, the numerical data associated with the first sensor corresponds to the first type of data measured by the first sensor at each date and time, and the numerical data associated with the second sensor corresponds to the second type of data measured by the second sensor at each date and time.

[0091] Furthermore, in step S11, similarly to the first embodiment described above, a convergence condition including a threshold value is acquired.

[0092] After the process of step S11 is executed, the process of step S12, which corresponds to the process of step S2 shown in FIG. 9, is executed.

[0093] In the following description, the training data group and the operational data group from which outliers have been processed in step S12 will be simply referred to as the training data group and the operational data group.

[0094] Next, the index calculation unit 13 creates histograms of the learning data group and the operation data group for each type of data according to the number of classes corresponding to the initial value described in the first embodiment (step S13). In step S13, a histogram showing the distribution of a plurality of first type data included in the learning data group (hereinafter referred to as the first histogram of the learning data group), a histogram showing the distribution of a plurality of second type data included in the learning data group (hereinafter referred to as the second histogram of the learning data group), a histogram showing the distribution of a plurality of first type data included in the operation data group (hereinafter referred to as the first histogram of the operation data group), and a histogram showing the distribution of a plurality of second type data included in the operation data group (hereinafter referred to as the second histogram of the operation data group) are created.

[0095] When the process of step S13 is executed, the index calculation unit 13 calculates an index for each data type based on the first and second histograms of the learning data group and the first and second histograms of the operation data group created in step S13 (step S14). In step S14, an index (hereinafter referred to as a first index) is calculated based on the first histogram of the learning data group and the first histogram of the operation data group, and an index (hereinafter referred to as a second index) is calculated based on the second histogram of the learning data group and the second histogram of the operation data group.

[0096] The processing in steps S13 and S14 described above corresponds to the processing in steps S3 and S4 shown in FIG. 9 except that the creation of the histogram and the calculation of the index are performed for each type of data, and therefore a detailed description thereof will be omitted here.

[0097] Next, the index totalizer 17 totals the first and second indexes calculated in step S14 (step S15).

[0098] After the process of step S15 is executed, the process of step S16, which corresponds to the process of step S5 shown in FIG. 9, is executed.

[0099] In this case, the index calculation unit 13 creates histograms of the learning data group and the operation data group for each data type according to the number of classes set in step S16 (step S17).

[0100] Next, the index calculation unit 13 calculates an index for each type of data based on the histograms of the learning data group and the operation data group created in step S17 (step S18).

[0101] The processes in steps S17 and S18 are the same as those in steps S13 and S14 described above, except that the number of classes is different. That is, in step S17, the first and second histograms of the learning data group and the first and second histograms of the operational data group are created. In addition, in step S18, the first and second indexes are calculated.

[0102] Next, the index summing unit 17 sums up the first and second indexes calculated in step S18 (step S19).

[0103] When the process of step S19 is executed, the index evaluation unit 14 judges whether or not the indexes (first and second indexes) satisfy the convergence condition acquired in step S11 based on the result of summing the first and second indexes in step S15 (hereinafter referred to as the first index total value) and the result of summing the first and second indexes in step S19 (hereinafter referred to as the second index total value) (step S20). In this case, the index evaluation unit 14 calculates the difference between the first index total value and the second index total value (i.e., the amount of change in the index total value), and judges that the index satisfies the convergence condition if the difference is less than a threshold value included in the convergence condition. On the other hand, the index evaluation unit 14 judges that the index does not satisfy the convergence condition if the calculated difference is not less than the threshold value included in the convergence condition.

[0104] In addition, when creating a histogram (frequency distribution) from a data group including different types of data as described above, it is considered preferable to uniformly align the number of bins between the types. For this reason, in this embodiment, a configuration is adopted in which a convergence condition is applied to the total value of the first and second indices calculated based on histograms created for each type of data according to the same number of bins.

[0105] If it is determined that the index does not satisfy the convergence condition (NO in step S20), the index is deemed not to have converged, and the process returns to step S16 and is repeated. In step S16 which is repeated in this manner, a number of classes is set that is at least greater than the number of classes set in the previous step S16.

[0106] On the other hand, if it is determined that the index satisfies the convergence condition (YES in step S20), the index evaluation unit 14 assumes that the index has converged, and outputs an index for each data type (step S21). Specifically, in step S21, the first and second indexes calculated in the above-mentioned step S18 are output as converged values.

[0107] When the process of step S21 is executed, for example, the display processing unit 16 can display on the display device 106 the first and second indicators output in step S21.

[0108] Furthermore, when the index evaluation unit 14 is configured to output all indexes calculated in steps S14 and S18 (i.e., the first and second indexes calculated for each class number), the display processing unit 16 may create a graph showing the first index calculated for each class number and a graph showing the second index calculated for each class (i.e., a graph for each data type) as shown in Fig. 15, and display the created graphs. Note that, although two graphs created for the first and second indexes are shown separately in Fig. 15, the display processing unit 16 may display the two graphs by overlapping them.

[0109] Furthermore, the display processing unit 16 may be configured to include a detection unit (not shown) that detects the occurrence of data drift (deviation between the learning data group and the operational data group) based on the first and second indexes (convergence values) output from the index evaluation unit 14, for example, and to display the detection result by the detection unit. Specifically, assuming that the first and second indexes output from the index evaluation unit 14 are JSD, the display processing unit 16 can detect the occurrence of data drift when at least one of the first and second indexes exceeds a threshold. The occurrence of data drift may be detected when a representative value such as the sum or average value of the first and second indexes exceeds a threshold.

[0110] In addition, the first and second indices output from the index evaluation unit 14 in the above-mentioned step S21 may be transmitted, for example, to an external device of the index calculation device 10 and displayed on the external device, or may be used in processing, etc., executed on the external device.

[0111] 13, the convergence condition is applied to the sum of the first and second indices, but the convergence condition may be applied to each of the first and second indices. In this case, the process of step S21 may be executed when it is determined that one of the first and second indices satisfies the convergence condition, or the process of step S21 may be executed when it is determined that both the first and second indices satisfy the convergence condition. That is, in this embodiment, it is sufficient that the first and second indices are output when it is determined that the indices including the first and second indices have converged.

[0112] As described above, in this embodiment, each of the learning data group and the operation data group includes a plurality of first type data and a plurality of second type data. In this case, the histogram of the learning data group includes the first histogram of the learning data group (the third histogram representing the distribution of the plurality of first type data included in the learning data group) and the second histogram of the learning data group (the fourth histogram representing the distribution of the plurality of second type data included in the learning data group). In addition, the histogram of the operation data group includes the first histogram of the operation data group (the fifth histogram representing the distribution of the plurality of first type data included in the operation data group) and the second histogram of the operation data group (the sixth histogram representing the distribution of the plurality of second type data included in the operation data group). Furthermore, the index in this embodiment includes a first index calculated based on the first histogram of the learning data group and the first histogram of the operational data group described above, and a second index calculated based on the second histogram of the learning data group and the second histogram of the operational data group, and when it is determined that the index including the first and second indexes has converged, the first and second indexes are output.

[0113] In this embodiment, with the above-described configuration, for example, the index calculation device 10 according to the first embodiment can be expanded so as to be applicable to data sets (a learning data group and an operational data group) including data of different types.

[0114] In the present embodiment, the learning data group and the operational data group are described as including two types of data, but the types of data may be three or more.

[0115] According to at least one of the embodiments described above, it is possible to provide an index calculation device, method, and program that assist in detecting a deviation between a training data group and an operational data group.

[0116] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0117] 10...index calculation device, 11...acquisition unit, 12...outlier processing unit, 13...index calculation unit, 14...index evaluation unit, 15...class number setting unit, 16...display processing unit, 17...index totalization unit, 101...CPU, 102...non-volatile memory, 103...main memory, 104...communication device, 105...input device, 106...display unit.

Claims

1. An acquisition means for acquiring a learning data group including a plurality of data used to generate a machine learning model, and an operational data group including a plurality of data to be input to the machine learning model in order to obtain a predicted value; a creation means for creating a first histogram representing a distribution of a plurality of data included in the acquired learning data group according to a first class number, and a second histogram representing a distribution of a plurality of data included in the acquired operational data group; a calculation means for calculating an index representing a similarity between the learning data group and the operational data group based on the created first and second histograms; A determination means for determining whether the calculated index has converged; an output means for outputting the calculated index when it is determined that the index has converged; Equipped with If it is determined that the calculated index has not converged, the first and second histograms are created again according to a second number of classes obtained by increasing the first number of classes, and the index is recalculated based on the re-created first and second histograms. Indicator calculation device.

2. 2. The index calculation device according to claim 1, wherein the output index is displayed on a display device.

3. The output means outputs the index calculated for each of the number of classes, The index calculated for each class number is displayed on a display device in the form of a graph. The index calculation device according to claim 1.

4. The method further includes detecting a deviation between the learning data group and the operational data group based on the output index, The detection result is displayed on a display device. The index calculation device according to claim 1.

5. The acquisition means acquires a convergence condition including a threshold value specified by a user, The determining means determines that the index has converged when an amount of change in the calculated index is less than a threshold included in the acquired convergence condition. The index calculation device according to claim 1.

6. The method further includes a processing means for processing outliers among a plurality of data included in the learning data group and a plurality of data included in the operational data group, The creation means creates the first and second histograms based on the result of the outlier processing. The index calculation device according to claim 1.

7. Each of the learning data group and the operational data group includes a plurality of first type data and a plurality of second type data, the first histogram includes a third histogram representing a distribution of a plurality of first type data included in the learning data group and a fourth histogram representing a distribution of a plurality of second type data included in the learning data group; the second histogram includes a fifth histogram representing a distribution of a plurality of first type data included in the operation data group and a sixth histogram representing a distribution of a plurality of second type data included in the operation data group; the index includes a first index calculated based on the third and fifth histograms and a second index calculated based on the fourth and sixth histograms; The output means outputs the first and second indices when it is determined that the indices including the first and second indices have converged. The index calculation device according to claim 1.

8. The index calculation device according to any one of claims 1 to 7, wherein the index includes JSD (Jensen-Shannon Divergence).

9. Obtaining a training data set including a plurality of data used to generate a machine learning model, and an operational data set including a plurality of data to be input to the machine learning model to obtain a predicted value; creating a first histogram representing a distribution of a plurality of data included in the acquired learning data group according to a first class number, and a second histogram representing a distribution of a plurality of data included in the acquired operational data group; calculating an index representing a similarity between the training data group and the operational data group based on the created first and second histograms; determining whether the calculated indices have converged; outputting the calculated index if it is determined that the index has converged; Equipped with If it is determined that the calculated index has not converged, the first and second histograms are created again according to a second number of classes obtained by increasing the first number of classes, and the index is recalculated based on the re-created first and second histograms. method.

10. On the computer, Obtaining a training data set including a plurality of data used to generate a machine learning model, and an operational data set including a plurality of data to be input to the machine learning model to obtain a predicted value; creating a first histogram representing a distribution of a plurality of data included in the acquired learning data group according to a first class number, and a second histogram representing a distribution of a plurality of data included in the acquired operational data group; calculating an index representing a similarity between the training data group and the operational data group based on the created first and second histograms; determining whether the calculated indices have converged; outputting the calculated index if it is determined that the index has converged; Run the command, If it is determined that the calculated index has not converged, the first and second histograms are created again according to a second number of classes obtained by increasing the first number of classes, and the index is recalculated based on the re-created first and second histograms. program.

Images