Management device and management method

Abstract

To provide appropriate information about retraining of a machine learning model.SOLUTION: A management device 100 is provided with: a first comparison unit 231 which calculates a first data drift index on the basis of a comparison result related to frequency distributions of one or a plurality of feature quantities included in input data; a second comparison unit 232 which calculates a second data drift index related to the plurality of frequency distributions on the basis of a comparison result different in comparison condition from comparison in the first comparison unit 231; and a determination unit 233 which outputs a retraining necessity level indicating a level of necessity of retraining of a machine learning model on the basis of the first data drift index, the second data drift index, and a determination method table defining a determination criterion.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a management device and a management method. [Background technology]

[0002] In recent years, various systems that perform inference using machine learning models have been proposed. However, the accuracy of inference using machine learning models often decreases over time. When the accuracy of a machine learning model decreases, a process is required to relearn and update the model. The accuracy of a machine learning model can be calculated if there is actual data that represents the correct answer to the inference, but it may take time to obtain the actual data. On the other hand, the main cause of the decrease in accuracy of machine learning models over time is data drift, in which the properties of input data change from the time of learning due to changes in the environment, etc. Therefore, a technology that focuses on this point and relearns the model when data drift is detected is known.

[0003] For example, the abstract of Patent Document 1 below states, "[Problem] To prevent unnecessary re-learning and reduce the processing costs of re-learning a learning model. [Solution] A management computer 1 that manages a system that performs inference using a learning model has a processor that performs processing in cooperation with a memory, and the processor executes a generation process that generates an accuracy-improved prediction model for predicting the accuracy of a re-learning model when re-learning is performed using re-learning data including newly collected data collected from the system after the system has started operating, based on a correlation between the features of the learning data used to learn the learning model and the accuracy of the learning model; a prediction process that predicts the accuracy of the re-learning model from the accuracy-improved prediction model and the features of the re-learning data; and a determination process that determines whether or not re-learning is necessary, based on the predicted accuracy of the re-learning model."

[0004] Furthermore, the abstract of Patent Document 2 below states that "there is provided an information processing device comprising: a data acquisition unit that acquires, from a target system, learning data used for learning a model for anomaly detection in the target system and testing data used for testing the model; and a determination unit that determines whether or not re-learning of the model is necessary based on the degree of deviation between a data distribution of the learning data and a data distribution of the testing data." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2021-184139 A [Patent Document 2] International Publication No. 2019 / 180778 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the above-mentioned technology, there is a demand for providing more appropriate information to a user when the user decides whether or not to retrain a machine learning model. The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a management device and a management method that can provide users with appropriate information regarding re-learning of a machine learning model. [Means for solving the problem]

[0007] In order to solve the above problem, the management device of the present invention is characterized by comprising: a state estimation unit that predicts a state of a target system using input data acquired from the target system and a machine learning model; a first comparison unit that calculates a first data drift index based on a comparison result regarding a frequency distribution of one or more feature amounts included in the input data; a second comparison unit that calculates a second data drift index based on a comparison result regarding a plurality of frequency distributions under comparison conditions different from those of the comparison in the first comparison unit; and a judgment unit that outputs a re-learning necessity level that indicates the degree of necessity for re-learning the machine learning model based on the first data drift index, the second data drift index, and a judgment method table that defines judgment criteria. Effect of the Invention

[0008] According to the present invention, it is possible to provide a user with appropriate information regarding re-learning of a machine learning model. [Brief description of the drawings]

[0009] [Figure 1] 1 is a hardware block diagram of a device management system according to a first embodiment. [Diagram 2] FIG. 4 is a diagram showing an example of input data in the first embodiment. [Diagram 3] FIG. 2 is a functional block diagram of a management device according to the first embodiment. [Figure 4] FIG. 11 is a schematic diagram showing a method for calculating a first data drift index. [Diagram 5] FIG. 13 is a diagram illustrating an example of a determination method table. [Figure 6] 4 is a flowchart of a first comparison routine. [Figure 7] 13 is a flowchart of a second comparison routine. [Figure 8] 13 is a flowchart of a re-learning required level calculation routine. [Figure 9] FIG. 13 is a diagram showing an example of an analysis result display window. [Figure 10] FIG. 11 is a diagram showing an example of clustered data applied to the second embodiment. [Figure 11] FIG. 11 is a functional block diagram of a management device according to a second embodiment. [Figure 12] FIG. 11 is a diagram illustrating an operation of a cluster creating unit. [Figure 13] FIG. 2 is a diagram illustrating an operation of a clustering execution unit. [Figure 14] FIG. 13 is a schematic diagram showing a method for calculating a first cluster drift index. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Prerequisites for the embodiment] By applying the contents of the above-mentioned Patent Document 1, it is considered possible to predict the accuracy of re-learning when re-learning is performed using re-learning data including newly collected data, based on the correlation between the feature amount of the learning data and the accuracy of the machine learning model. Then, it is considered possible to perform re-learning if the accuracy is predicted to exceed a predetermined value. However, in many cases, the decision of whether or not to actually perform re-learning and updating of the model is ultimately made by a human being. According to the contents of Patent Document 1, there is a problem in that no particular consideration is given to a function to assist a human judgement.

[0011] In addition, by applying the contents of Patent Document 2, it is possible to detect changes in data trends and instruct the execution of model re-learning. However, the timing for model re-learning is not necessarily immediately after the detection of data drift, and it is necessary to take into account the amount of accumulated data and whether the drift has ended, which creates a problem in that it may not be possible to present an appropriate update timing. The embodiment described below has been devised in consideration of the above problems, and indicates the necessary level of retraining a machine learning model based on the occurrence of data drift and the duration of data drift, thereby enabling even those who are not machine learning experts to easily determine whether or not a model needs to be retrained.

[0012] [First embodiment] FIG. 1 is a hardware block diagram of a device management system 10 according to the first embodiment. The equipment management system 10 includes a management device 100, a target system 110, and a network 120. The target system 110 is, but is not limited to, a gas engine system, for example. The target system 110 supplies input data DI representing various physical quantities and the like in the target system 110 to the management device 100 via the network 120. The management device 100 is a device that estimates the state of the target system 110 by using the input data DI and a machine learning model.

[0013] The management device 100 includes a storage device 101, a memory 102, a processor 103, a network I / F 104, an input device 105, and an output device 106. It should be noted that "I / F" stands for interface.

[0014] The storage device 101 stores control programs for executing various processes of this embodiment, data necessary for executing the processes of this embodiment, etc. The memory 102 is used as a working area for executing the programs. The arithmetic device 103 is, for example, a CPU (Central Processing Unit), which reads the control programs and data for the processes of this embodiment from the storage device 101 into the memory 102 and executes the control programs. The network I / F 104 controls data communication between the management device 100 and the network 120.

[0015] The management device 100 receives data from outside the management device 100 through the network I / F 104. The input device 105 is, for example, a mouse, a keyboard, or a touch panel. A user of the management device 100 can input settings for the management device 100 and input instructions to execute processing via the input device 105. The output device 106 is, for example, a display, and displays the processing results of this embodiment.

[0016] FIG. 2 is a diagram showing an example of input data DI in the first embodiment. The input data DI has multiple records (rows), and each record includes date / time information DT and multiple (n) feature quantities DS1 to DSn. Hereinafter, n is referred to as the number of feature quantities. Note that the number of feature quantities n does not necessarily have to be multiple, and may be at least one. For example, when the target system 110 is a gas engine system, the feature quantities DS1 to DSn are quantities such as outside air temperature, cooling water pressure, and cooling temperature. Furthermore, the date / time information DT indicates the sampling timing at which the feature quantities DS1 to DSn were acquired. In the illustrated example, the sampling timing is every 30 seconds.

[0017] The input data DI is classified into multiple data on a record-by-record basis according to the date and time information DT. That is, among the input data DI, data in the time range used for learning to construct a machine learning model is called learning data DL. Also, data newer than the learning data DL is called operational data DP. For example, among the input data DI, data whose date and time information DT belongs to 2020 can be considered as learning data DL, and data belonging to 2021 or later can be considered as operational data DP. Also, input data DI for a shorter period, such as "December 2021", can be considered as operational data DP.

[0018] Fig. 3 is a functional block diagram of the management device 100 in the first embodiment. In Fig. 3, the inside of the management device 100 shows functions realized by a control program and the like. That is, the management device 100 includes an input / output unit 210, a storage unit 220, and a control unit 230. The input / output unit 210 includes a communication unit 211, an input unit 212, and an output unit 213 (display device). The storage unit 220 includes a learning data storage unit 221, an operation data storage unit 222, a comparison result storage unit 223, a determination method table storage unit 224, and a determination result storage unit 225.

[0019] The control unit 230 also includes a first comparison unit 231, a second comparison unit 232, a determination unit 233, a display control unit 234, and a state estimation unit 250 (state estimation process). The state estimation unit 250 stores a machine learning model (not shown) that has been trained using the training data DL. The state estimation unit 250 then uses the machine learning model and the operation data DP to estimate the state of the target system 110 that corresponds to the operation data DP. The machine learning model is, for example, a neural network, but other machine learning techniques may also be applied.

[0020] In the input / output unit 210, the communication unit 211 corresponds to the network I / F 104 (see FIG. 1) in terms of hardware configuration. The communication unit 211 receives input data DI (see FIG. 2), i.e., learning data DL and operation data DP, from the target system 110. The input unit 212 corresponds to the input device 105 in terms of hardware configuration. The input unit 212 receives input of settings for the management device 100 from a user, and stores the contents in the storage unit 220. The input unit 212 also receives input of an instruction to execute processing from a user, and notifies the control unit 230 of the contents. The output unit 213 corresponds to the output device 106 in terms of hardware configuration. The output unit 213 displays contents instructed by the display control unit 234, such as a re-learning necessity level LVR (details will be described later) indicating the degree of necessity for re-learning the machine learning model.

[0021] The learning data storage unit 221 in the storage unit 220 stores learning data DL used for learning the machine learning model among the input data DI supplied from the target system 110. In addition, the operational data storage unit 222 stores operational data DP to be input to the trained machine learning model among the input data DI supplied from the target system 110. Note that, as described above, the learning data DL is, for example, input data DI belonging to the year 2020, and the operational data DP is, for example, input data DI belonging to 2021 or later.

[0022] In addition, the first comparison unit 231 in the control unit 230 acquires the learning data DL and the operational data DP from the learning data memory unit 221 and the operational data memory unit 222, respectively, and calculates the frequency distribution FDL1 to FDLn (not shown) of the features DS1 to DSn in the learning data DL and the frequency distribution FDP1 to FDPn (not shown) of the features DS1 to DSn in the operational data DP.

[0023] Then, the first comparing unit 231 calculates the first data drift index CD1Mk, i.e., the first data drift indexes CD1M1 to CD1Mn, by comparing the frequency distribution FDLk (where 1≦k≦n) with the frequency distribution FDPk, and stores the first data drift index CD1Mk in the comparison result storage unit 223. The first data drift index CD1Mk is a data drift index indicating the distance or similarity between the frequency distributions FDLk and FDPk. More specifically, the first data drift index CD1Mk may be KLD (Kullback-Leibler Divergence) or JSD (Jensen-Shannon Divergence). In addition, when an index indicating the similarity of distributions is applied as the first data drift index CD1Mk, for example, HI (Histogram Intersection) may be applied. Note that a data drift index other than the above may be applied.

[0024] FIG. 4 is a schematic diagram showing a method of calculating the first data drift index CD1Mk. Graph G12 shown in Fig. 4 is a frequency distribution FDLk of feature DSk (where 1≦k≦n) in the learning data DL. The horizontal axis of graph G12 indicates the range of a quantity (e.g., temperature, pressure) related to feature DSk temperature. The "frequency" on the vertical axis is the number of sampling timings in which feature DSk is included in the "range" of the horizontal axis.

[0025] Moreover, graph G14 is a frequency distribution FDPk of the feature amount DSk in the operational data DP. The vertical and horizontal axes of graph G14 have the same meanings as those of graph G12. It can be considered that the first comparing unit 231 calculates the first data drift index CD1Mk by comparing graph G12 with graph G14.

[0026] Returning to FIG. 3, the second comparison unit 232 in the control unit 230 acquires two types of partial data with different periods from the operation data DP stored in the operation data storage unit 222. The acquired partial data are called partial operation data DPx and DPy. Here, the partial operation data DPx (first partial operation data) may be, for example, the operation data DP from 60 days before the present to 30 days before, and the partial operation data DPy (second partial operation data) may be the operation data DP from 30 days before the present to the present. Then, the second comparison unit 232 calculates the frequency distribution FDPx1 to FDPxn (not shown) of the feature amounts DS1 to DSn in the partial operation data DPx and the frequency distribution FDPy1 to FDPyn (not shown) of the feature amounts DS1 to DSn in the partial operation data DPy.

[0027] Then, the second comparing section 232 compares the frequency distribution FDPxk (where 1≦k≦n) with the frequency distribution FDPyk to calculate a second data drift index CD2Mk, and stores this in the comparison result storage section 223. Note that the method of calculating the second data drift index CD2Mk is similar to the method of calculating the first data drift index CD1Mk.

[0028] The comparison result storage unit 223 in the storage unit 220 stores the first comparison data CD1k (criterion) and the second comparison data CD2k (criterion) (where 1≦k≦n), both of which are not shown. Here, the first comparison data CD1k includes the above-mentioned first data drift index CD1Mk as well as various data related to the first data drift index CD1Mk (details will be described later). Similarly, the second comparison data CD2k includes the above-mentioned second data drift index CD2Mk as well as various data related to the second data drift index CD2Mk (details will be described later).

[0029] The determination section 233 in the control section 230 obtains the first comparison data CD1k and the second comparison data CD2k from the comparison result storage section 223, and obtains from the determination method table storage section 224 a determination method table JT (see FIG. 5) which will be described later.

[0030] Then, the determination unit 233 determines the degree of necessity for re-learning the machine learning model corresponding to the feature amount DSk, that is, the relearning requirement level LVRk, according to the determination method described in the determination method table JT, and stores the determination result in the determination result storage unit 225. The display control unit 234 displays the relearning requirement level LVRk, etc. via the output unit 213. Details of the determination content executed by the determination unit 233 and the display content by the display control unit 234 will be described later.

[0031] FIG. 5 is a diagram showing an example of the determination method table JT stored in the determination method table storage unit 224. As shown in FIG. The judgment method table JT is a table that associates first comparison data CD1k, second comparison data CD2k, re-learning required level LVRk, and judgment reason information DRNk.

[0032] The first comparison data CD1k stored in the comparison result storage unit 223 includes, in addition to the above-mentioned first data drift index CD1Mk, a first data drift period CD1Tk (not shown), a first occurrence determination value CD1MJk (see FIG. 5), a first period determination value CD1TJk (see FIG. 5), and a first progress determination value CD1PJk (see FIG. 5). Moreover, the second comparison data CD2k includes, in addition to the above-mentioned second data drift index CD2Mk, a second occurrence determination value CD2MJk. These data will be described below.

[0033] First occurrence determination value CD1MJk: This is binary data indicating whether or not data drift has occurred in the operational data DP. That is, the operational data DP is divided into predetermined determination periods (e.g., one day), and the first data drift index CD1Mk is obtained for each determination period. Then, the first occurrence determination value CD1MJk becomes "1" (data drift occurs) when the first data drift index CD1Mk becomes equal to or greater than a predetermined threshold value Th1Mk in at least one determination period, and becomes "0" (data drift does not occur) otherwise.

[0034] First data drift period CD1Tk: This represents the period during which the first data drift indicator CD1Mk is equal to or greater than the threshold value Th1Mk. First period judgment value CD1TJk: This is binary data that becomes "1" (above threshold) when this first data drift period CD1Tk is above a predetermined threshold Th1Tk, and becomes "0" (below threshold) otherwise. This threshold Th1Tk should be set to a period sufficient for appropriate re-learning of the machine learning model.

[0035] First progress determination value CD1PJk: This is an index indicating whether or not data drift has occurred in a predetermined "past reference period." The "past reference period" is the period between the period in which the learning data DL was collected and the period in which the operational data DP was collected. In addition, it is preferable that the past reference period is close to the period in which the operational data DP was collected. For example, assume that the learning data DL is data belonging to 2020, and the operational data DP is data belonging to November 1 to December 31, 2021.

[0036] In this case, the past reference period may be, for example, from October 25, 2021 to October 31, 2021, which is 7 days to 1 day before the collection period of the operational data DP. The method of determining whether data drift has occurred may be the same as that of obtaining the first occurrence determination value CD1MJk described above. That is, the past reference period is divided into predetermined determination periods (for example, one day), and the drift index (an amount similar to the first data drift index CD1Mk described above) is obtained for each determination period. Then, the first progress determination value CD1PJk becomes "1" (data drift has occurred in the recent past) when the drift index becomes equal to or greater than a predetermined threshold value Th1Mk in at least one determination period, and becomes "0" (no data drift has occurred in the recent past) otherwise.

[0037] Second occurrence judgment value CD2MJk: This is binary data indicating whether or not a data drift has occurred between the partial operation data DPx and the partial operation data DPy. The method for determining whether or not a data drift has occurred may be the same as the method for determining the first occurrence judgment value CD1MJk described above.

[0038] Below, each of the cases C1 to C8 in FIG. 5 will be explained. Case C1: In this case, the first occurrence judgment value CD1MJk is "1" (data drift exists), the first period judgment value CD1TJk is "1" (above threshold), the second occurrence judgment value CD2MJk is "1" (data drift exists), and the re-learning required level LVRk is set to the highest "3." The judgment reason information DRNk, which is the reason for this, is "R1: The drift state has continued for a long period of time, and sufficient data for re-learning has been accumulated."

[0039] Case C2: In this case, the first occurrence judgment value CD1MJk is "1" (data drift exists), the first period judgment value CD1TJk is "0" (less than threshold), the second occurrence judgment value CD2MJk is "1" (data drift exists), and the re-learning required level LVRk is set to a slightly high value of "2." The judgment reason information DRNk, which is the reason for this, is "R2: Drift state, but sufficient data for re-learning has not yet been accumulated."

[0040] Case C3: In this case, the first occurrence determination value CD1MJk is "1" (data drift exists), the first period determination value CD1TJk is "1" (above threshold), the second occurrence determination value CD2MJk is "0" (no data drift), and the re-learning required level LVRk is set to the highest level "3." The determination reason information DRNk, which is the reason for this, is "R3: Stable in an unlearned operating state, and sufficient data for re-learning has been accumulated."

[0041] Case C4: In this case, the first occurrence judgment value CD1MJk is "1" (data drift exists), the first period judgment value CD1TJk is "0" (less than threshold), the second occurrence judgment value CD2MJk is "0" (no data drift), and the re-learning required level LVRk is set to a slightly high value of "2." The judgment reason information DRNk, which is the reason for this, is "R4: Stabilized in an unlearned operating state, but sufficient data for re-learning has not yet been accumulated."

[0042] Case C5: In this case, the first occurrence determination value CD1MJk is "0" (no data drift), the first progress determination value CD1PJk is "1" (data drift in the recent past), the second occurrence determination value CD2MJk is "1" (data drift), and the re-learning required level LVRk is set to the highest level "3." The determination reason information DRNk, which is the reason for this, is "R5: There was a long period of unlearned operating state (currently returning to a learned operating state)."

[0043] Case C6: In this case, the first occurrence judgment value CD1MJk is "0" (no data drift), the first progress judgment value CD1PJk is "0" (no data drift in the recent past), the second occurrence judgment value CD2MJk is "1" (data drift exists), and the re-learning required level LVRk is set to a slightly low value of "1." The judgment reason information DRNk, which is the reason for this, is "R6: Because the nature of the input data is changing, there is a possibility that data drift with the learning data will occur in the future."

[0044] Case C7: In this case, the first occurrence determination value CD1MJk is "0" (no data drift), the first progress determination value CD1PJk is "1" (there is data drift in the recent past), the second occurrence determination value CD2MJk is "0" (no data drift), and the re-learning required level LVRk is set to the highest "3." The determination reason information DRNk, which is the reason for this, is "R7: There was a long period of unlearned operating state (it has now returned to a learned operating state and stabilized)."

[0045] Case C8: In this case, the first occurrence determination value CD1MJk is "0" (no data drift), the first progress determination value CD1PJk is "0" (no data drift in the recent past), the second occurrence determination value CD2MJk is "0" (no data drift), and the re-learning required level LVRk is set to the lowest "0." And the determination reason information DRNk, which is the reason for this, is "R8: The nature of the input data has not changed."

[0046] Here, a specific example of determining the re-learning required level LVRk for a certain feature amount DSk (where 1≦k≦n) will be described. First, suppose that the first occurrence determination value CD1MJk (see FIG. 4) for the feature amount DSk is "0.6." Also, suppose that the threshold value Th1Mk for determining the presence or absence of data drift is "0.5." Then, the first occurrence determination value CD1MJk becomes "1" (data drift exists).

[0047] In addition, assume that the second occurrence determination value CD2MJk for the feature amount DSk is "0.1" and the threshold value Th2Mk for determining the presence or absence of data drift is also "0.5." In this case, the second occurrence determination value CD2MJk becomes "0" (no data drift).

[0048] Also, assume that the first data drift period CD1Tk is "33 days" and the corresponding threshold value Th1Tk is "30 days." Also assume that the past reference period for acquiring the first progress determination value CD1PJk is "7 days to 1 day before the collection period of the operational data DP." In this case, since the first data drift period CD1Tk is longer than the threshold value Th1Tk, the first period determination value CD1TJk is "1" (threshold or more). Since the above example corresponds to case C3 in the determination method table JT (see FIG. 5), the re-learning required level LVRk is "3."

[0049] As described above, in the example shown in Fig. 5, the range of the relearning requirement level LVRk is "0 to 3", and the higher the value, the greater the need for relearning. However, the relearning requirement level LVRk is not limited to this, and may be set by the user of the management device 100. In addition, in the determination method table JT, the determination reason information DRNk is not necessarily required, but it is preferable to include the determination reason information DRNk as shown in Fig. 5. In addition, the contents of the determination method table JT may be set by the user of the management device 100.

[0050] FIG. 6 is a flowchart of the first comparison routine R231 (first comparison process) executed by the first comparing unit 231. 6, when the process proceeds to step S101, the first comparison unit 231 acquires the learning data DL and the operational data DP from the learning data storage unit 221 and the operational data storage unit 222, respectively. For example, as in the above-mentioned example, the learning data DL may be data belonging to the year 2020, and the operational data DP may be data belonging to November 1 to December 31, 2021. However, the period of the learning data DL and the operational data DP may be set by the user of the management device 100 to, for example, 30 days, 90 days, etc.

[0051] Next, when the process proceeds to step S102, the first comparing unit 231 compares the frequency distribution of all the feature quantities DSk (where 1≦k≦n) in the learning data DL with the frequency distribution of the feature quantities DSk in the operational data DP to calculate the first comparison data CD1k. Then, the first comparing unit 231 stores the calculated first comparison data CD1k in the comparison result storage unit 223.

[0052] FIG. 7 is a flowchart of the second comparison routine R232 (second comparison process) executed by the second comparing section 232. 7, when the process proceeds to step S201, the second comparison unit 232 acquires the partial operation data DPx, DPy from the operation data storage unit 222. Next, when the process proceeds to step S202, the second comparison unit 232 compares the frequency distribution of all feature values ​​DSk (where 1≦k≦n) in the partial operation data DPx with the frequency distribution of feature values ​​DSk in the partial operation data DPy to calculate second comparison data CD2k. Then, the second comparison unit 232 stores the calculated second comparison data CD2k in the comparison result storage unit 223.

[0053] FIG. 8 is a flowchart of the relearning requirement level calculation routine R233 (determination process) executed by the determination unit 233. The determination unit 233 executes this routine n times for each of the feature quantities DSk (where 1≦k≦n). 8, when the process proceeds to step S301, the determination unit 233 acquires the determination method table JT (see FIG. 5) from the determination method table storage unit 224. Next, when the process proceeds to step S302, the determination unit 233 acquires the first comparison data CD1k and the second comparison data CD2k from the comparison result storage unit 223.

[0054] Next, when the process proceeds to step S303, the determination unit 233 obtains a first occurrence determination value CD1MJk and determines whether or not this is "1" (data drift occurs). More specifically, the determination unit 233 compares the first data drift indicator CD1Mk included in the first comparison data CD1k with the above-mentioned threshold value Th1Mk. Then, the determination unit 233 sets the first occurrence determination value CD1MJk to "1" (data drift occurs) if "CD1Mk≧Th1Mk" is established, and sets it to "0" (no data drift) otherwise.

[0055] If the result of the determination in step S303 is "Yes" (data drift exists), the process proceeds to step S304. Here, the determination unit 233 obtains a first period determination value CD1TJk, and determines whether or not this is "1" (above threshold value). More specifically, the determination unit 233 calculates a first data drift period CD1Tk, which is a period during which the first data drift index CD1Mk is above threshold value Th1Tk, and compares this with the above-mentioned threshold value Th1Tk. Then, the determination unit 233 sets the first period determination value CD1TJk to "1" (above threshold value) if "CD1Tk≧Th1Tk" is established, and sets it to "0" (below threshold value) otherwise.

[0056] If the result of the determination in step S304 is "Yes" (greater than or equal to the threshold), the process proceeds to step S307, where the determination unit 233 sets the relearning requirement level LVRk to "3". On the other hand, if the result of the determination in step S304 is "No" (less than the threshold), the process proceeds to step S308, where the determination unit 233 sets the relearning requirement level LVRk to "2". Then, in both steps S307 and S308, the determination unit 233 stores the relearning requirement level LVRk in the determination result storage unit 225. At this time, the corresponding determination reason information DRNk (see FIG. 5) may also be stored in the determination result storage unit 225. With the above, the process of this routine ends.

[0057] If the result of step S303 is “No” (no data drift), the process proceeds to step S305. Here, the determination unit 233 calculates the first progress determination value CD1PJk and determines whether this is “1” (there is data drift in the recent past).

[0058] If the result of step S305 is "Yes" (there is data drift in the recent past), the process proceeds to step S307, and as described above, the determination unit 233 sets the re-learning requirement level LVRk to "3." On the other hand, if the result of step S305 is "No" (there is no data drift in the recent past), the process proceeds to step S306. Here, the determination unit 233 obtains the second occurrence determination value CD2MJk and determines whether or not this is "1" (there is data drift).

[0059] More specifically, the determination unit 233 compares the second data drift indicator CD2Mk included in the second comparison data CD2k with a predetermined threshold value Th2Mk (not shown). Then, the determination unit 233 sets the second occurrence determination value CD2MJk to “1” (data drift exists) if “CD2Mk≧Th2M” is satisfied, and to “0” (no data drift exists) otherwise.

[0060] If the determination in step S306 is "Yes" (data drift exists), the process proceeds to step S309, where the determination unit 233 sets the relearning requirement level LVRk to "1". On the other hand, if the determination in step S306 is "No" (data drift does not exist), the process proceeds to step S310, where the determination unit 233 sets the relearning requirement level LVRk to "0". Then, in both steps S309 and S310, the determination unit 233 stores the relearning requirement level LVRk in the determination result storage unit 225. At this time, the corresponding determination reason information DRNk (see FIG. 5) may also be stored in the determination result storage unit 225. With the above, the process of this routine ends.

[0061] FIG. 9 is a diagram showing an example of the analysis result display window 300. As shown in FIG. The analysis result display window 300 is displayed on the output unit 213 by the display control unit 234 (see FIG. 3). The analysis result display window 300 includes a relearning requirement level display unit 310 and a graph display unit 320.

[0062] The relearning requirement level display unit 310 has n display rows, each corresponding to a feature DS1 to DSn, and each display row includes a name column 312 showing the name of the feature DSk (where 1≦k≦n), a relearning requirement level column 314 showing the relearning requirement level LVRk, and a determination reason column 316 showing the determination reason information DRNk.

[0063] When the user clicks on any display line in the relearning requirement level display section 310, the display control section 234 (see FIG. 3) displays information on the corresponding feature amount DSk in the graph display section 320. In the illustrated example, it is assumed that the display line of the feature amount DS2 is clicked in the relearning requirement level display section 310, and as a result, information on the feature amount DS2 is displayed in the graph display section 320.

[0064] The graph display unit 320 includes a first data drift index display field 322, a second data drift index display field 324, and a relearning requirement level display field 326. These display, as graphs, the time series changes of the first data drift index CD1Mk, the second data drift index CD2Mk, and the relearning requirement level LVRk, respectively, for a specified feature amount DSk (feature amount DS2 in the illustrated example).

[0065] Moreover, the horizontal lines 322a, 324a indicate the levels of the threshold value Th1Mk and the threshold value Th2Mk. In addition, in the relearning requirement level display field 326, the transition of the determination reason information DRNk may be displayed together with the transition of the relearning requirement level LVRk.

[0066] [Second embodiment] Next, a second embodiment will be described. In the following description, parts corresponding to those in the first embodiment described above are given the same reference numerals, and the description thereof may be omitted.

[0067] FIG. 10 is a diagram showing an example of clustered data DC applied to the second embodiment. The clustered data DC includes the same components as the input data DI (see FIG. 2), and in addition, each record includes cluster information DBC. Suppose that m (where 2≦m≦n) representative (e.g. important) features are selected from n features DS1 to DSn included in each record. These m features correspond to one point in m-dimensional space. Among the multiple points corresponding to each record in m-dimensional space, those that are close to each other or have high similarity can be considered to belong to one "cluster."

[0068] Each cluster can be assigned unique identification information. The cluster affiliation information DBC included in each record of the clustered data DC is identification information of the cluster to which the record belongs. In the following explanation, "clustering" refers to a method of grouping data based on the similarity between the data. Also, "cluster" refers to a collection of similar data created by clustering.

[0069] Fig. 11 is a functional block diagram of the management device 100 in the second embodiment. That is, Fig. 11 shows the inside of the management device 100 showing functions realized by a control program and the like. The hardware configuration of the second embodiment is similar to that of the first embodiment (see Fig. 1). And, like the first embodiment (see Fig. 3), the management device 100 includes an input / output unit 210, a storage unit 220, and a control unit 230.

[0070] However, in addition to the components of the first embodiment, the storage unit 220 includes a cluster storage unit 226. Also, the control unit 230 includes a cluster creation unit 235 and a clustering execution unit 236 in addition to the components of the first embodiment.

[0071] The cluster creation unit 235 acquires the learning data DL from the learning data storage unit 221, and clusters the learning data DL to calculate the characteristics of each cluster. The calculated characteristics of each cluster are called cluster characteristic data CP (not shown). Then, the cluster creation unit 235 adds belonging cluster information DBC (see FIG. 10) to the learning data DL and stores it in the learning data storage unit 221.

[0072] As a result, the data structure of the learning data DL becomes the clustered data DC (see FIG. 10). Furthermore, the cluster creating unit 235 stores in the cluster storage unit 226 the cluster characteristic data CP of each cluster in the learning data DL.

[0073] The clustering execution unit 236 acquires the operational data DP from the operational data storage unit 222 and the cluster characteristic data CP from the cluster storage unit 226. Then, the clustering execution unit 236 calculates to which cluster each record of the operational data DP belongs, adds belonging cluster information DBC to each record, and stores it in the operational data storage unit 222. As a result, the data structure of the operational data DP also becomes clustered data DC (see FIG. 10).

[0074] FIG. 12 is a diagram showing a schematic diagram of the operation of the cluster creating section 235. As shown in FIG. 12 shows an example in which the learning data DL is clustered by two feature quantities DS1 and DS2. Therefore, the horizontal and vertical axes are the feature quantities DS1 and DS2, and each record of the learning data DL is displayed by a white circle on the graph G22. Clusters CS1, CS2, and CS3 are formed by mutually approximate white circles. The central points of the clusters CS1, CS2, and CS3 are called cluster centers CC1, CC2, and CC3. The cluster centers CC1, CC2, and CC3 are included in the cluster feature data CP described above.

[0075] FIG. 13 is a diagram showing a schematic diagram of the operation of the clustering execution unit 236. As shown in FIG. The meanings of the vertical axis, horizontal axis, and cluster centers CC1, CC2, and CC3 in Fig. 13 are the same as those shown in Fig. 12. Furthermore, the coordinates of circles PP1, PP2, and PP3 in Fig. 13 correspond to each record of the operation data DP. The clustering execution unit 236 calculates the cluster centers CC1, CC2, and CC3 that are closest to each of the circles PP1, PP2, and PP3.

[0076] Then, the clustering execution unit 236 adds belonging cluster information DBC (see FIG. 10) to the operational data DP, assuming that each corresponding record in the operational data DP belongs to the clusters CS1, CS2, CS3 (see FIG. 12) corresponding to the closest cluster centers CC1, CC2, CC3.

[0077] As described above, the first comparison unit 231 in the first embodiment calculates the frequency distribution FDL1 to FDLn (first frequency distribution, not shown) of the features DS1 to DSn in the learning data DL and the frequency distribution FDP1 to FDPn (second frequency distribution, not shown) of the features DS1 to DSn in the operational data DP, and compares the two to calculate the first data drift indexes CD1M1 to CD1Mn and store them in the comparison result memory unit 223.

[0078] The first comparison unit 231 of the second embodiment has, in addition to the functions of the first embodiment, a function of calculating a frequency distribution CFDL (see FIG. 14) of clusters CS1, CS2, CS3, ... in the learning data DL, a function of calculating a frequency distribution CFDP (see FIG. 14) of clusters CS1, CS2, CS3, ... in the operational data DP, and a function of calculating a first cluster drift index CD1MC (see FIG. 14) by comparing the frequency distributions CFDL, CFDP and storing it in the comparison result memory unit 223.

[0079] FIG. 14 is a schematic diagram showing a method of calculating the first cluster drift index CD1MC. Graph G22 shown in Fig. 14 is a frequency distribution CFDL of clusters CS1, CS2, CS3, ... in the learning data DL. The horizontal axis of graph G22 indicates clusters CS1, CS2, CS3, .... The "frequency" on the vertical axis is the number of sampling timings at which the cluster appears in the learning data DL.

[0080] Moreover, graph G24 is a frequency distribution CFDP of clusters CS1, CS2, CS3, ... in the operational data DP. The vertical and horizontal axes of graph G24 have the same meanings as those of graph G22. It can be considered that the first comparing unit 231 calculates the first cluster drift index CD1MC by comparing graph G22 with graph G24. The calculation method of the first cluster drift index CD1MC is the same as the calculation method of the first data drift index CD1Mk described above.

[0081] As described above, the second comparison unit 232 in the first embodiment calculates the frequency distribution FDPx1 to FDPxn (third frequency distribution, not shown) of the features DS1 to DSn in the partial operation data DPx and the frequency distribution FDPy1 to FDPyn (fourth frequency distribution, not shown) of the features DS1 to DSn in the partial operation data DPy, and compares the two to calculate the second data drift indexes CD2M1 to CD2Mn and store them in the comparison result memory unit 223.

[0082] The second comparison unit 232 of the second embodiment, in addition to the functions of the first embodiment, has the following functions: a function to calculate a frequency distribution CFDPx (not shown) of clusters CS1, CS2, CS3, ... in the partial operational data DPx, a function to calculate a frequency distribution CFDPy (not shown) of clusters CS1, CS2, CS3, ... in the partial operational data DPy, and a function to calculate a second cluster drift index CD2MC (not shown) by comparing the frequency distributions CFDPx, CFDPy and store it in the comparison result memory unit 223.

[0083] As described above, the judgment unit 233 in the first embodiment calculates the re-learning required level LVRk based on the first comparison data CD1k (where 1≦k≦n), the second comparison data CD2k, and the judgment method table JT (see FIG. 5), and stores the calculated re-learning required level LVRk in the judgment result memory unit 225.

[0084] In addition to the functions of the first embodiment, the judgment unit 233 of the second embodiment has a function of calculating a relearning requirement level LVRC (not shown) based on cluster comparison, based on the first cluster drift index CD1MC, the second cluster drift index CD2MC, and the judgment method table JT, and storing the relearning requirement level LVRC in the judgment result storage unit 225. The method of calculating the relearning requirement level LVRC is similar to the method of calculating the relearning requirement level LVRk.

[0085] As described above, the display control unit 234 in the first embodiment displays the analysis result display window 300 (see FIG. 9) on the output unit 213. The display control unit 234 in the second embodiment adds a display line for displaying a relearning requirement level LVRC (not shown) based on cluster comparison, in the relearning requirement level display unit 310. Furthermore, the display control unit 234 in the second embodiment has a function of displaying, in the case where the relearning requirement level LVRC is selected (for example, when the corresponding display line is clicked), a transition of the first cluster drift index CD1MC (not shown), a transition of the second cluster drift index CD2MC (not shown), and a transition of the relearning requirement level LVRC (not shown) on the graph display unit 320.

[0086] [Effects of the embodiment] As described above, according to the embodiment, the management device 100 includes a first comparison unit 231 that calculates first data drift indexes CD1M1-CD1Mn based on comparison results for frequency distributions FDL1-FDLn, FDP1-FDPn of one or more feature amounts DS1-DSn included in the input data DI, a second comparison unit 232 that calculates second data drift indexes CD2M1-CD2Mn based on comparison results for the multiple frequency distributions FDL1-FDLn, FDP1-FDPn under comparison conditions different from those of the comparison in the first comparison unit 231, and a judgment unit 233 that outputs a re-learning necessity level LVRk indicating the degree of necessity for re-learning of the machine learning model (250) based on the first data drift indexes CD1M1-CD1Mn, the second data drift indexes CD2M1-CD2Mn, and a judgment method table JT that defines the judgment criteria (CD1k, CD2k). As a result, the re-learning required level LVRk can be output using the first data drift indices CD1M1 to CD1Mn and the second data drift indices CD2M1 to CD2Mn, so that appropriate information regarding the re-learning of the machine learning model can be provided to the user.

[0087] In addition, the first comparison unit 231 calculates first data drift indexes CD1M1 to CD1Mn based on a first frequency distribution (FDL1 to FDLn) which is a frequency distribution of the features DS1 to DSn in the learning data DL which is data within a time range used for learning to construct a machine learning model among the input data DI, and a second frequency distribution (FDP1 to FDPn) which is a frequency distribution of the features DS1 to DSn in the operational data DP which is input data DI belonging to a newer range than the learning data DL. It is more preferable to calculate the second data drift indexes CD2M1-CD2Mn based on a third frequency distribution (FDPx1-FDPxn) which is a frequency distribution of the features DS1-DSn in the first partial operation data (DPx) which is the input data DI belonging to a new time range, and a fourth frequency distribution (FDPy1-FDPyn) which is a frequency distribution of the features DS1-DSn in the second partial operation data (DPy) which is the input data DI belonging to a newer time range than the learning data DL and belongs to a different time range from the first partial operation data (DPx). This makes it possible to provide more appropriate information regarding the re-learning of the machine learning model based on the transition state of the frequency distribution.

[0088] Furthermore, it is more preferable that the first data drift indexes CD1M1-CD1Mn and the second data drift indexes CD2M1-CD2Mn are indexes that represent the mutual distances between the multiple frequency distributions FDL1-FDLn and FDP1-FDPn of the feature quantities DS1-DSn. This makes it possible to provide more appropriate information regarding re-learning of the machine learning model based on the distances between the frequency distributions.

[0089] Furthermore, it is more preferable that the first data drift indexes CD1M1-CD1Mn and the second data drift indexes CD2M1-CD2Mn are indexes representing the mutual similarity between the multiple frequency distributions FDL1-FDLn, FDP1-FDPn of the feature quantities DS1-DSn. This makes it possible to provide more appropriate information regarding re-learning of the machine learning model based on the similarity between the frequency distributions.

[0090] Moreover, it is more preferable that the determination method table JT includes determination reason information DRNk, which is character string data explaining the reason for determining the corresponding relearning requirement level LVRk, and further includes a display control unit 234 that displays the relearning requirement level LVRk and the corresponding determination reason information DRNk on the display device (213). This allows the user to know the reason why the relearning requirement level LVRk was assigned from the determination reason information DRNk, which is character string data, and therefore provides more appropriate information regarding the relearning of the machine learning model.

[0091] Moreover, it is more preferable that the determination method table JT is configured so that its contents can be set by the user, thereby allowing the user to configure an optimal determination method table JT based on the properties of the target system 110 and the features DS1 to DSn, etc.

[0092] Also, as in the configuration of the second embodiment, the management device 100 further includes a cluster creation unit 235 that clusters the input data DI based on the feature quantities DS1 to DSn to classify the input data DI into a plurality of clusters CS1, CS2, and CS3, and a clustering execution unit 236 that determines to which of the already created clusters CS1, CS2, and CS3 the input data DI belongs, and the first comparison unit 231 has a function of calculating a first cluster drift index CD1MC based on a comparison result regarding the frequency distributions CFDL and CFDP of the clusters CS1, CS2, and CS3 in the input data DI, and the second comparison unit 232 has a function of calculating a second cluster drift index CD2MC based on a comparison result regarding the frequency distributions CFDL and CFDP of the clusters CS1, CS2, and CS3 in the input data DI under different comparison conditions from the comparison in the first comparison unit 231. This makes it possible to provide more appropriate information regarding the re-learning of the machine learning model based on the state of the cluster.

[0093] [Variations] The present invention is not limited to the above-mentioned embodiment, and various modifications are possible. The above-mentioned embodiment is exemplified to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to delete a part of the configuration of each embodiment, or to add or replace other configurations. In addition, the control lines and information lines shown in the figure show those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines necessary on the product. In reality, it may be considered that almost all the configurations are connected to each other. Possible modifications of the above-mentioned embodiment are, for example, as follows.

[0094] (1) In each of the above embodiments, the comparison condition of the first comparing unit 231 was to calculate the first data drift indexes CD1M1 to CD1Mn based on a comparison between the frequency distributions FDL1 to FDLn of the features DS1 to DSn in the learning data DL and the frequency distributions FDP1 to FDPn of the features DS1 to DSn in the operational data DP.

[0095] In addition, the comparison condition of the second comparing section 232 was to calculate the second data drift indexes CD2M1 to CD2Mn based on the comparison result between the frequency distribution FDPx1 to FDPxn of the features DS1 to DSn in the partial operation data DPx and the frequency distribution FDPy1 to FDPyn of the features DS1 to DSn in the partial operation data DPy.

[0096] However, the comparison conditions of the first comparing section 231 and the second comparing section 232 are not limited to those described above. In other words, as long as the comparison conditions of the first comparing section 231 and the second comparing section 232 are different, various comparison conditions other than those described above can be applied.

[0097] (2) Since the hardware of the management device 100 in the above embodiment can be realized by a general-purpose computer, the processes corresponding to the above-mentioned block diagrams and flowcharts, as well as programs for executing the various processes described above, may be stored on a storage medium (a computer-readable recording medium on which a program is recorded) or distributed via a transmission line.

[0098] (3) In the above embodiment, the processes corresponding to the block diagrams and flowcharts, and the programs for executing the various processes described above have been described as software-based processes using programs. However, some or all of them may be replaced with hardware-based processes using an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), etc.

[0099] (4) The various processes executed in the above-described embodiments may be executed by a server computer via a network (not shown), and the various data stored in the above-described embodiments may also be stored in the server computer. [Explanation of symbols]

[0100] 100 Management device 110 Target System 213 Output section (display device) 231 First Comparison Section 232 Second Comparison Section 233 Judgment section 234 Display control section 235 Cluster Creation Department 236 Clustering Execution Unit 250 State Estimation Unit (State Estimation Process) DI Input Data DL training data DP Operational Data JT Judgment Method Table DPx partial operation data (first partial operation data) DPy partial operation data (second partial operation data) CD1k 1st comparison data (criteria) CD2k 2nd comparison data (criteria) CFDL frequency distribution CFDP frequency distribution DRNk Judgment Reason Information LVRk Re-learning Level R231 First comparison routine (first comparison process) R232 Second comparison routine (second comparison process) R233 Relearning Required Level Calculation Routine (Determination Process) CD1MC First cluster drift index CD2MC Second Cluster Drift Index CS2,CS3 (cluster) DS1~DSn feature quantity FDL1~FDLn frequency distribution (first frequency distribution) FDP1~FDPn frequency distribution (second frequency distribution) CD1M1~CD1Mn First data drift index CD2M1~CD2Mn Second data drift index CS1,CS2,CS3 Cluster FDPx1~FDPxn frequency distribution (third frequency distribution) FDPy1~FDPyn frequency distribution (fourth frequency distribution)

Claims

1. A state estimation unit that predicts a state of the target system by using input data acquired from the target system and a machine learning model; a first comparison unit that calculates a first data drift index based on a comparison result regarding a frequency distribution of one or more feature amounts included in the input data; a second comparison unit that calculates a second data drift index based on a comparison result regarding a plurality of frequency distributions, the comparison condition of which is different from a comparison condition in the first comparison unit; a determination unit that outputs a re-learning necessity level indicating a degree of necessity for re-learning the machine learning model based on the first data drift index, the second data drift index, and a determination method table that defines a determination criterion. A management device comprising:

2. the first comparison unit calculates the first data drift index based on a first frequency distribution that is a frequency distribution of the feature amounts in learning data, which is data in a time range used for learning to construct the machine learning model among the input data, and a second frequency distribution that is a frequency distribution of the feature amounts in operational data, which is the input data belonging to a newer range than the learning data; The second comparison unit calculates the second data drift index based on a third frequency distribution which is a frequency distribution of the feature amount in first partial operation data which is the input data belonging to a newer time range than the learning data, and a fourth frequency distribution which is a frequency distribution of the feature amount in second partial operation data which is the input data belonging to a newer time range than the learning data and belongs to a different time range than the first partial operation data.

2. The management device according to claim 1.

3. The first data drift index and the second data drift index are indexes that represent distances between the plurality of frequency distributions of the feature quantities.

2. The management device according to claim 1.

4. The first data drift index and the second data drift index are indexes representing a mutual similarity between the plurality of frequency distributions of the feature amount.

2. The management device according to claim 1.

5. the determination method table includes determination reason information which is character string data explaining a reason for determining the corresponding re-learning requirement level, The vehicle further includes a display control unit that displays the relearning requirement level and the corresponding determination reason information on a display device.

2. The management device according to claim 1.

6. The determination method table is configured so that its contents can be set by the user.

2. The management device according to claim 1.

7. a cluster creating unit that clusters the input data based on the feature amount and classifies the input data into a plurality of clusters; A clustering execution unit that determines to which of the already created clusters the input data belongs, the first comparison unit has a function of calculating a first cluster drift index based on a comparison result regarding a frequency distribution of the cluster in the input data; The second comparison unit has a function of calculating a second cluster drift index based on a comparison result regarding a frequency distribution of the cluster in the input data, the comparison condition being different from that of the comparison in the first comparison unit.

2. The management device according to claim 1.

8. a state estimation process for predicting a state of the target system using input data acquired from the target system and a machine learning model; a first comparison step of calculating a first data drift index based on a comparison result regarding a frequency distribution of one or more feature amounts included in the input data; a second comparison step of calculating a second data drift index based on a comparison result of a plurality of frequency distributions, the comparison condition of which is different from that of the first comparison step; and a determination step of outputting a re-learning necessity level indicating a degree of necessity for re-learning the machine learning model based on the first data drift index, the second data drift index, and a determination method table that defines a determination criterion. A management method comprising:

Images