Machine learning method and machine learning device

Abstract

To provide a machine learning method that can improve the performance of a learning model by machine learning.SOLUTION: A machine learning method includes machine learning of a learning model using a data set composed of data at respective points of time in a predetermined period. The machine learning method comprises: data set dividing processing of sectioning a period into a learning period and a verification period, and dividing the data set into learning data and verification data; learning candidate period sectioning processing of sectioning a learning candidate period into a plurality of divided learning periods; learning period evaluation processing of using learning data corresponding to the divided learning periods to generate learning period evaluation model for each divided learning period, and using the verification data to calculate an evaluation index of the learning period evaluation model for each divided learning period; learning period selecting processing of repeatedly performing the learning candidate period sectioning processing and the learning period evaluating processing until predetermined end conditions are met, and selecting a divided learning period with a high evaluation index as a learning object period when the predetermined conditions are met; and model output processing of outputting a learning model having machine learning completed using learning data corresponding to the learning object period.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a machine learning method and a machine learning apparatus. [Background technology]

[0002] Traditionally, when performing machine learning on a dataset, a technique called cross-validation has been used. Cross-validation involves dividing the dataset into training data and validation data, performing machine learning on the training data, and evaluating the performance of the trained model using the validation data (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2019-003408 [Patent Document 2] Japanese Patent Publication No. 2019-061565 [Overview of the project] [Problems that the invention aims to solve]

[0004] When using cross-validation, if the training data includes unique data, such as data with significantly different features from other data, there was a possibility that the performance of the training model would be degraded due to the influence of that unique data.

[0005] The present invention has been made in view of the above-mentioned problems, and aims to provide a machine learning method and a machine learning apparatus that enable the improvement of the performance of machine learning learning models. [Means for solving the problem]

[0006] To achieve the above objective, a machine learning method according to one aspect of the present invention is: A machine learning method for performing machine learning on a learning model using a dataset constructed by arranging data acquired by a computer at various points in time in accordance with the temporal changes of a predetermined phenomenon over a predetermined period, in a time series, The aforementioned period is divided into a learning period and a validation period, the learning period is set as the initial candidate learning period, and the dataset is divided into learning data corresponding to the learning period and validation data corresponding to the validation period. A learning candidate period division process that divides the aforementioned learning candidate period into multiple divided learning periods, A learning period evaluation process that generates a learning period evaluation model for each divided learning period by performing machine learning of the learning model using the learning data corresponding to the divided learning period, and calculates an evaluation index for the learning period evaluation model for each divided learning period by performing verification of the learning period evaluation model for each divided learning period using the verification data, The learning candidate period segmentation process and the learning period evaluation process are repeated until a predetermined termination condition is met, setting the segmented learning period with the highest evaluation index as the next learning candidate period. When the termination condition is met, a learning period selection process is performed to select the segmented learning period with the highest evaluation index as the learning target period. The process involves outputting the trained model, which has been machine-learned using the training data corresponding to the training period, as a trained model.

[0007] Furthermore, a machine learning method according to another aspect of the present invention is: A machine learning method for performing machine learning on a learning model using a dataset constructed by arranging data acquired by a computer at various points in time in accordance with the temporal changes of a predetermined phenomenon over a predetermined period, in a time series, The aforementioned period is divided into a learning period and a validation period, the learning period is set as the initial candidate learning period, and the dataset is divided into learning data corresponding to the learning period and validation data corresponding to the validation period. A learning data splitting process that divides the learning period into multiple sub-learning periods and splits the learning data into multiple sub-learning data corresponding to the multiple sub-learning periods, A learning candidate period division process that divides the multiple sub-learning periods included in the aforementioned learning candidate period into multiple divided learning periods, A learning period evaluation process that generates a learning period evaluation model for each divided learning period by performing machine learning of the learning model for each divided learning period using the sub-learning data corresponding to the sub-learning period included in the divided learning period, and calculates an evaluation index for the learning period evaluation model for each divided learning period by performing verification of the learning period evaluation model for each divided learning period using the verification data, The learning candidate period segmentation process and the learning period evaluation process are repeated until a predetermined termination condition is met, setting the segmented learning period with the highest evaluation index as the next learning candidate period. When the termination condition is met, a learning period selection process is performed to select the sub-learning period included in the segmented learning period with the highest evaluation index as the learning target period. The process involves outputting the trained model, which has been machine-trained using the sub-training data corresponding to the training period, as a trained model. [Effects of the Invention]

[0008] According to the machine learning method of the present invention, the learning period selection process repeatedly performs a learning candidate period division process and a learning period evaluation process. As a result, a divided learning period with a high evaluation index of the learning period evaluation model is selected as the learning target period, and machine learning of the learning model is performed using the learning data corresponding to that learning target period. Therefore, the performance of the learning model by machine learning can be improved.

[0009] Other issues, configurations, and effects will be clarified in the embodiments for carrying out the invention described later. [Brief explanation of the drawing]

[0010] [Figure 1]This is a diagram showing an example of a machine learning device 1. [Figure 2] This is a hardware configuration diagram showing an example of computer 200. [Figure 3] This is a flowchart showing an example of a machine learning method S1 according to the first embodiment. [Figure 4] This is a flowchart (continuation of Figure 3) showing an example of the machine learning method S1 according to the first embodiment. [Figure 5] This figure shows an example of the execution of the machine learning method S1 according to the first embodiment. [Figure 6] This figure (a continuation of Figure 5) shows an example of the execution of the machine learning method S1 according to the first embodiment. [Figure 7] This figure shows an example of calculating an evaluation index in the machine learning method S1 according to the first embodiment. [Figure 8] This is a flowchart showing an example of a machine learning method S2 according to the second embodiment. [Figure 9] This is a flowchart (continuation of Figure 8) showing an example of the machine learning method S2 according to the second embodiment. [Figure 10] This figure shows an example of the execution of the machine learning method S2 according to the second embodiment. [Figure 11] This figure (a continuation of Figure 10) shows an example of the execution of the machine learning method S2 according to the second embodiment. [Figure 12] This figure shows an example of calculating an evaluation index in the machine learning method S2 according to the second embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In the following, the scope necessary for explaining how to achieve the objectives of the present invention will be schematically shown, and the scope necessary for explaining the relevant parts of the present invention will be mainly explained, with any parts that are omitted from explanation being based on prior art.

[0012] (First embodiment) Figure 1 is a configuration diagram showing an example of a machine learning device 1 according to the first embodiment. The machine learning device 1 is a device capable of executing a machine learning method S1 that performs machine learning on a learning model M using a dataset D, and is composed of, for example, a general-purpose or dedicated computer (see Figure 2 described later).

[0013] Dataset D is constructed by arranging data acquired at various points in time in accordance with the temporal changes of a predetermined phenomenon over a predetermined period P. The predetermined phenomenon is, for example, one or more variables representing the state of the observed object that change over time, such as mechanical phenomena, electromagnetic phenomena, thermal phenomena, acoustic phenomena, chemical phenomena, physiological phenomena, etc., and may also be natural phenomena or economic phenomena. For each phenomenon, the variables representing the state of the observed object are measured at predetermined time intervals by various sensors, for example, and these measured variables are digitized as data. By collecting and accumulating this data in chronological order, it is constructed as Dataset D used in the machine learning method S1. Dataset D may be data collected in the past or data collected in real time.

[0014] The machine learning device 1 comprises, as its main components, a control unit 10, a storage unit 11, an input unit 12, an output unit 13, a communication unit 14, and a device connection unit 15.

[0015] The control unit 10 functions as a machine learning unit 100 by executing the machine learning program 110 stored in the memory unit 11, and performs various processes performed by the machine learning method S1. In addition to storing the machine learning program 110 and various data used by the machine learning program 110, the memory unit 11 also stores, for example, the operating system (OS), other programs and data. The input unit 12 accepts various input operations, and the output unit 13 functions as a user interface by outputting various information via a display screen and sound. The communication unit 14 is connected to a wired or wireless network and functions as a communication interface for sending and receiving various data with other devices (not shown). The device connection unit 15 functions as a connection interface for external devices such as scanners and printers, measuring instruments such as sensors, and storage media such as USB memory, CD-ROMs, and DVDs.

[0016] In machine learning method S1, machine learning is performed on the trained model M using the dataset D, and the trained model is then provided to any user system. If the dataset D is based on variables of monitored or controlled objects in equipment or facilities (or even just one of the components of the equipment or facilities), the trained model can be used to estimate the state of the monitored object (anomalies, failures, their precursors, etc.) or to control the controlled object by inputting unknown variables (which may be collected in the past or collected in real time).

[0017] The learning model M is a model that implements any machine learning algorithm or method, and can handle classification problems such as binary classification or multi-class classification, or it can handle regression problems. The learning model M can be, for example, a tree-type model such as decision trees or regression trees, an ensemble learning model such as bagging or boosting, a neural network type such as a deep neural network, a recurrent neural network, a convolutional neural network, or an LSTM (deep neural network). (including hierarchical clustering), hierarchical clustering, non-hierarchical clustering, k-nearest neighbors, k-means clustering Examples of clustering methods include, but are not limited to, multivariate analyses such as principal component analysis, factor analysis, logistic regression, and support vector machines. Furthermore, the learning model M may employ any of the following machine learning techniques: supervised learning, unsupervised learning, or reinforcement learning.

[0018] Figure 2 is a hardware configuration diagram showing an example of computer 200. Computer 200 is an example of a device that makes up machine learning apparatus 1, and is configured as a general-purpose or dedicated computer.

[0019] As shown in Figure 2, the computer 200 comprises, as its main components, a bus 210, a processor 212, memory 214, an input device 216, an output device 217, a display device 218, a storage device 220, a communication interface unit 222, an external device interface unit 224, an I / O device interface unit 226, and a media input / output unit 228. Note that the above components may be omitted as appropriate depending on the intended use of the computer 200.

[0020] The processor 212 consists of one or more arithmetic processing units (CPU (Central Processing Unit), MPU (Micro-processing unit), DSP (digital signal processor), GPU (Graphics Processing Unit), etc.) and operates as a control unit 10 that oversees the entire computer 200. The memory 214 stores various data and programs 230 and consists of volatile memory (DRAM, SRAM, etc.) that functions as main memory, and non-volatile memory (ROM), flash memory, etc.

[0021] The input device 216 consists of, for example, a keyboard, mouse, numeric keypad, or electronic pen, and functions as the input unit 12. The output device 217 consists of, for example, a sound (voice) output device or a vibration device, and functions as the output unit 13. The display device 218 consists of, for example, a liquid crystal display, an organic EL display, electronic paper, or a projector, and functions as the output unit 13. The input device 216 and the display device 218 may be configured as an integrated unit, such as a touch panel display. The storage device 220 consists of, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and functions as the storage unit 11. The storage device 220 stores various data necessary for the execution of the operating system and the program 230.

[0022] The communication I / F unit 222 is connected to a network 240 such as the Internet or an intranet by wire or wireless connection and functions as a communication unit 14 that sends and receives data with other computers according to a predetermined communication standard. The external device I / F unit 224 is connected to external devices 250 such as cameras, printers, scanners, and reader / writers by wire or wireless connection and functions as a device connection unit 15 that sends and receives data with the external devices 250 according to a predetermined communication standard. The I / O device I / F unit 226 is connected to I / O devices 260 such as various sensors and actuators and functions as a device connection unit 15 that sends and receives various signals and data with the I / O devices 260, such as detection signals from sensors and control signals to actuators. The media input / output unit 928 consists of, for example, a drive device such as a DVD (Digital Versatile Disc) drive or a CD (Compact Disc) drive, a memory card slot, and a USB connector, and supports DVDs, CDs, and memory cards. It functions as a device connection unit 15 that reads and writes data to media (non-temporary storage media) 970 such as a USB memory stick.

[0023] In the computer 200 having the above configuration, the processor 212 calls and executes the program 230 stored in the storage device 220 in the memory 214, and controls various parts of the computer 200 via the bus 210. The program 230 may also be stored in the memory 214 instead of the storage device 220. The program 230 may be recorded on the media 270 in an installable or executable file format and provided to the computer 200 via the media input / output unit 228. The program 230 may also be provided to the computer 200 by downloading it via the network 240 through the communication interface unit 222. Furthermore, the computer 200 may implement the various functions realized by the processor 212 executing the program 230 using hardware such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit).

[0024] Computer 200 is an electronic device of any form, consisting of, for example, a stationary computer or a portable computer. Furthermore, Computer 200 may be a client computer, a server computer, or a cloud computer.

[0025] Next, the details of the machine learning method S1 performed by the control unit 10 (machine learning unit 100) of the machine learning device 1 having the above configuration will be explained with reference to Figures 3 to 7.

[0026] Figures 3 and 4 are flowcharts showing an example of the machine learning method S1 according to the first embodiment. Figures 5 and 6 show an example of the execution of the machine learning method S1 according to the first embodiment. Figure 7 shows an example of the calculation of an evaluation index in the machine learning method S1 according to the first embodiment.

[0027] First, in step S100 (dataset acquisition process), the machine learning device 1 acquires a dataset D containing data from each point in time during a predetermined period P in chronological order. The dataset D may be read from the storage unit 11, received from another device via the communication unit 14, referenced from a storage medium via the device connection unit 15, or measured by a measuring instrument via the device connection unit 15.

[0028] Next, in step S110 (dataset splitting process), period P is divided into a training period PL and a validation period PT, the training period PL is set as the initial training candidate period PL1, and the dataset D is split into training data DL corresponding to the training period PL and validation data DT corresponding to the validation period PT.

[0029] Period P is divided into a learning period PL (e.g., 90%) and a validation period PT (e.g., 10%) based on a predetermined data ratio, such as "9:1". In this case, the validation period PT may be set discretely within the time series of period P. For example, if period P is 35 days, it is preferable to distribute the validation period PT across the time series of period P, such as one day in the early period (days 1-12), one day in the middle period (days 13-24), and one day in the later period (days 25-35), and the validation period PT may be set as day 6, day 18, and day 30. This helps to suppress bias in the data included in the validation data.

[0030] Furthermore, the initial candidate learning period PL1 is repeatedly subjected to the candidate learning period classification process S130 and the learning period evaluation processes S140 and S141 in the learning period selection process S150-S152 described later. The candidate learning period when it is performed PL iIt indicates the initial value of (where i = 1, 2, …, R (R is an integer greater than or equal to 2)). "i" is a variable (index) indicating the number of repetitions, and in the dataset division process S110, it is initialized with the number of repetitions i = 1. Also, "R" indicates the upper limit of the number of repetitions, and in this embodiment, it will be described assuming that the upper limit of repetitions R = 3.

[0031] Next, in step S130 (learning candidate period classification process), the learning candidate period PL i is classified into a plurality of divided learning periods PLD i,j (where j = 1, 2, …, S (S is an integer greater than or equal to 2)). "j" is a variable (index) indicating an identifier for identifying a plurality of divided learning periods PLD i,j . Also, "S" indicates the number of divisions when classifying a plurality of divided learning periods PLD i,j , and in this embodiment, it will be described assuming that the number of divisions S = 2. Therefore, in this embodiment, when the number of repetitions i is "1", the initial learning candidate period PL1 is divided into two divided learning periods PLD 1,1 , PLD [[ID=1⑤]] 1,2 corresponding to the number of divisions S being "2".

[0032] Note that each of the plurality of divided learning periods PLD i,j may be divided into a plurality of divided learning periods PLD i such that each includes a discrete period in the time series of the learning candidate period PL i,j . For example, if the learning candidate period PL1 is 32 days out of 35 days, excluding the 6th, 18th, and 30th days, the divided learning period PLD 1,1 includes 16 days from the 1st to the 5th and from the 19th to the 29th days, and the divided learning period PLD 1,2 includes 16 days from the 7th to the 17th and from the 31st to the 35th days. Thus, the bias of the data included in each of the plurality of divided learning periods PLD i,j can be suppressed.

[0033] Next, in steps S140 and S141 (learning period evaluation process), the divided learning period PLD i,jThe corresponding training data DL is the segmented training data DLD. i,j Using the learning model M, the machine learning process is divided into parts and the learning period is divided into parts. i,j Performing this each time divides the learning period into PLDs. i,j Evaluation model ML for each learning period i,j This generates the data. Then, the validation data DT is used to divide the learning period into PLD. i,j Evaluation model ML for each learning period i,j By performing verification on each of these, the learning period evaluation model ML can be developed. i,j Evaluation index EL i,j The learning period is divided into PLD i,j Calculate each time.

[0034] Learning Period Assessment Model (ML) i,j Evaluation index EL i,j Examples of evaluation metrics include accuracy, precision, recall, specificity, and F-score. A single evaluation metric may be used, or a combination of multiple evaluation metrics may be used. In this embodiment, the evaluation metric EL i,j Assuming precision is used, if the number of iterations i is "1", then the split learning period PLD 1,1 Machine learning was used to create a learning period evaluation model ML. 1,1 Precision (evaluation index EL) 1,1 =36.5%) and the segmented learning period PLD 1,2 Machine learning was used to create a learning period evaluation model ML. 1,2 Precision (evaluation index EL) 1,2 =23.6% and these values ​​are calculated respectively.

[0035] Next, in steps S150 to S152 (learning period selection process), it is determined whether or not to terminate the iterative process of repeatedly performing the learning candidate period classification process S130 and the learning period evaluation processes S140 and S141, according to predetermined termination conditions (step S150). The termination conditions are, for example, whether or not the number of repetitions i exceeds the upper limit of repetitions R (in this embodiment, R=3) when the number of repetitions i is incremented, and the evaluation index EL i,j This is determined by whether or not the value falls outside a predetermined limit (for example, a lower limit).

[0036] Then, if the termination condition is not met (step S151: No), the evaluation index EL is used to continue the iterative process. i,j High-segmented learning period PLD i,j Next learning candidate period PL i By setting it to (step S151) and returning to step S130, the above learning candidate period segmentation process S130 and learning period evaluation processes S140 and S141 are completed, satisfying the termination condition. Repeat until the end. For example, in this embodiment, if the number of repetitions i is "1", the evaluation index EL 1,1 High-segmented learning period PLD 1,1 This is set as the next candidate learning period PL2. Note that if the number of categories S is "2", the evaluation index EL is set. i,j The higher of the two split learning periods PLD i,j The next candidate learning period PL i It is set to, but if the number of categories S is "3 or more", the evaluation index EL i,j The single split learning period with the highest value (PLD) i,j The next candidate learning period PL i It may be set to the evaluation index EL i,j Multiple split learning periods in descending order of learning duration i,j The next candidate learning period PL i It can also be used as a setting.

[0037] On the other hand, when the termination condition is met (step S151: Yes), the loop process is terminated, and the evaluation index EL is reached. i,j High-segmented learning period PLD i,j Select the learning period PLF (Step S152). Note that if the number of categories S is "2", the evaluation index EL i,j The higher of the two split learning periods PLD i,j The learning period PLF is set, but if the number of categories S is "3 or more", the evaluation index EL i,j The single split learning period with the highest value (PLD) i,j The learning period PLF may be set, and the evaluation metric EL may be set. i,j Multiple split learning periods in descending order of learning duration i,j The learning period PLF may be set to this.

[0038] Then, in step S160 (first model output processing), the trained model M, which has been machine-trained using the training data DL (hereinafter referred to as "training target data DLF") corresponding to the training period PLF, is output as the first trained model MF1. The output destination of the first trained model MF1 may be the storage unit 11 or another device. The first trained model MF1 includes, for example, trained coefficients and parameters for characterizing the trained model M. Note that if the second trained model MF1 is output in the second model output processing step S190 described later, the output of the first trained model MF1 may be omitted.

[0039] Next, in step S170 (training data splitting process), the training period PLC, excluding the training period PLF, is divided into multiple sub-training periods PLS. k The learning data DLC, excluding the learning data DLF corresponding to the learning period PLF, is divided into multiple sub-learning periods PLS. k Multiple sub-training datasets corresponding to DLS k It is divided into multiple sub-training data DLS. "T" represents multiple sub-training data DLS. k This indicates the number of divisions when dividing into parts, and in this embodiment, the number of divisions T = 7 (= 2 R -1=2 3 -1) and 7 sub-training data DLS k We will explain this assuming that it is divided into (k=2,3,…,8).

[0040] Next, in steps S180 and S181 (additional period selection process), multiple sub-training data DLS k By validating the first trained model MF1 using each of these, the evaluation metric EF of the first trained model MF1 is calculated. k Sub-learning period PLS k It is calculated each time. Then, the evaluation metric EF of the first trained model MF1 is calculated. k Sub-learning period PLS that meets the specified additional conditions k Select the period to be added to the PLA. Additional conditions include, for example, the evaluation metric EF. kUsing a threshold value for the evaluation index EF, k The determination is made based on whether or not that threshold is exceeded. As a result, one sub-learning period PLS is selected as the additional target period PLA. k One may be selected, or multiple sub-learning periods PLS k The following may be selected. In the examples in Figures 6 and 7, two sub-learning periods PLS 2、 The diagram illustrates the case where PLS6 is selected as the PLA for the additional period.

[0041] Then, in step S190 (second model output processing), a trained model M, which has been machine-trained using training data DL corresponding to at least one of the training period PLF and the additional training period PLA, is output as the second trained model MF2. That is, training data DLF corresponding to the training period PLF, and sub-training data DLS corresponding to the additional training period PLA. k (In the examples in Figures 6 and 7, DLS 2、 Machine learning is performed on the learning model M using at least one of DLS6), and the output is as a second trained model MF2, and a series The process is terminated. The output destination for the second trained model MF2 may be the memory unit 11, as with the first trained model MF1, or it may be another device. The second trained model MF2, like the first trained model MF1, includes, for example, trained coefficients and parameters for characterizing the trained model M.

[0042] The second pre-trained model, MF2, uses sub-training data DLS corresponding to the additional target period PLA. k You may use this to perform additional machine learning (retraining) on ​​the first trained model MF1, or you may use sub-training data DLS corresponding to the training data DLF and the additional training period PLA. k You may use this to perform machine learning on the initialized training model M. Alternatively, the second trained model MF2 may consist of the training data DLF and the sub-training data DLS. kBy arbitrarily combining these and performing machine learning on the learning model M, multiple second pre-trained models MF2 may be output. For example, the training data DLF and the sub-training data DLS 2、 The output may include a machine learning model M using DLS6, a machine learning model M using the target data DLF and sub-training data DLS2, and a machine learning model M using the target data DLF and sub-training data DLS6.

[0043] As described above, according to the machine learning method S1 of this embodiment, the learning period selection process S150 to S152 repeatedly performs the learning candidate period classification process S130 and the learning period evaluation processes S140 and S141, thereby creating a learning period evaluation model ML i,j Evaluation index EL i,j High-segmented learning period PLD i,j The period PLF is selected as the learning target period PLF, and machine learning of the learning model M is performed using the learning target data DLF corresponding to that learning target period PLF. Therefore, the performance of the learning model can be improved by machine learning.

[0044] Furthermore, the additional period selection processing steps S180 and S181 determine the evaluation index EF of the first trained model MF1. k Sub-learning period PLS that meets the specified additional conditions k The PLA was selected as an additional target period, and the sub-training data DLS corresponding to that PLA was selected. k Machine learning of the learning model M is performed using the training data DLF. Therefore, overfitting caused by performing machine learning using the training data DLF is prevented by using the sub-training data DLS. k This allows for suppression while improving the performance of the learning model.

[0045] (Second embodiment) The basic configuration of the machine learning device 1 according to this embodiment is the same as that of the first embodiment, but the details of the machine learning method S2 performed by the control unit 10 (machine learning unit 100) of the machine learning device 1 differ from the machine learning method S1 in the first embodiment. Specifically, in the first embodiment, the training data splitting process S170 is performed after the first model output process S160, whereas in the second embodiment, the training data splitting process S220 is performed after the dataset splitting process S210. Below, the details of the machine learning method S2 performed by the control unit 10 (machine learning unit 100) of the machine learning device 1 according to this embodiment will be described with reference to Figures 8 to 12.

[0046] Figures 8 and 9 are flowcharts showing an example of the machine learning method S2 according to the second embodiment. Figures 10 and 11 show an example of the execution of the machine learning method S2 according to the second embodiment. Figure 12 shows an example of the calculation of an evaluation index in the machine learning method S2 according to the second embodiment.

[0047] First, in step S200 (dataset acquisition process), similar to step S100, the machine learning device 1 acquires a dataset D containing data from each point in time during a predetermined period P in chronological order.

[0048] Next, in step S210 (dataset splitting process), similar to step S110, Period P is divided into a training period PL and a validation period PT. The training period PL is set as the initial candidate training period PL1, and the dataset D is split into training data DL corresponding to the training period PL and validation data DT corresponding to the validation period PT.

[0049] The initial candidate learning period PL1 is the candidate learning period PL when the candidate learning period classification process S230 and the learning period evaluation processes S240 and S241 are repeatedly executed in the learning period selection process S250~S252 described later. iIt indicates the initial value of (i = 1, 2, …, R (R is an integer greater than or equal to 2)). "i" is a variable (index) indicating the number of repetitions, and in the dataset splitting process S210, it is initialized with the number of repetitions i = 1. Also, "R" indicates the upper limit value of the number of repetitions. In this embodiment, it will be described assuming that the upper limit value of repetitions R = 3.

[0050] Next, in step S220 (learning data splitting process), the learning period PL is divided into a plurality of sub - learning periods PLS k (k = 1, 2, …, T (S is an integer greater than or equal to 2)), and the learning data DL is divided into a plurality of sub - learning data DLS k corresponding to the plurality of sub - learning periods PLS k "T" indicates the number of divisions when dividing into a plurality of sub - learning data DLS k In this embodiment, the number of divisions T = 8 (= 2 R = 2 3 ), and it will be described as being divided into eight sub - learning data DLS k (k = 1, 3, …, 8).

[0051] Next, in step S230 (learning candidate period classification process), the plurality of sub - learning periods PLS i included in the learning candidate period PL k are classified into a plurality of divided learning periods PLD i,j (where j = 1, 2, …, S (S is an integer greater than or equal to 2)). "j" is a variable (index) indicating an identifier for identifying the plurality of divided learning periods PLD i,j Also, "S" indicates the number of classes when classifying the plurality of divided learning periods PLD i,j In this embodiment, it will be described assuming that the number of classes S = 2. Therefore, in this embodiment, when the number of repetitions i is "1", the initial learning candidate period PL1 is classified into two divided learning periods PLD 1,1 , PLD 1,2 corresponding to the number of classes S being "2". At that time, the divided learning period PLD 1,1 including four sub - learning periods PLS1 to PLS4, and the divided learning period PLD1,2 They are classified into two categories.

[0052] Note that there are multiple split learning periods PLD i,j Each of these is a candidate learning period PL i Multiple sub-learning periods PLS, each containing discrete periods in the time series. k Randomize the time series in the PLD with multiple split learning periods. i,j It may be divided into the following categories. For example, the divided learning period PLD 1,1 This includes four sub-learning periods PLS1, PLS3, PLS5, and PLS7, and a segmented learning period PLD 1,2 It may be divided to include four sub-learning periods PLS2, PLS4, PLS6, and PLS8. This allows for multiple sub-learning periods PLS k This can suppress bias in the data contained in each of them.

[0053] Next, in steps S240 and S241 (learning period evaluation process), the divided learning period PLD i,j Sub-learning period included in PLS k Corresponding sub-training data DLS k Using the learning model M, the machine learning process is divided into parts and the learning period is divided into parts. i,j Performing this each time divides the learning period into PLDs. i,j Evaluation model ML for each learning period i,j This generates the data. Then, the validation data DT is used to divide the learning period into PLD. i,j Evaluation model ML for each learning period i,j By performing verification on each of these, the learning period evaluation model ML can be developed. i,j Evaluation index EL i,j The learning period is divided into PLD i,j Calculate each time.

[0054] Next, in steps S250-S252 (learning period selection process), a determination is made, according to predetermined termination conditions, whether to terminate the repetitive process of repeating the learning candidate period classification process S230 and the learning period evaluation processes S240 and S241, similar to steps S150-S152 (step S250). The termination conditions are, for example, when the number of repetitions i is incremented. In this case, the evaluation index EL determines whether the number of repetitions i exceeds the upper limit of repetitions R (in this embodiment, R=3). i,j This is determined by whether or not the value falls outside a predetermined limit (for example, a lower limit).

[0055] Then, if the termination condition is not met (step S251: No), the evaluation index EL is used to continue the iterative process. i,j High-segmented learning period PLD i,j Next learning candidate period PL i By setting it to (step S251) and returning to step S230, the above learning candidate period classification process S230 and learning period evaluation processes S240 and S241 are repeated until the termination condition is met.

[0056] On the other hand, when the termination condition is met (step S251: Yes), the loop process is terminated, and the evaluation index EL is reached. i,j High-segmented learning period PLD i,j Sub-learning period included in PLS k Select the PLF as the learning period (Step S252).

[0057] Then, in step S260 (first model output processing), similar to step S160, the sub-training data DLS corresponding to the PLF for the training period is generated. k The trained model M, which has been machine-learned using the (hereinafter referred to as "training target data DLF"), is output as the first trained model MF1.

[0058] Next, in steps S280 and S281 (additional period selection process), multiple sub-training data DLS are selected, excluding the training target data DLF corresponding to the training target period PLF. k By validating the first trained model MF1 using each of these, the evaluation metric EF of the first trained model MF1 is calculated. k Sub-learning period PLS k It is calculated each time. Then, the evaluation metric EF of the first trained model MF1 is calculated. k Sub-learning period PLS that meets the specified additional conditions kSelect the additional target period PLA. In the examples in Figures 11 and 12, there are two sub-learning periods PLS. 2、 The diagram illustrates the case where PLS6 is selected as the PLA for the additional period.

[0059] Then, in step S290 (second model output processing), sub-training data DLS corresponding to at least one of the training period PLF and the additional training period PLA is generated. k The trained model M, which was created using machine learning, is output as the second trained model MF2. Specifically, the training data DLF corresponds to the training period PLF, and the sub-training data DLS corresponds to the additional training period PLA. k (In the examples in Figures 11 and 12, DLS 2、 Machine learning is performed on the trained model M using at least one of DLS6), and the second trained model MF2 is output, ending the series of processes.

[0060] As described above, according to the machine learning method S2 of this embodiment, the learning period selection process S250 to S252 repeatedly performs the learning candidate period classification process S230 and the learning period evaluation processes S240 and S241, thereby creating a learning period evaluation model ML i,j Evaluation index EL i,j High-segmented learning period PLD i,j Sub-learning period included in PLS k The period PLF is selected as the learning target period PLF, and machine learning of the learning model M is performed using the learning target data DLF corresponding to that learning target period PLF. Therefore, the performance of the learning model can be improved by machine learning.

[0061] Furthermore, the additional period selection processing steps S280 and S281 determine the evaluation index EF of the first trained model MF1. k Sub-learning period PLS that meets the specified additional conditions k The PLA was selected as an additional target period, and the sub-training data DLS corresponding to that PLA was selected. k Machine learning of the learning model M is performed using the training data DLF. Therefore, overfitting caused by performing machine learning using the training data DLF is prevented by using the sub-training data DLS.k This allows for suppression while improving the performance of the learning model.

[0062] (Other embodiments) The present invention is not limited to the embodiments described above, and can be implemented with various modifications without departing from the spirit of the invention. All such modifications are included in the technical concept of the present invention.

[0063] In the above embodiment, the machine learning methods S1 and S2 performed by the control unit 10 (machine learning unit 100) of the machine learning device 1 were described as being implemented in a single device. However, the machine learning methods S1 and S2 may be implemented in multiple devices by distributing the processing included in them across multiple devices.

[0064] In the above embodiment, the execution order of each step in the flowcharts showing the details of the machine learning methods S1 and S2 may be rearranged as appropriate, or some steps may be omitted. For example, in machine learning method S1, the processing from step S170 onwards may be omitted, and in machine learning method S2, the processing from step S280 onwards may be omitted.

[0065] In the above embodiment, the machine learning methods S1 and S2 were described as being performed according to various conditions and settings such as termination conditions, additional conditions, data proportion, repetition limit R, number of divisions S, and number of divisions T. However, these conditions and settings may be changed as appropriate, and for example, they may be configured to be changeable on the display screen by the input unit 12 and output unit 13 which function as a user interface.

[0066] In the above embodiment, the dataset splitting processes S110 and S210 in the machine learning methods S1 and S2 split the dataset D into training data DL and validation data DT, employing a so-called holdout validation method. However, the machine learning methods S1 and S2 may also employ, for example, k-fold cross-validation, stratified k-fold cross-validation, shuffled cross-validation, and time-series splitting. Cross-validation methods such as cross-validation and sliding cross-validation may be combined as appropriate. [Explanation of Symbols]

[0067] 1...Machine learning device, 10...Control unit, 11...Storage unit, 12...Input unit, 13...Output unit, 14...Communication section, 15...Equipment connection section, 100...Machine Learning Department, 110...Machine Learning Program, 200...Computer

Claims

1. A machine learning method for performing machine learning of a learning model by a computer using a dataset configured by arranging data at each point in time acquired in accordance with a time-dependent change in a predetermined phenomenon over a predetermined period in a chronological order, the method comprising: a dataset division process that divides the period into a learning period and a validation period, sets the learning period as an initial candidate learning period, and divides the dataset into learning data corresponding to the learning period and validation data corresponding to the validation period; a candidate learning period division process for dividing the candidate learning period into a plurality of divided learning periods; a learning period evaluation process that generates a learning period evaluation model for each divided learning period by performing machine learning of the learning model for each divided learning period using the learning data corresponding to the divided learning period, and calculates an evaluation index of the learning period evaluation model for each divided learning period by verifying the learning period evaluation model for each divided learning period using the verification data; a learning period selection process for repeatedly performing the candidate learning period division process and the candidate learning period evaluation process by setting the divided learning period with a high evaluation index as the next candidate learning period until a predetermined termination condition is met, and selecting the divided learning period with a high evaluation index as the target learning period when the termination condition is met; a model output process for outputting, as a trained model, the learning model that has been subjected to machine learning using the learning data corresponding to the learning target period; Machine learning methods.

2. A machine learning method for performing machine learning of a learning model by a computer using a dataset configured by arranging data at each point in time acquired in accordance with a time-dependent change in a predetermined phenomenon over a predetermined period in a chronological order, the method comprising: a dataset division process that divides the period into a learning period and a validation period, sets the learning period as an initial candidate learning period, and divides the dataset into learning data corresponding to the learning period and validation data corresponding to the validation period; a learning data division process for dividing the learning period into a plurality of sub-learning periods and dividing the learning data into a plurality of sub-learning data corresponding to the plurality of sub-learning periods; a candidate learning period division process for dividing the plurality of sub-learning periods included in the candidate learning period into a plurality of divided learning periods; a learning period evaluation process that generates a learning period evaluation model for each divided learning period by performing machine learning of the learning model for each divided learning period using the sub-learning data corresponding to the sub-learning period included in the divided learning period, and calculates an evaluation index of the learning period evaluation model for each divided learning period by verifying the learning period evaluation model for each divided learning period using the verification data; a learning period selection process for repeatedly performing the candidate learning period division process and the candidate learning period evaluation process by setting the divided learning period with a high evaluation index as the next candidate learning period until a predetermined termination condition is met, and selecting, when the termination condition is met, the sub-learning period included in the divided learning period with a high evaluation index as a target learning period; a model output process of outputting, as a trained model, the learning model that has been subjected to machine learning using the sub-learning data corresponding to the learning target period; Machine learning methods.

3. a learning data division process for dividing the learning period excluding the learning target period into a plurality of sub-learning periods, and dividing the learning data excluding the learning data corresponding to the learning target period into a plurality of sub-learning data corresponding to the plurality of sub-learning periods; The trained model is verified using each of the plurality of sub-learning data, and an evaluation index of the trained model is calculated for each of the sub-learning periods, and the trained model and further performing an additional period selection process to select the sub-learning period in which the evaluation index satisfies a predetermined additional condition as an additional target period; The model output process includes: outputting, as a trained model, the learning model that has been subjected to machine learning using the learning data corresponding to at least one of the learning target period and the additional target period; The machine learning method of claim 1 .

4. calculating an evaluation index of the trained model for each sub-learning period by verifying the trained model using each of the plurality of sub-learning data excluding the sub-learning data corresponding to the training target period, and further performing an additional period selection process of selecting, as an additional target period, the sub-learning period in which the evaluation index of the trained model satisfies a predetermined additional condition; The model output process includes: outputting, as a trained model, the learning model on which machine learning has been performed using the sub-learning data corresponding to at least one of the learning target period and the additional target period; The machine learning method of claim 2 .

5. The candidate learning period division process includes: dividing the candidate learning period into a plurality of divided learning periods such that each of the divided learning periods includes a discrete period in the time series of the candidate learning period; The machine learning method of claim 1 .

6. The candidate learning period division process includes: randomly dividing the time series of the plurality of sub-learning periods into a plurality of divided learning periods so that each of the plurality of divided learning periods includes a discrete period in the time series of the candidate learning period; The machine learning method of claim 2 .

7. The data set division process includes: When dividing the period into the learning period and the verification period, the verification period is set discretely in the time series of the period. The machine learning method according to any one of claims 1 to 6.

8. A machine learning device including a machine learning unit that performs machine learning of a learning model using a dataset configured by arranging data at each point in time acquired in accordance with a time-dependent change in a predetermined phenomenon over a predetermined period in a chronological order, The machine learning unit a dataset division process that divides the period into a learning period and a validation period, sets the learning period as an initial candidate learning period, and divides the dataset into learning data corresponding to the learning period and validation data corresponding to the validation period; a candidate learning period division process for dividing the candidate learning period into a plurality of divided learning periods; a learning period evaluation process that generates a learning period evaluation model for each divided learning period by performing machine learning of the learning model for each divided learning period using the learning data corresponding to the divided learning period, and calculates an evaluation index of the learning period evaluation model for each divided learning period by verifying the learning period evaluation model for each divided learning period using the verification data; a learning period selection process for repeatedly performing the candidate learning period division process and the candidate learning period evaluation process by setting the divided learning period with a high evaluation index as the next candidate learning period until a predetermined termination condition is met, and selecting the divided learning period with a high evaluation index as the target learning period when the termination condition is met; a learning model in which machine learning is performed using the learning data corresponding to the learning period; and performing a model output process to output the model as a trained model. Machine learning device.

9. A machine learning device including a machine learning unit that performs machine learning of a learning model using a dataset configured by arranging data at each point in time acquired in accordance with a time-dependent change in a predetermined phenomenon over a predetermined period in a chronological order, The machine learning unit a dataset division process that divides the period into a learning period and a validation period, sets the learning period as an initial candidate learning period, and divides the dataset into learning data corresponding to the learning period and validation data corresponding to the validation period; a learning data division process for dividing the learning period into a plurality of sub-learning periods and dividing the learning data into a plurality of sub-learning data corresponding to the plurality of sub-learning periods; a candidate learning period division process for dividing the plurality of sub-learning periods included in the candidate learning period into a plurality of divided learning periods; a learning period evaluation process that generates a learning period evaluation model for each divided learning period by performing machine learning of the learning model for each divided learning period using the sub-learning data corresponding to the sub-learning period included in the divided learning period, and calculates an evaluation index of the learning period evaluation model for each divided learning period by verifying the learning period evaluation model for each divided learning period using the verification data; a learning period selection process for repeatedly performing the candidate learning period division process and the candidate learning period evaluation process by setting the divided learning period with a high evaluation index as the next candidate learning period until a predetermined termination condition is met, and selecting, when the termination condition is met, the sub-learning period included in the divided learning period with a high evaluation index as a target learning period; a model output process of outputting, as a trained model, the learning model that has been subjected to machine learning using the sub-learning data corresponding to the learning target period; Machine learning device.

Images