Detection device, detection program, and detection method

Abstract

This technology provides the ability to automatically set detection thresholds using a common method for operational data acquired from various devices, thereby suppressing unnecessary retraining and enabling the detection of concept drift. [Solution] The control unit of the detection device inputs a set of test data acquired by the first model to calculate a set of first output values, calculates the first mean and first standard deviation of the set of first output values, inputs a predetermined number of acquired operational data to the first or second model to calculate a second mean by averaging a predetermined number of second output values, calculates the second standard deviation of the second mean from the first standard deviation and a predetermined number, sets a drift threshold from the first mean and the second standard deviation, and detects concept drift based on the fact that the calculated second mean is greater than or equal to the set drift threshold.

Description

Technical Field

[0001] The present disclosure relates to a detection method and the like.

Background Art

[0002] Machine learning consists of a learning phase in which learning data is analyzed to learn a learning model and a learned learning model is generated, and an operation phase in which operation data generated after learning is input to the learned learning model to infer a result. Here, due to changes in temperature over time and aging of equipment, a concept drift may occur in which the tendency of operation data changes from the tendency of learning data. The occurrence of such a concept drift may reduce the inference accuracy of the learning model.

[0003] The display method described in Patent Document 1 generates a replica model by changing the parameters of an original model generated from learning data. Then, when the difference between the distribution of output values when learning data is input to the replica model and the distribution of output values when operation data is input to the replica model is greater than a threshold value, a concept drift is detected.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the above display method, when the distribution of output values does not follow a normal distribution, a threshold value for realizing appropriate detection sensitivity is not set, and a concept drift is misdetected and unnecessary relearning is performed, or the concept drift is overlooked.

[0006] One aspect of this disclosure is the provision of a technology that can suppress false detection and oversight of concept drift. [Means for solving the problem]

[0007] One aspect detection device of this disclosure includes a control unit, which acquires a set of test data, which is time-series data, from the equipment, inputs the acquired set of test data into a trained first model to calculate a set of first output values, calculates a first mean, which is the average value of the calculated set of first output values, and a first standard deviation, which is the standard deviation of the set of first output values, acquires a predetermined number of operational data, which is time-series data later than the test data, inputs the acquired predetermined number of operational data into the first model or a trained second model of the same type as the first model, averages a predetermined number of second output values ​​obtained to calculate a second mean, calculates a second standard deviation, which is the standard deviation of the second mean, from the first standard deviation and a predetermined number, sets a drift threshold from the first mean and the second standard deviation, and detects concept drift based on the fact that the calculated second mean is greater than or equal to the set drift threshold.

[0008] According to one aspect of the detection device of this disclosure, the first mean and first standard deviation of a set of first output values ​​are calculated, and the second standard deviation is calculated from the first standard deviation and a predetermined number. In addition, a second mean is calculated by averaging a predetermined number of second output values ​​obtained by inputting a predetermined number of operational data into a second model. According to the central limit theorem, if the distribution of the second output values ​​and the distribution of the first output values ​​can be considered identical, and the predetermined number is sufficiently large, then regardless of the distribution of the set of first output values, the second mean will follow a normal distribution of the first mean and the second standard deviation. Therefore, a drift threshold that achieves appropriate detection sensitivity can be automatically set based on the first mean and the second standard deviation. In other words, a drift threshold can be automatically set using a common method for operational data acquired from various devices, thereby suppressing false detections and missed detections of concept drift.

[0009] A detection program for another aspect of this disclosure causes a processing unit to perform the following actions: acquire a set of test data, which is time-series data, from the equipment; input the acquired set of test data into a trained first model to calculate a set of first output values; calculate a first mean, which is the average value of the calculated set of first output values, and a first standard deviation, which is the standard deviation of the set of first output values; acquire a predetermined number of operational data, which is time-series data later than the test data, from the equipment; input the acquired predetermined number of operational data into a first model or a trained second model of the same type as the first model to calculate a second mean by averaging a predetermined number of second output values ​​obtained; calculate a second standard deviation, which is the standard deviation of the second mean from the first standard deviation and a predetermined number; set a drift threshold from the first mean and the second standard deviation; and detect the occurrence of concept drift based on the fact that the calculated second mean is greater than or equal to the set drift threshold.

[0010] The same effect as the above-described detection device is achieved by the processing device executing a detection program for another aspect of the present disclosure.

[0011] A detection method for yet another aspect of this disclosure is performed by a processing device and comprises: acquiring a set of test data, which are time-series data, from a device; inputting the acquired set of test data into a trained first model to calculate a set of first output values; calculating a first mean, which is the average value of the calculated set of first output values, and a first standard deviation, which is the standard deviation of the set of first output values; acquiring a predetermined number of operational data, which are time-series data later than the test data, from a device; inputting the acquired predetermined number of operational data into a first model or a trained second model of the same type as the first model to calculate a second mean by averaging a predetermined number of second output values ​​obtained; calculating a second standard deviation, which is the standard deviation of the second mean from the first standard deviation and a predetermined number; setting a drift threshold from the first mean and the second standard deviation; and detecting the occurrence of concept drift based on the fact that the calculated second mean is greater than or equal to the set drift threshold.

[0012] The processing device achieves the same effect as the above-described detection device by executing a detection method for yet another aspect of the present disclosure.

[0013] Another aspect of this disclosure is the desirable ability to detect equipment anomalies, distinct from concept drift. A detection device of yet another aspect of the present disclosure comprises a control unit which acquires operational data input to a model generated by learning training data from equipment, detects concept drift of the operational data relative to the training data, retrains the model if concept drift is detected after a set time has elapsed since the last time concept drift was detected, and detects that there is no concept drift if concept drift is detected within a set time since the last time concept drift was detected.

[0014] According to a detection device in yet another aspect of this disclosure, if concept drift is detected again after a certain period of time has elapsed since the initial detection, the model is retrained. If concept drift is detected again within that period, it is detected that it is not concept drift. When a phenomenon is caused by concept drift, the operational data changes slowly over a long period of time compared to when it is caused by something other than concept drift. Therefore, if concept drift occurs, retraining the model and adapting it to the drifted operational data makes it unlikely that concept drift will be detected again for a while. Thus, if concept drift is detected multiple times within a certain period of time, false detection of concept drift can be avoided by detecting that it is not concept drift. Furthermore, in such cases, equipment abnormalities can be detected. [Brief explanation of the drawing]

[0015] [Figure 1] This is a diagram illustrating the status monitoring system according to the embodiment. [Figure 2] This diagram shows the configuration of the detection device according to the embodiment. [Figure 3]This is a diagram for explaining the processing flow of the detection device according to the embodiment. [Figure 4] This is a diagram showing the distribution of operation data and the operation data classified as normal and abnormal by the learning model. [Figure 5] This is a diagram showing the variation of the state monitoring data of the device along with the change of seasons. [Figure 6] This is a diagram showing the variation of the output value of the learning model associated with concept drift and the variation of the output value of the learning model associated with mechanical abnormality. [Figure 7] This is a diagram showing the change in the distribution of the output value of the learning model associated with concept drift. [Figure 8] This is a diagram showing the distribution of the output value before and after the drift of the operation data, and the drift threshold. [Figure 9] This is a diagram showing the distribution of the output value and the drift threshold when the output value is a normal distribution and when the output value is not a normal distribution. [Figure 10] Figure 10A is a diagram showing a scatter plot of the population of the output value. Figure 10B is a diagram showing the frequency distribution of the population of the output value. Figure 10C is a diagram showing the frequency distribution of the sample mean with a sample size of 9. Figure 10D is a diagram showing the frequency distribution of the sample mean with a sample size of 18. Figure 10E is a diagram showing the frequency distribution of the sample mean with an output value of 30 sample sizes. [Figure 11] This is a diagram showing how the distribution of the sample mean with the population of the output value varies along with the variation of the operation data according to the embodiment. [Figure 12] This is a diagram showing the detection process executed by the detection device according to the embodiment. [Figure 13] This is a diagram showing the details of the detection process executed by the detection device according to the embodiment. [Figure 14] This is a diagram for explaining the relearning of the first model for calculating the drift threshold and the second model for actual operation along with the detection of concept drift according to the embodiment. [Figure 15] Figure 15A is a diagram showing an example of the distribution of the output value. Figure 15B is a diagram showing the distribution obtained by removing the outliers from the distribution of the output value shown in Figure 15A. [Figure 16] It is a diagram showing the temporal change of the output value of the learning model according to the embodiment.

Mode for Carrying Out the Invention

[0016] (1. First Embodiment) <1-1. Configuration of Detection System> Referring to FIGS. 1 and 2, the configuration of the state monitoring system 100 according to the present embodiment will be described. The state monitoring system 100 includes a detection device 10, a terminal device 20, a first device 30, a second device 40, a third device 50, a fourth device 60, and a communication network 70. The state monitoring system 100 monitors and centrally manages the states of a plurality of types of devices.

[0017] The communication network 70 connects the detection device 10, the terminal device 20, the first device 30, the second device 40, the third device 50, and the fourth device 60 to each other. The communication network 70 is a wireless communication network, a wired communication network, or a network in which a wireless communication network and a wired communication network are mixed.

[0018] The first device 30, the second device 40, the third device 50, and the fourth device 60 are devices to be monitored, and are, for example, various mechanical equipment. The first device 30 is, for example, a platform movable gate installed on the platform of a train. The second device 40 is, for example, an elevator. The third device 50 is, for example, a snow melting device installed on a train line or a station. The fourth device 60 is, for example, a bogie assembly device for railway vehicle maintenance. The first device 30, the second device 40, the third device 50, and the fourth device 60 transmit state monitoring data to the detection device 10 via the communication network 70.

[0019] The state monitoring data is time-series data and includes measurement data of the device measured by a sensor and control data for controlling the drive of the device. The measurement data and the control data include current values, voltage values, torque values, etc. for driving a drive unit (for example, an actuator included in the device).

[0020] The status monitoring system 100 may include one or more first devices 30, one or more second devices 40, one or more third devices 50, and one or more fourth devices 60 as monitoring targets. Furthermore, the status monitoring system 100 does not have to include four types of devices as monitoring targets; it may include one to three types of devices, or five or more types of devices.

[0021] The terminal device 20 is a portable terminal device such as a tablet or notebook PC. The terminal device 20 may be used by workers on-site (i.e., at the location where the equipment is installed). The terminal device 20 displays the analysis results of the equipment by the detection device 10, which will be described later.

[0022] The detection device 10 is a server device, a PC, etc., and is located in a facility such as an office. As shown in Figure 2, the detection device 10 comprises a control unit 11, a storage unit 13, and a communication unit 15. The communication unit 15 receives status monitoring data from each device via a communication network 70. The storage unit 13 stores the status monitoring data received by the communication unit 15 and the learning model generated by the control unit 11.

[0023] The control unit 11 is a microcomputer equipped with a CPU 111, ROM 112, RAM 113, and I / O. The control unit 11 learns status monitoring data and generates a learning model by having the CPU 111 execute a program stored in the ROM 112, and then uses the learning model to analyze the status of each device. For example, the control unit 11 quantifies the degree of abnormality and remaining lifespan of various devices, and sets priorities for inspections and other tasks that take into account the importance of the devices.

[0024] <1-2. Learning Phase> As shown in Figure 3, in the learning phase, the control unit 11 uses the status monitoring data received from the first device 30 as learning data to perform machine learning and generate a learning model for detecting abnormalities in the first device 30 or their precursors. The status monitoring data is, for example, time-series data of the torque values ​​of the actuators of the platform movable fence. In this embodiment, the control unit 11 calculates a learning model and a drift threshold for detecting concept drift, described later, individually for each device using unsupervised learning. The control unit 11 also calculates learning models and drift thresholds for the second device 40, the third device 50, and the fourth device 60.

[0025] If the status monitoring system 100 has multiple first devices 30, it is conceivable to generate a common learning model for the multiple first devices 30. However, if the first devices 30 are general-purpose products, it is difficult to deductively derive the drift threshold from the specifications of the first devices 30. Furthermore, even if the devices are of the same type, they are installed in various locations such as outdoors and indoors, so the external environment differs for each, and the status monitoring data also varies greatly. Therefore, if the drift threshold is standardized, there is a possibility of false detection or overlooking concept drift. Moreover, if the number of devices of the same type is small, there are also few past anomaly cases, making it difficult to derive the drift threshold from past cases. In particular, the number of important devices is small, and there are few past anomaly cases. Also, even if past anomaly cases exist, they are diverse. Therefore, it is difficult to generate a comprehensive learning model for devices of the same type and standardize the drift threshold.

[0026] Therefore, the control unit 11 does not standardize the learning model and drift threshold by device type, but calculates the learning model and drift threshold individually for each device. However, the control unit 11 calculates the drift threshold for a wide variety of devices using the same method. That is, the control unit 11 generates drift thresholds for all of the first device 30, second device 40, third device 50, and fourth device 60 using the same method.

[0027] Furthermore, the control unit 11 performs unsupervised learning without defining an "abnormal" state (i.e., training data) based on a small number of past abnormal cases. In detail, as shown in Figure 4, the control unit 11 generates a learning model that classifies the learning data, which is time-series data, based on the distance between data points, considers a set of data that falls within a certain range as normal data, and considers data that falls outside the set of normal data as abnormal data. The learning model may quantify and output the degree of abnormality of the input data input to the learning model based on the degree of deviation from the set of normal data. For example, the learning model may quantify the degree of abnormality based on the Euclidean distance from the normal data group or the general Mahalanobis distance, or it may quantify the degree of abnormality from the probability of occurrence on a probability distribution.

[0028] <1-3. Operational Phase> Next, as shown in Figure 3, during the operation phase, the control unit 11 inputs the status monitoring data received from the first device 30 as operation data into the learning model generated for the first device 30. Then, the control unit 11 detects whether or not there is an abnormality in the first device 30 based on the output value (i.e., the abnormality value) of the learning model. An abnormality here refers to equipment abnormalities that require repair or replacement, such as actuator damage or the breaking or detachment of parts.

[0029] As shown in Figure 5, the trend in condition monitoring data may fluctuate due to changes in equipment or sensors, or changes in the external environment. Changes in equipment or sensors include aging, maintenance, and parts replacement. Changes due to changes in the external environment include thermal expansion or contraction of mechanical parts due to daily or yearly temperature changes, and annual changes in lubricating oil. For example, as shown in Figure 5, if the condition monitoring data is time-series data of torque values, in winter when temperatures are low, the distribution of condition monitoring data will fluctuate more towards the side with larger values ​​due to thermal contraction of mechanical parts and decrease in lubricating oil viscosity compared to summer when temperatures are high.

[0030] Such underlying fluctuations in state monitoring data are referred to as concept drift, data drift, covariate drift, etc. In the following, underlying fluctuations in state monitoring data will be referred to as concept drift. When concept drift occurs, the inference accuracy of the learning model decreases as the distribution of operational data deviates from the distribution of training data. Therefore, the output value of the learning model fluctuates in accordance with the concept drift of the operational data. As shown in Figure 6, when concept drift occurs in operational data, the anomaly score does not rise sharply as it would when an equipment malfunction occurs, but the anomaly score gradually increases in accordance with the concept drift.

[0031] To suppress the decrease in inference accuracy of a learning model caused by concept drift, it is conceivable to incorporate the factors causing concept drift as features into the learning model. However, for example, if the temperature change over one year is incorporated as a feature into the learning model, the relationship between at least one year's worth of temperature data and various physical quantities must be learned, meaning that a learning model cannot be generated for equipment that has not been in operation for one year. Furthermore, the factors causing concept drift are diverse, and some factors are difficult to measure or quantify. In addition, because there are differences in aging degradation for each piece of equipment and temporary recovery due to maintenance, it is difficult to model them using the same algorithm.

[0032] As shown in Figure 3, the control unit 11 automatically detects the concept drift of operational data relative to the training data, retrains the model, and automatically replaces the training model. That is, when the control unit 11 detects concept drift, it retrains the training model using the changed training data, causing the training model to track the state monitoring data in which the concept drift occurred. The control unit 11 also identifies and detects concept drift in operational data and equipment abnormalities based on the output values ​​of the training model. Details of the method for automatically detecting concept drift will be described later.

[0033] <1-3-1. Detection of Concept Drift> Next, we will explain the method for detecting concept drift by the control unit 11. One possible method for detecting concept drift is to focus on the output side of the learning model to detect concept drift.

[0034] When detecting concept drift by focusing on the output side of a learning model, it is sufficient to detect concept drift when the trend of the output values ​​changes or when the error rate of the output values ​​increases. Methods for determining changes in the trend of output values ​​include methods that monitor changes in the distribution of multiple output values ​​and methods that monitor changes in individual output values.

[0035] As shown in Figure 7, a method for monitoring changes in the distribution of multiple output values ​​is to detect concept drift when the distribution of output values ​​changes and recent output values ​​deviate from the distribution of past output values. In this embodiment, the control unit 11 employs a method for monitoring changes in the distribution of multiple output values. When the amount of change in the distribution of output values ​​exceeds a set drift threshold, the control unit 11 detects concept drift and retrains the learning model. Details of setting the drift threshold will be described later.

[0036] A method for monitoring changes in individual output values ​​involves comparing time-series data of output values ​​before and after the point in time to be verified, and detecting concept drift if the time-series data before and after has changed. However, even if there is a difference in the time-series data before and after, that difference may not affect the accuracy of the learning model. In such cases, retraining the learning model after detecting concept drift would be a waste of time. Therefore, in this embodiment, the control unit 11 does not employ a method for monitoring changes in individual output values, but rather employs a method for monitoring changes in the distribution of output values ​​of the learning model.

[0037] Furthermore, when detecting concept drift based on an increase in the error rate of output values, a human pre-sorts the state monitoring data into normal and abnormal values ​​and inputs the sorted state monitoring data into the learning model. Then, the output values ​​are compared with the sorting results by the human, and concept drift can be detected if the error rate of the output values ​​exceeds a predetermined threshold. However, in this case, a human needs to sort the state monitoring data for each device, and concept drift cannot be detected automatically. Therefore, in this embodiment, the control unit 11 does not employ a method of monitoring an increase in the error rate of output values, but rather employs a method of monitoring changes in the distribution of output values ​​of the learning model.

[0038] Another possible method for detecting concept drift is to focus on the input side of the learning model. When detecting concept drift by focusing on the input side of the learning model, one can compare the time-series state monitoring data input to the learning model before and after the point in time to be verified, and detect concept drift if the state monitoring data has changed. However, if the state monitoring data contains a large number of features, comparing high-dimensional time-series data becomes computationally intensive and difficult. Also, even if data changes are detected by comparing data related to some of the many features before and after the verification point, the change in data related to those features may not affect the accuracy of the learning model's inference. Therefore, in this embodiment, the control unit 11 does not employ a method to monitor the input side of the learning model, but rather employs a method to monitor changes in the distribution of the learning model's output values.

[0039] Another possible method for detecting concept drift is to focus on the learning model itself. When detecting concept drift by focusing on the learning model itself, two learning models can be generated using time-series state monitoring data before and after the point in time to be verified as training data, and concept drift can be detected if there is a difference between the two learning models. However, in this case, it is difficult to quantify and compare the characteristics of the learning models obtained by unsupervised learning, or to evaluate whether the difference between the two learning models is significant. Therefore, in this embodiment, the control unit 11 does not employ a method to monitor the learning model itself, but rather employs a method to monitor changes in the distribution of the output values ​​of the learning model.

[0040] <1-3-2. Setting the drift threshold> Next, we will explain how to set the drift threshold. As shown in Figure 8, whether the recent output value deviates from the distribution of past output values ​​can be determined by comparing the difference between the first distribution (the distribution of past output values) and the second distribution (the distribution of recent output values) with the drift threshold. Since the output values ​​of the learning model are scalars, for the sake of simplifying and automating the process, it is conceivable to use the difference between the first mean A of the first distribution and the second mean B of the second distribution as the difference between the two distributions. The drift threshold can then be set to A+3σ, which is the value obtained by adding three times the standard deviation σ of the first distribution to the first mean A of the first distribution, based on the 3σ method.

[0041] As shown in Figure 9, if the output values ​​follow a normal distribution, it is possible to detect a change in the distribution of the output values, i.e., a conceptual drift in the operational data, based on the output values ​​exceeding the drift threshold once or multiple times. However, the output values ​​do not necessarily follow a normal distribution.

[0042] If the output values ​​do not follow a normal distribution, the probability of the output values ​​being above A+3σ is approximately 5.5% at most. When the output values ​​follow a distribution with a wide range of outliers, the sensitivity to detecting concept drift becomes excessively high, leading to false detection of concept drift with even slight fluctuations in operational data, and frequent unnecessary retraining of the learning model. On the other hand, as shown in Figure 9, when the output values ​​follow a distribution with a low probability of being above A+3σ, the sensitivity to detecting concept drift becomes excessively low, potentially leading to missed concept drift even with fluctuations in operational data. However, manually setting the drift threshold for each device to ensure appropriate sensitivity to concept drift according to the shape of the output value distribution requires considerable effort. It is desirable to be able to set the drift threshold universally and automatically using a common method for a wide variety of devices.

[0043] Therefore, in this embodiment, the control unit 11 does not directly set the drift threshold from the population distribution of the output values ​​of the learning model, but rather sets the drift threshold from the distribution of the sample population generated from the population of output values.

[0044] According to the central limit theorem, if we repeatedly draw sample sets of n samples from the population, each set having a different sample, and average the n samples in each set, the resulting sample population will approach a normal distribution if n is sufficiently large, regardless of the distribution of the population.

[0045] Figure 10A shows an example of a scatter plot of the population of output values, and Figure 10B shows an example of a frequency distribution of the population of output values. Figure 10C shows the distribution of the sample population when 9 output values ​​are used as one sample set, Figure 10D shows the distribution of the sample population when 18 output values ​​are used as one sample set, and Figure 10E shows the distribution of the sample population when 30 output values ​​are used as one sample set. As shown in Figures 10C to 10E, the larger the sample size, the closer the distribution of the sample population approaches a normal distribution, and when the sample size reaches 30, the distribution of the sample population can be approximated by a normal distribution.

[0046] Therefore, the control unit 11 inputs time-series state monitoring data from a predetermined past period into the learning model and calculates the first mean μ, which is the average value of the set of output first output values, and the first standard deviation σ, which is the standard deviation. Furthermore, the control unit 11 acquires n state monitoring data from the current time-series state monitoring data and inputs them into the learning model, and calculates the second mean by averaging the n output second output values. If the distribution of the first output values ​​and the distribution of the second output values ​​are the same and the sample size n is sufficiently large, the second mean, which is the sample mean, follows a normal distribution of (μ,S). The mean μ of the above normal distribution is equal to the first mean μ. The second standard deviation S, which is the standard deviation of the above normal distribution, is σ 2 It is the square root of / n. Therefore, the control unit 11 calculates the second standard deviation S from the first standard deviation σ and the number of samples n, and sets the drift threshold based on the first mean μ and the second standard deviation S. Specifically, the control unit 11 sets the drift threshold to μ + αS. α is a predetermined value, for example, 3, 5, 10, etc. The larger α is, the lower the sensitivity to detecting concept drift. If the occurrence rate of underlying fluctuations in the state monitoring data is relatively low, it is advisable to set α to a relatively large value, and if the occurrence rate of underlying fluctuations in the state monitoring data is relatively high, it is advisable to set α to a relatively small value.

[0047] As shown in Figure 11, the control unit 11 compares a drift threshold set based on the set of first output values ​​during training with a second mean value, which is the sample mean calculated from n second output values ​​corresponding to operational data over a predetermined period. If the second mean value exceeds the drift threshold, the control unit 11 determines that the likelihood of the second mean value being calculated from the same set of values ​​as the first output values ​​during training is low. In other words, the control unit 11 determines that the trend in the distribution of the second output values ​​has changed from the distribution of the first output values, and retrains the learning model.

[0048] <1-3-3. Equipment Anomaly Detection> Next, the method for detecting equipment abnormalities by the control unit 11 will be explained. The control unit 11 detects equipment abnormalities if concept drift is detected again within a certain period of time after concept drift in the operational data is detected and retraining is performed. As shown in Figure 6, when equipment abnormalities occur, the state monitoring data increases rapidly in a short period of time compared to concept drift. Therefore, even if the learning model is retrained, there is a high possibility that the second mean value will soon exceed the drift threshold. On the other hand, when concept drift occurs, the state monitoring data changes slowly over a long period of time. Therefore, if the learning model is retrained and adapted to the changed state monitoring data, there is a low possibility that the second mean value will exceed the drift threshold for a while.

[0049] Therefore, if the control unit 11 detects concept drift again after a certain period of time has elapsed since concept drift was detected and the learning model was retrained, it determines that concept drift has occurred and retrains the learning model. On the other hand, if the control unit 11 detects concept drift again within a certain period of time since concept drift was detected and the learning model was retrained, it determines that it is not concept drift and does not retrain the learning model. Specifically in this embodiment, the control unit 11 detects an abnormality in the equipment and does not retrain the learning model.

[0050] However, as described above, when detecting an abnormality in the equipment, the control unit 11 determines the first time the second mean value exceeds the drift threshold to be a concept drift and retrains, using the state monitoring data in which the abnormality occurred. Therefore, the detection of an abnormality by the control unit 11 may be delayed. To address this, the control unit 11 may set an abnormality threshold that is larger than the drift threshold, in addition to the drift threshold, and immediately detect an abnormality in the equipment if the second mean value exceeds the abnormality threshold even once.

[0051] <1-3-4. Detection Process> Next, the detection process executed by the control unit 11 will be described based on Figures 12 and 13. This detection process is repeatedly executed by the control unit 11 at a fixed interval (for example, once a week). In this embodiment, a learning model for calculating the drift threshold (hereinafter referred to as the first learning model) and a learning model for actual operation (hereinafter referred to as the second learning model) are generated. The second learning model is a model of the same type as the first learning model, trained on training data using the same algorithm. As shown in Figure 14, the training data used to train the second learning model has a different range from the training data used to train the first learning model.

[0052] The first learning model is generated by training it using the most recent 5% of the state monitoring data from the past year from the training point as test data, and the remaining 95% of the state monitoring data (excluding the test data) as the first training dataset. Then, the first mean μ and first standard deviation σ of the set of first output values ​​output by inputting the test data into the first learning model are calculated. Furthermore, the second standard deviation S is calculated from the first standard deviation σ and the number of samples n of the second output values, and a drift threshold is set based on the first mean μ and the second standard deviation S. On the other hand, the second learning model is generated by training it using the state monitoring data from the past year from the training point as the second training dataset. In other words, the second learning model is generated using more recent training data than the first learning model. In another embodiment, the first learning model may be generated using the state monitoring data from the past year from the point where the most recent 5% was excluded. In other words, the first learning model may be generated using the same amount of training data as the second learning model. Also, 5% is just an example, and the test data may not be limited to 5% but may be several percent to 10%. A larger test dataset is desirable because it better matches the distribution of the first training dataset; for example, 1000 or more data points are preferable.

[0053] First, in S10, the control unit 11 inputs the M state monitoring data (i.e., operation data) received between the previous processing cycle and the current processing cycle into the second learning model.

[0054] In S20, the control unit 11 inputs M state monitoring data into the second learning model and calculates M second output values ​​(specifically, abnormality values).

[0055] Next, in S30, the control unit 11 utilizes the central limit theorem to obtain M / n sample sets, each set consisting of n values, from the M second output values ​​calculated in S20. Then, the control unit 11 calculates the second mean, which is the sample mean, by averaging the n second output values ​​of each sample set.

[0056] Next, in S40, the control unit 11 determines whether each of the M / n second mean values ​​calculated in S30 is greater than or equal to the set drift threshold. If it determines that (M / nk) of the M / n second mean values ​​are less than the drift threshold, the process returns to S10. k is a natural number, for example, 1 or 2. Then, the M state monitoring data received during the next processing cycle are input to the second learning model. In S40, if the control unit 11 determines that the k second mean values ​​are greater than or equal to the drift threshold, the process proceeds to S50.

[0057] In S50, the control unit 11 determines whether the number of days elapsed since the last training of the first and second learning models is less than the set number of days. The set number of days is, for example, 14 days. If it determines that the number of days elapsed is equal to or greater than the set number of days, the process proceeds to S60, where concept drift is detected and the learning model is retrained.

[0058] The control unit 11 executes the processes S70 to S90 during the retraining of the learning model. In S70, the control unit 11 retrains the first learning model and the second learning model. Specifically, in S710, the learning dataset for the most recent year is divided into 5% test data and the remaining 95% as the first learning dataset. Furthermore, the learning dataset for the most recent year is made the second learning dataset. Then, in S720, the control unit 11 trains the first learning dataset to generate the first learning model, and trains the second learning dataset to generate the second learning model.

[0059] In S80, the control unit 11 inputs test data into the first learning model and creates an anomaly distribution, which is the distribution of the first output values.

[0060] In S90, the control unit 11 sets the drift threshold based on the anomaly distribution created in S80 and the central limit theorem. Specifically, as shown in Figures 15A and 15B, in S910, outliers are removed from the anomaly distribution created in S80. State monitoring data may contain large outliers. If the first mean value μ is calculated while including such outliers, the first mean value μ may be greatly influenced by the outliers. Consequently, the drift threshold may be set excessively high, potentially causing concept drift to be missed. Therefore, 5% of the anomaly values ​​with the highest anomaly values ​​are removed from the total number of anomaly values, leaving 95%. Note that the anomaly values ​​to be removed are not limited to 5%, but may range from a few percent to 10%.

[0061] Next, in S920, the control unit 11 calculates the first mean μ and first standard deviation σ of the anomaly distribution from which outliers have been removed.

[0062] Next, in S930, the second standard deviation S is calculated from the first standard deviation σ and the sample size n calculated in S920, and the drift threshold μ+αS is set based on the first mean μ and the second standard deviation S. Then, the process proceeds to S40, where it is determined whether the M / n second mean values ​​are greater than or equal to the drift threshold.

[0063] Furthermore, if the control unit 11 determines in S50 that the elapsed time is less than the set number of days, it detects that this is not concept drift, specifically that there is an abnormality in the equipment. The control unit 11 terminates this process without retraining the learning model.

[0064] Figure 16 shows the time progression of the anomaly score when the control unit 11 performs the detection process described above. Between the start of operation of the equipment and time point t1, concept drift is detected multiple times, and each time, the first and second learning models are retrained. Then, at time point t1, the anomaly score rises sharply, and an anomaly in the equipment is detected.

[0065] <1-4. Effects> The first embodiment described in detail above provides the following effects. (1) Test data is input to the first learning model, and the first mean μ and first standard deviation σ of the set of first output values ​​output are calculated. The second standard deviation S is calculated from the first standard deviation σ and the number of samples n. In addition, the second mean is calculated by averaging the n second output values ​​obtained by inputting n operational data into the second learning model. As shown by the central theorem, if the distribution of the second output values ​​and the distribution of the first output values ​​can be considered identical, and the number of samples n is sufficiently large, then regardless of the distribution of the set of first output values, the second mean will follow a normal distribution of the first mean and the second standard deviation. Therefore, based on the first mean μ and the second standard deviation S, a drift threshold that achieves appropriate detection can be automatically set. In other words, a common method can be used to automatically set a drift threshold for operational data acquired from various devices, thereby suppressing false detections and missed detections of concept drift.

[0066] (2) The set of first output values ​​of the first learning model is updated to a set from which a predetermined number of outliers have been removed. The first mean μ and first standard deviation σ of the updated set of first output values ​​are then calculated. This reduces the influence of outliers on the first mean μ. Consequently, it is possible to prevent the drift threshold from being set too high due to the influence of outliers and to avoid missing concept drift.

[0067] (3) If concept drift is detected again after a certain period of time has elapsed since the initial detection, the first and second learning models will be retrained. If concept drift is detected again within that period of time, the first and second learning models will not be retrained, and an abnormality in the device will be detected. By detecting an abnormality in the device when concept drift is detected multiple times within a certain period of time, it is possible to avoid the first and second learning models being retrained and the device adapting to the abnormal state.

[0068] (4) By generating a second learning model for practical use using training data that includes more recent state monitoring data than the first learning model used for calculating the drift threshold, the inference accuracy of the second learning model can be improved.

[0069] (5) Concept drift is detected based on the fact that k sample means out of the most recent M / n sample means are above the drift threshold, thereby suppressing the false detection of temporary noise as concept drift and improving the accuracy of concept drift detection.

[0070] (6) If an abnormality threshold is set to be greater than the drift threshold to detect equipment abnormalities, the equipment abnormality can be detected immediately.

[0071] (Second Embodiment) <2-1. Differences from the First Embodiment> The second embodiment has the same basic configuration as the first embodiment, so the differences will be explained below. Note that the same reference numerals as in the first embodiment indicate the same components, and refer to the preceding description.

[0072] In the first embodiment described above, a drift threshold was set based on the sample distribution, and concept drift was detected when the sample mean exceeded the drift threshold. In contrast, the second embodiment differs from the first embodiment in that the method for detecting concept drift is not limited.

[0073] In this embodiment, the control unit 11 detects concept drift using any method. That is, the control unit 11 may detect concept drift by focusing on the input side of the learning model, or by focusing on the learning model itself. Alternatively, the control unit 11 may focus on the output side of the learning model and detect concept drift by individually comparing the output values ​​before and after the verification point.

[0074] The control unit 11 retrains the learned model if concept drift is detected after a set time has elapsed since the last time concept drift was detected using any method and the learned model was retrained. Conversely, if concept drift is detected within a set time since the last time concept drift was detected using any method and the learned model was retrained, the control unit 11 detects that there is no concept drift without retraining the learned model. Specifically, the control unit 11 detects an abnormality in the equipment.

[0075] <2-2. Effects> According to the second embodiment described above, the same effect as effect (3) of the first embodiment is achieved.

[0076] (Other embodiments) Although embodiments of the present disclosure have been described above, the present disclosure is not limited to the embodiments described above and can be implemented in various modified forms.

[0077] (a) In the above embodiment, the control unit 11 generated a first learning model for calculating the drift threshold and a second learning model for actual operation. However, it is also possible to generate only the first learning model and not the second learning model. That is, the first learning model may be used for both calculating the drift threshold and for actual operation. When concept drift is detected, the control unit 11 only needs to retrain the first learning model, thereby reducing the processing load.

[0078] (b) The first to fourth devices 30 to 60 are not limited to mechanical equipment, but may be any type of equipment, such as movable equipment or portable equipment.

[0079] (c) Multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Also, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Furthermore, some of the configurations of the above embodiment may be omitted. Furthermore, at least some of the configurations of the above embodiment may be added to or replaced with the configurations of other above embodiments. [Explanation of Symbols]

[0080] 10...Detection device, 11...Control unit, 20...Terminal device, 30...First device, 40...Second device, 50...Third device, 60...Fourth device, 70...Communication network, 100...Status monitoring system.

Claims

1. A detection device comprising a control unit, The control unit, We obtain a collection of test data, which is time-series data, from the equipment. The set of test data acquired is input to the first trained model to calculate the set of first output values. The first mean, which is the group mean of the calculated first output values, and the first standard deviation, which is the group standard deviation of the first output values, are calculated. A predetermined number of operational data, which are time-series data from a later period than the test data, are acquired from the aforementioned equipment. The predetermined number of second output values ​​obtained by inputting the acquired predetermined number of operational data into the first model or a second trained model of the same type as the first model are averaged to calculate a second average value. From the first standard deviation and the predetermined number, the second standard deviation, which is the standard deviation of the second mean, is calculated. A drift threshold is set from the first mean and the second standard deviation. Concept drift is detected based on the fact that the calculated second average value is greater than or equal to the set drift threshold. Detection device.

2. The control unit, If a concept drift is detected again after a set time has elapsed since the previous detection of the concept drift, the first model, or both the first and second models, will be retrained. The detection device according to claim 1, which detects an abnormality in the equipment if the concept drift is detected within the set time since the concept drift was detected last time.

3. The control unit, The set of first output values ​​is updated by removing a set number of first output values ​​from the set of calculated first output values, starting with the largest values. The detection device according to claim 1 or 2.

4. The control unit, The first model is generated by learning from the first training data, which is the time series data immediately preceding the test data. The second model is generated by training on the test data and second training data including the time series data immediately preceding the test data. The detection device according to claim 3.

5. The control unit, Multiple input sets are input to the second learning model, with the predetermined number of operational data from the aforementioned device being treated as a single input set. The second average value is calculated for each of the above multiple input sets. Concept drift is detected based on the fact that the second average value of the second of the most recently calculated first number is greater than or equal to the drift threshold. The detection device according to claim 1 or 2.

6. The control unit, Set an anomaly threshold that is greater than the aforementioned drift threshold, If the calculated second average value is greater than or equal to the set abnormality threshold, an abnormality in the device is detected. The detection device according to claim 1 or 2.

7. To the processing unit This involves obtaining a collection of test data, which is time-series data, from the equipment, The first trained model is input to the acquired set of test data to calculate a set of first output values, The first mean, which is the average value of the group of first output values ​​that have been calculated, and the first standard deviation, which is the standard deviation of the group of first output values, The acquisition of a predetermined number of operational data, which are time-series data from the aforementioned equipment that are later than the test data, The process involves inputting the acquired predetermined number of operational data into the first model or a second trained model of the same type as the first model, averaging the predetermined number of second output values ​​obtained, and calculating a second average value. The second standard deviation, which is the standard deviation of the second mean, is calculated from the first standard deviation and the predetermined number. Setting a drift threshold from the first mean and the second standard deviation, Based on the fact that the calculated second average value is greater than or equal to the set drift threshold, the system will perform the following actions: Detection program.

8. A detection method performed by a processing unit, This involves obtaining a collection of test data, which is time-series data, from the equipment, The first trained model is input to the acquired set of test data to calculate a set of first output values, The first mean, which is the average value of the group of first output values ​​that have been calculated, and the first standard deviation, which is the standard deviation of the group of first output values, The acquisition of a predetermined number of operational data, which are time-series data from the aforementioned equipment that are later than the test data, The process involves inputting the acquired predetermined number of operational data into the first model or a second trained model of the same type as the first model, averaging the predetermined number of second output values ​​obtained, and calculating a second average value. The second standard deviation, which is the standard deviation of the second mean, is calculated from the first standard deviation and the predetermined number. Setting a drift threshold from the first mean and the second standard deviation, The system includes detecting the occurrence of concept drift based on the fact that the calculated second average value is greater than or equal to the set drift threshold, Detection method.

9. A detection device comprising a control unit, The control unit, The device acquires operational data that will be input into the model generated by learning from the training data. The concept drift of the operational data relative to the training data is detected, If the concept drift is detected again after a set time has elapsed since the previous detection, the model is retrained. If, in the previous instance, the concept drift was detected within the set time since the previous detection, it was determined that it was not a concept drift. Detection device.

Images