State detection device

Abstract

To provide a state detection device which is applied to an edge device, and is capable of easily realizing a state detection of a facility based on a sensor without being influenced by a phase of a time series signal of the sensor.SOLUTION: A time series signal from a sensor is converted into a spectral intensity using a Fast Fourier Transform (FFT) and the like, so as to generate one-dimensional data of the spectral intensity. Further, for instance, a pseudo-like image (hereinafter referred to as a pseudo image) is generated by that the one-dimensional data is arranged repeatedly in a longitudinal direction, or the one-dimensional data for a plurality of sensors is arranged in a longitudinal direction. Moreover, the pseudo image is analyzed by an image analysis unit for a neural network and the like; a distribution of a feature quantity in normal time is studied by using an intermediate feature quantity of the neural network; and a state of the facility is identified based on a distribution of the normal time and distance computing of an intermediate feature quantity of an observed value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a condition detection device. [Background technology]

[0002] Infrastructure and plant facilities are aging, and their maintenance and management is an important social issue. In response to this, there is a growing demand for automatic equipment monitoring technology. As a result, in recent years, as one example, systems that detect external abnormalities (scratches, cracks, etc.) in equipment by applying image recognition AI processing to camera images and satellite images are beginning to be put into practical use.

[0003] However, methods that use only images have problems such as limitations in detection due to image resolution restrictions and the fact that they can only detect abnormalities in appearance.Therefore, there is a need for technology that can detect the condition of equipment with high accuracy and reliability by analyzing the time-series signals of various sensors attached to or placed near the equipment.

[0004] However, in the past, analysis of time-series signals required the manual creation of dedicated analysis algorithms according to the type of sensor, the characteristics of the signal, the type of state to be detected, etc. As a result, it was not easy to build a system in a short period of time and at low cost.

[0005] On the other hand, there are also known efforts to identify conditions by processing time-series signals from sensors using AI such as deep learning. However, deep learning, which can generally handle time-series signals, generally has the problem that it is not easy to train a neural network and requires a lot of time and effort.

[0006] Patent Document 1 also discloses a technology for detecting abnormalities in equipment by arranging time-series signals from sensors, converting them into pseudo RGB images, and analyzing these images with an image recognition AI that uses a convolutional neural network and a support vector machine. The convolutional neural network extracts features from the above images, and the support vector machine judges the presence or absence of abnormalities (binary judgment) based on these features.

[0007] Patent Document 1 enables the application of image recognition AI that is easy to learn by replacing anomalies in a time series signal with anomalies in an image. However, in the case of a sensor such as a vibration sensor, the phase of the sensor signal and the phase difference between the sensor signals can have countless variations, which results in countless variations in the converted image. This creates new challenges in ensuring the accuracy of learning and inference.

[0008] In addition, in Non-Patent Document 1, in image anomaly detection, normal data is used as learning data for a pre-trained neural network, a multidimensional Gaussian distribution is calculated from intermediate features of the normal data, and the normal distribution is learned, so that normal and abnormal states are detected from the observed value and the distance calculation for normal states without learning abnormal data. However, for different types of sensor devices, it is necessary to appropriately select a pre-training model, and there are issues with application to a large-scale anomaly detection system using multiple sensors and with improving accuracy by detecting states collectively using different types of sensors.

[0009] Furthermore, to promote widespread use in actual fields, it is necessary to be able to build a condition detection system based on low-cost edge devices with limited hardware resources. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] JP 2020-144619 A [Non-patent literature]

[0011] [Non-Patent Document 1] Thomas Defard et al., "PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization", Proceedings of International Conference on Pattern Recognition 2021 (ICPR2021), Springer, 5 March 2021, pp.475-489. Summary of the Invention [Problem to be solved by the invention]

[0012] In view of the above circumstances, an object of the present disclosure is to provide a status detection device that can be applied to an edge device and can detect the status of equipment based on a large number of sensors easily, with high accuracy and robustness, and without being affected by the phase of the sensor's time-series signals. [Means for solving the problem]

[0013] According to the present disclosure, the following condition detection device is provided. That is, the condition detection device includes one or multiple sensors and a calculation unit that outputs data detected by the sensors, and the calculation unit includes a processing device that executes data processing. The processing device converts digitized time-series signal data of the multiple sensors into data related to spectral intensity, generates a pseudo image based on the data related to spectral intensity, analyzes the pseudo image, learns a distribution from intermediate features in image analysis, and outputs an equipment condition detection result using distance calculation and probability calculation between the learned distribution and observed values. Effect of the Invention

[0014] The present invention provides a status detection device that can be applied to edge devices and can realize the status detection of equipment based on a large number of sensors with ease, high accuracy, and robustness, without being affected by the phase of the time-series signal of the sensor. Note that problems, configurations, and effects other than those described above will become apparent from the following description of the embodiment of the invention. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a diagram showing an example of the configuration of a status detection device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the arrangement of an image analysis unit in the first embodiment. [Diagram 3] FIG. 4 is a diagram for explaining a procedure of state detection in the first embodiment. [Figure 4] FIG. 13 is a diagram showing an example of the configuration of a status detection device according to a second embodiment. [Diagram 5] FIG. 11 is a diagram for explaining the effect of the probability calculation unit in the second embodiment. [Figure 6] FIG. 11 is a diagram showing the configuration of a probability calculation unit in the second embodiment. [Figure 7] FIG. 11 is a diagram for explaining state updating and observed value combination in the second embodiment. [Figure 8] FIG. 11 is a diagram for explaining a state probability calculation in the second embodiment. [Figure 9] FIG. 13 is a diagram showing an example of the configuration of a status detection device according to a third embodiment. [Figure 10] FIG. 13 is a diagram for explaining the effect of probability synthesis in the third embodiment. [Figure 11] FIG. 13 is a diagram showing an example of the configuration of probability synthesis in the third embodiment. [Figure 12] FIG. 1 is a diagram showing an example of the configuration of a condition detection system using the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. The embodiment is an example for explaining the present disclosure, and for the sake of clarity, some parts have been omitted or simplified as appropriate. The present disclosure can be implemented in various other forms.

[0017] When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts, or when there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0018] In the embodiments, the processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., CPU, GPU), and performs processing defined by the program while using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the subject of the processing performed by executing the program may be the processor. Similarly, the subject of the processing performed by executing the program may be a controller, device, system, computer, or node having a processor.

[0019] The program may be installed in the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers.

[0020] First Embodiment The present disclosure relates to a technology for automatically detecting the status of equipment and the like from a sensor signal, and can contribute to solving social issues such as the maintenance and management of infrastructure equipment. According to the present disclosure, it is possible to realize image recognition-based status detection that is not affected by the phase of the sensor signal and is suitable for automation and hardware weight reduction. FIG. 1 shows a configuration example of a first embodiment of the present disclosure. Each of a plurality of sensors for monitoring the status of equipment outputs a time series signal. These time series signals are assumed to have already been digitized at the stage of sensor output, or to have been digitized by an A / D converter downstream of the sensor.

[0021] These digitized time-series signals are input to the spectrum intensity conversion unit 11. The spectrum intensity conversion unit 11 converts each of the digitized time-series signals into a vector related to frequency spectrum intensity by, for example, discrete Fourier transform or fast Fourier transform (FFT).

[0022] That is, the digitized time series signal is converted into information of complex components I+j*Q at each frequency position by discrete Fourier transform, FFT, etc., and further, from the complex components, for example, √(I 2 +Q 2 ) to calculate the spectral intensity. Note that a window function may be applied, as is commonly done in discrete Fourier transform and fast Fourier transform. In this way, the spectral intensity at each frequency position is obtained as a spectral intensity vector. This spectral intensity vector is output for each sensor.

[0023] Each of the spectral intensity vectors is input to the quantization unit 12. The quantization unit 12 reduces the representation word length for each element value of the spectral intensity vector (i.e., the value of the frequency spectral intensity) to, for example, 8 bits so as to match the bit depth of a pseudo image described later, and outputs the reduced length. As a result, the number of quantized spectral intensity vectors output is the same as the number of sensors.

[0024] Each of the quantized spectral intensity vectors is input to the pseudo image generating unit 13. The pseudo image generating unit 13 treats one quantized spectral intensity vector as one row, and vertically arranges the quantized spectral intensity vectors corresponding to each sensor to generate a two-dimensional image. This image is a pseudo image (pseudo image) because it is not a normal image such as a camera image. The bit depth of this pseudo image is equal to the expression word length reduced by the quantization unit 12. By reducing the bit depth of the pseudo image in this way, the amount of calculation in the next stage, the image analysis unit 14, can be reduced.

[0025] The pseudo image is input to the image analysis unit 14. The image analysis unit 14 regards the pseudo image as an image, classifies the image using a general image processing algorithm or AI processing for image recognition, and outputs the classification result. This allows a classification result to be obtained for the state of the equipment monitored by the sensor (for example, a first normal state, a second normal state, a first abnormal state, a second abnormal state, a third abnormal state, etc.).

[0026] Here, when the image analysis unit 14 is composed of a multi-stage feature extractor, it is possible to obtain intermediate features during the analysis process from the image analysis unit. In addition, since normal images other than pseudo images can be input to the image analysis unit, it is possible to extract intermediate features for multiple types of sensor signals with the same image analysis unit. Furthermore, by calculating intermediate features for inputs that combine pseudo images and normal images, it is possible to perform state detection using a combination of multiple sensors, thereby improving the accuracy and robustness of state detection. By comparing the distance with the learned distribution, the intermediate features output from the image analysis unit can be quantified for the type of sensor or the distribution for unknown states. Therefore, it is possible to perform state detection in the judgment processing unit 16 from the result of the distance calculation.

[0027] This embodiment will be described in more detail with reference to Fig. 2. When the image analysis unit is composed of an input process 21, a feature extraction process 22, and an output process, it is possible to obtain a plurality of intermediate features from the output result of the feature extraction. A distribution is calculated using one or a plurality of types of the intermediate features. For example, the expected value and covariance matrix of the multidimensional intermediate features can be calculated using a multidimensional Gaussian distribution. This distribution is learned using normal data, and the distance from the observed value is calculated, thereby making it possible to detect whether the state is normal or not.

[0028] The procedure for anomaly detection is explained using Figure 3. First, to calculate intermediate features, a neural network is pre-trained31. There are many pre-trained models for image recognition neural networks using huge data sets, so they may be used directly. Next, the distribution of known states is trained. Usually, in anomaly detection, it is easy to obtain normal data but difficult to obtain abnormal data, so the distribution of only normal data is trained. Normal data is input into a pre-trained network and intermediate features are calculated32. A distribution is calculated from the calculated intermediate features33. For example, if a multidimensional Gaussian distribution is assumed, the expected value and covariance matrix are obtained as parameters. Next, state detection is performed from the observed value from the sensor. Using the same trained network as when the normal data was trained, the intermediate features when the observed value is input are calculated34. Since the distribution of normal data has already been trained, it is possible to calculate the distance between the intermediate features of the observed value and the distribution of normal data. Generally, the Mahalanobis distance is used as the distance, and the distance is calculated taking into account the variance in the distribution of normal data35. Finally, based on the calculated distance, it is determined whether the observed value belongs to normal data36. If the distance is small enough, the condition is determined to be normal, and if it is large enough, the condition is determined to be abnormal.

[0029] <Second embodiment> FIG. 4 shows the configuration of the second embodiment of the present disclosure. In the second embodiment of the present disclosure, instead of detecting the state using the distance between the intermediate feature and a known distribution, the probability calculation unit 41 converts the intermediate feature to a probability and then performs a judgment process. By calculating the probability in the probability calculation unit, it is possible to automatically reduce false detections due to outliers of the sensor and the influence of observation values ​​from a broken or low-precision sensor. The effect of the probability calculation unit will be explained using FIG. 5. When anomaly detection is performed using threshold judgment from the result of the distance calculation, the state estimation result 51 has only two values, '0' and '1', for the probability of normality or anomaly. On the other hand, if the state is expressed as a probability of normality or anomaly, it is possible to probabilistically estimate the state 52 from data observed in the past and current observation values, thereby reducing outliers, false detections, and non-detection.

[0030] The probability calculation unit will be described with reference to FIG. 6. In the probability calculation unit, the state estimate X k-1 Prediction of current state from X k Calculate the prediction X k and the intermediate feature value Y k The current state is updated and used as the state estimation for the next step. This loop is repeated to estimate the state, and the state probability is calculated from the state prediction result when the observed values ​​are combined and the distance between the observed values ​​and the distribution of the current state. Here, the probability that the state is normal is P(X n ), the probability of abnormality P(X a ) the following equation holds: P(X n ) + P(X a ) = 1 (Equation 1) We only need to find the probability of either one, so the probability of normality is P(X n ), the following formula holds using Bayes' theorem: P(X n |Y) = P(Y|X n ) P(X n ) / P(Y) (Equation 2) In Equation 2, the left side is the probability that the state is normal when observed value Y is obtained, and the first term on the right side is the probability that observed value Y is obtained in a normal state. The distribution of normal states has already been learned, and the first term can be found from the distance between the distribution and the observed value. The second term is the prior probability of the state, and can be found from the state prediction results.

[0031] An example of probability calculation is shown below. Using a Kalman filter, the stochastic process is treated as a first-order Markov process, and state estimation is performed sequentially using information from the previous time k-1 and the current time k. The probability calculation is performed in two stages: prediction before combining observed values ​​and updating after the combination. Let [](k,k) be the estimate for time k, [](k,k-1) be the value for time k estimated from observed values ​​up to time k-1, and [](k-1,k-1) be the estimate for time k-1. If the true state is X_true, the estimate of the state is X, the variance matrix is ​​P, the noise of the predicted value is V, the covariance matrix of the noise of the predicted value is Q, and N(A,B) is a multidimensional normal distribution with expected value A and covariance matrix B, then X = X_true + V X(k,k-1) = X(k-1,k-1) P(k,k-1) = P(k-1,k-1) + Q V ~ N(0,Q) (Equation 3) Then, calculate P(k,k-1) before the observation value at time k is synthesized. The normality probability P(X n ) is also calculated in two stages. The probability of normal and abnormal states is P(X n ), P(X a ), then P(X n ) + P(X a ) = 1, P(X n (k,k-1)) = p 11 P(X n (k-1,k-1))+p 10 (1-P(X n (k-1,k-1)) (Formula 4) Here, p 11 , p 10 is the state transition probability. (For example, p 11 is 0.9, p 10 is 0.1)

[0032] Next, the observation Y at time k k The normal state probability P(X n (k,k)) is calculated. The combination of observations is calculated using Bayes' theorem. k This is done by finding the conditional probability of P(X n (k,k)) = (1-δ(k)) P(X n (k,k-1)) / (1-δ(k) P(X n (k,k-1))) δ(k) = pd (1- V(k) P(Y(k) | Y(k-1)) / m k ) m k = 1 - pd P(X n (k, k-1) V(k) = C n Gamma n / 2 |S(k)| 2 (Formula 5) γ is the Mahalanobis distance between the predicted value Y(k,k-1) and the observed value Y(k,k), γ = (Y(k,k) - Y(k,k-1)) T (S(k)) -1 (Y(k,k) - Y(k,k-1)) (Eq. 6) C n is the number of dimensions of the feature vector, n, C n = π(n / 2) / (n / 2)! (where n is an even number) C n = 2 / n!!·(2π)(n-1) / 2 (where n is an odd number) (Equation 7) S is the covariance matrix, and the noise of the observations is W ~ N(0,R), S = P(k,k-1) + R k (Formula 8) In addition, the covariance matrix P(k,k) of the state at time k is P(k,k) = (1-K k ) P(k,k-1) K k = P(k,k-1) S(k) -1 (Formula 9) The same steps are then repeated to perform estimation.

[0033] The method of probability calculation will be explained using Figure 7. Figure 7(a) is a system model showing the state transition between normal and abnormal. Using this model, we assume a system that transitions between normal and abnormal with a certain probability, and set the hyperparameter p 00 ,p 01 ,p 10 ,p 11 In other words, the transition to the normal state is predicted by p 11 andp 10 Figure 7(b) is a diagram explaining the calculation of the probability when an observation value is obtained for the current state, and the probability that the observation value follows a multidimensional Gaussian distribution can be calculated from the normal expected value and covariance matrix.

[0034] The method of calculating the probability will be explained using Figure 8. The final normality probability P(X n ) is calculated by considering all combinations of hypotheses about what signal source the observed value came from. In other words, the process θ k Then, the probability that the sum of all the processes θ that occur when the observations are normal is normal is P(X n For example, if there is one observation at time k, the two total hypotheses are either an observation from a normal state or an observation from a non-normal state. The probability of an anomaly can be calculated from the probability of normality using Equation 1.

[0035] <Third embodiment> The configuration of the third embodiment of the present disclosure is shown in Fig. 9. In the third embodiment of the present disclosure, the anomaly detection results of multiple sensors are combined by performing probability calculations to improve robustness and detection accuracy. By combining the results after converting them into a probability state, it becomes possible to adaptively calculate the influence of the reliability of each sensor and the presence or absence of failure or deterioration, improving the robustness of state detection.

[0036] The effect of the third embodiment of the present disclosure will be described with reference to Fig. 10. Assume that sensor 1 and sensor 2 output correlated results for the same physical phenomenon. If either sensor determines that the sensor is normal, the sensor is determined to be normal. If the observed value of sensor 1 is Y1 and the observed value of sensor 2 is Y2, the probability of each normality and the combined probability are expressed by the following formula. P(X n |Y1,Y2) = P(X n 1|Y1,Y2) + P(X n 2|Y1,Y2)+P(X n 1,X n 2|Y1,Y2) (Eq. 10) Here, the first term on the right-hand side is the probability that only sensor 1 detects normality, the second term is the probability that only sensor 2 detects normality, and the third term is the probability that both sensors 1 and 2 detect normality. By combining the results of sensors 1 and 2 according to Equation 10, it is possible to reduce false alarms.

[0037] In addition, if the abnormal and normal transitions of the same physical phenomenon are synthesized from the viewpoint of being sensor-independent, P(X n |Y1,Y2) = P(X n |Y1) P(X n |Y2) / (P(X n |Y1) P(X n |Y2) + P(X a |Y1) P(X a |Y2)) (Eq. 11) This reduces the number of undetected state transitions.

[0038] In addition, when the accuracy of each sensor is synthesized using the magnitude of the variance of the observed values, the covariance matrices P1 and P2 of the estimated value noise of the state are used to obtain P(X n |Y1,Y2) = P1 -1 P(X n |Y1) / (P1 -1 +P2 -1 ) + P2 -1 P(X n |Y2) / (P1 -1 +P2-1 ) (Formula 12) In addition, when using the noise covariances S1 and S2 at the time of updating the observation value, P(X n |Y1,Y2) = S1 -1 P(X n |Y1) / (S1 -1 +S2 -1 ) + S2 -1 P(X n |Y2) / (S1 -1 +S2 -1 ) (Formula 13) This is a composite result that is weighted to reflect the spread of the noise distribution for each sensor.

[0039] The method of probability synthesis will be further explained using Figure 11. When directly calculating Equation 10, as shown in Figure 11(b), intermediate feature quantities from the sensors are directly input and the probability is calculated. On the other hand, as shown in Figure 11(a), the amount of calculation required for synthesis can be reduced by using approximate calculations, by calculating state probabilities once for each sensor and then synthesizing the probabilities. For example, Linear opinion pooling is known as a synthesis method such as that shown in Figure 11(a), in which multiple types of probabilities p1, p2, p3, p n The probability P f is expressed by the following formula: P f = w1p1+ w2p2+ w3p3+ … + w n p n (Formula 14) Here, w1,…,w n are nonnegative coefficients that sum to one.

[0040] FIG. 12 shows an example of a system configuration according to an embodiment of the present disclosure. FIG. 12 shows an IoT system consisting of a gateway to which a plurality of sensor terminals are connected via a network, in which the sensor terminals can acquire sensor information connected to each terminal, and the gateway can acquire sensor information acquired by all sensor terminals. However, when all sensor information is synthesized at the gateway, it is expected that the communication band between the sensor terminals and the gateway will be congested, so the sensor information after synthesis may be synthesized at the sensor terminals and then further synthesized at the gateway. In this case, it is considered to use the method of FIG. 11(b) in which the sensor terminals can synthesize using information from all sensors, and to use the method of synthesizing probabilities in FIG. 11(a) in which the gateway does not have the original information of the sensors. [Explanation of symbols]

[0041] 11: Spectral intensity conversion section 12: Quantization section 13: Pseudo image generation section 14: Image analysis section 15: Distance calculation section 16: Judgment processing unit 21: Input processing 22: Feature Extraction 23: Output processing 24: Feature selection 41: Probability calculation section

Claims

1. A plurality of sensors; A calculation unit that outputs data detected by the plurality of sensors; Equipped with The calculation unit is A processing device for performing data processing, The processing device includes: converting the digitized time series signal data of the plurality of sensors into data regarding spectral intensity; generating a pseudo image based on the data on the spectral intensities; Analyzing the pseudo image and outputting a classification result of the equipment state; A condition detection device that outputs intermediate features from the results of image analysis and determines the condition by calculating the distance from a known distribution.

2. The condition detection device according to claim 1 , The processing device includes: The transformation is performed by a discrete Fourier transform or a fast Fourier transform. A condition detection device characterized by:

3. 2. The condition detection device according to claim 1, wherein the transformation uses at least one of a discrete cosine transformation, a discrete sine transformation, a discrete Fourier transformation, and a Walsh-Hadamard transformation.

4. 4. The condition detection device according to claim 3, wherein the plurality of orthogonal transformations are used in combination.

5. The condition detection device according to claim 1 , The processing device includes: Analyzing the simulated image using a convolutional neural network. A condition detection device characterized by:

6. A plurality of sensors; A calculation unit that outputs data detected by the plurality of sensors; Equipped with The calculation unit is A processing device for performing data processing, The processing device includes: converting the digitized time series signal data of the plurality of sensors into data regarding spectral intensity; generating a pseudo image based on the data on the spectral intensities; A condition detection device characterized by analyzing the pseudo image, outputting a classification result of the equipment condition, calculating a condition probability from the intermediate feature amount and a known distribution, and performing a judgment process.

7. 7. A condition detection device according to claim 6, wherein the probability calculation unit calculates a probability using Bayes' theorem from a predicted value of a state at a current time from a state at a previous time, an observed value, and a distance calculation of a known distribution.

8. 7. A condition detection device according to claim 6, wherein outputs of a probability calculation unit from a plurality of sensor signals are synthesized, and a judgment process is performed from the synthesized probability.

Images