Status detection device

Abstract

The present invention aims to provide a state detection device that can be applied to edge devices and can easily perform sensor-based equipment state detection without being affected by the phase of the sensor's time-series signal. [Solution] The time-series signals from the sensor are converted into spectral intensity using a Fast Fourier Transform (FFT) or the like to generate one-dimensional spectral intensity data. Furthermore, a pseudo-image (hereinafter referred to as a pseudo-image) is generated by, for example, repeatedly arranging the aforementioned one-dimensional data vertically, or by arranging the aforementioned one-dimensional data for multiple sensors vertically. Furthermore, the status of the equipment is identified by analyzing the aforementioned pseudo-image using an image analysis unit such as a convolutional neural network.

Description

Technical Field

[0001] The present invention relates to a state detection device.

Background Art

[0002] The aging of infrastructure facilities and plant facilities is progressing, and their maintenance management is an important social issue. In response to this, there is an increasing demand for automatic monitoring technology for facilities. Therefore, in recent years, as an example, a system for detecting appearance abnormalities (such as scratches and cracks) of facilities by performing image recognition AI processing on camera images and satellite images is being put into practical use.

[0003] However, in the method using only images, the limitations of detection due to the resolution constraints of the images and the fact that only appearance abnormalities can be known are problems. Therefore, a technology that can detect the state of facilities with high accuracy and reliability by analyzing the time-series signals of various sensors attached to the facilities or arranged in the vicinity of the facilities is desired.

[0004] However, conventionally, for the analysis of time-series signals, it has been necessary to manually create a dedicated analysis algorithm according to the type of sensor, the characteristics of the signal, the type of state to be detected, etc. As a result, it has not been easy to construct a system in a short period of time and at a low cost.

[0005] On the other hand, attempts to identify states by processing the time-series signals of sensors with AI such as deep learning are also known. However, in general, deep learning that can handle time-series signals has the problem that the learning of neural networks is not easy and requires a lot of time and labor.

[0006] Furthermore, Patent Document 1 discloses a technology for detecting equipment anomalies by arranging time-series signals from a sensor to convert them into a pseudo-RGB image, and then analyzing this image using an image recognition AI that employs a convolutional neural network and a support vector machine. The convolutional neural network extracts features from the image, and the support vector machine determines whether or not an anomaly exists (binary determination) based on these features.

[0007] Patent Document 1 enables the application of an image recognition AI that is easy to train by replacing anomalies in time-series signals with anomalies in images. However, in the case of sensors such as vibration sensors, there are countless variations in the phase of the sensor signal and the phase difference between sensor signals, resulting in countless variations in the converted image. Therefore, new challenges arise in ensuring the accuracy of training and inference.

[0008] Furthermore, in order to promote widespread adoption in actual field settings, it is necessary to be able to build a state detection system based on low-cost edge devices with limited hardware resources. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2020-144619 [Overview of the project] [Problems that the invention aims to solve]

[0010] In view of the above circumstances, the present invention aims to provide a state detection device that can be applied to edge devices and can easily perform sensor-based equipment state detection without being affected by the phase of the sensor's time-series signal. [Means for solving the problem]

[0011] The present invention provides the following state detection device. Specifically, the state detection device comprises a plurality of sensors or one sensor, and a calculation unit that outputs data detected by the sensor, the calculation unit comprising a processing unit that performs data processing. The processing unit converts the digitized time-series signal data from the plurality of sensors into spectral intensity data, generates a pseudo-image based on the spectral intensity data, analyzes the pseudo-image, and outputs a classification result of the equipment state. [Effects of the Invention]

[0012] According to the present invention, a state detection device is provided that can be applied to edge devices and easily perform sensor-based equipment state detection without being affected by the phase of the sensor's time-series signal. Other problems, configurations, and effects will be clarified by the following description of embodiments for carrying out the invention. [Brief explanation of the drawing]

[0013] [Figure 1(a)] A diagram showing an example configuration of the state detection device in the first embodiment. [Figure 1(b)] A diagram illustrating in detail an example of the pseudo-image generation process in the first embodiment. [Figure 1(c)] A diagram illustrating in detail an example of the pseudo-image generation process in the first embodiment. [Figure 2(a)] A diagram showing an example configuration of the state detection device in the second embodiment. [Figure 2(b)] A diagram showing an example of a convolutional neural network used in the second embodiment. [Figure 2(c)] A diagram illustrating an example of the learning process for the convolutional neural network used in the second embodiment. [Figure 3(a)] A diagram showing an example configuration of the state detection device in the third embodiment. [Figure 3(b)] A diagram illustrating an example of the effect of nonlinear quantization in a state detection device. [Figure 4(a)]A diagram showing a configuration example of the state detection device in the fourth embodiment. [Figure 4(b)] A diagram showing an example of the pseudo-image generation process using the intensity processing unit. [Figure 4(c)] A diagram showing an example of the pseudo-image generation process using the intensity processing unit. [Figure 5(a)] A diagram showing a configuration example of the state detection device in the fifth embodiment. [Figure 5(b)] A diagram for explaining an example of a plurality of pseudo-images generated by a plurality of pseudo-image generation units. [Figure 5(c)] A diagram for explaining an example of a plurality of pseudo-images generated by a plurality of pseudo-image generation units. [Figure 6(a)] A diagram showing a configuration example of the state detection device in the sixth embodiment. [Figure 6(b)] A diagram for explaining an example of the operation of the detection control unit. [Figure 7] A diagram showing a configuration example of the state detection device in the seventh embodiment. [Figure 8(a)] A diagram for explaining an example of the pseudo-image generation process in the eighth embodiment. [Figure 8(b)] A diagram for explaining an example of a pseudo-image. [Figure 8(c)] A diagram for explaining an example of a pseudo-image. [Figure 9(a)] A diagram showing a configuration example of the state detection device in the ninth embodiment. [Figure 9(b)] A diagram showing a configuration example of the state detection device in the ninth embodiment. [Figure 9(c)] A diagram showing a configuration example of the state detection device in the ninth embodiment. [Figure 10] A diagram showing a system configuration example in the tenth embodiment. [Figure 11] A diagram showing an example of the hardware configuration of the state detection device.

Embodiments for Carrying Out the Invention

[0014] Embodiments of the present invention will be described below with reference to the drawings. These embodiments are illustrative examples for illustrating the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. When there are multiple components with the same or similar function, they may be described using the same symbol but with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description. In embodiments, processing performed by executing a program may be described. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs processing defined by the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in some embodiments, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0015] <First Embodiment> The present invention relates to a technology for automatically detecting the state of equipment and other devices from sensor signals, and can contribute to solving social issues such as the maintenance and management of infrastructure equipment. Furthermore, according to the present invention, it is possible to realize image recognition-based state detection that is not affected by the phase of the sensor signal and is suitable for automation and hardware lightweighting. Figure 1(a) shows an example of the configuration of the first embodiment of the present invention. Each of a plurality of sensors for monitoring the state of equipment outputs a time-series signal. These time-series signals are assumed to be already digitized at the sensor output stage or to be digitized by an A / D converter downstream of the sensor.

[0016] These digitized time-series signals are input to the spectral intensity conversion unit 11. The spectral intensity conversion unit 11 converts each of the digitized time-series signals into a vector relating to frequency spectral intensity, for example, by performing a discrete Fourier transform or a fast Fourier transform (FFT).

[0017] In other words, the digitized time-series signal is converted into information of the complex component I+j*Q at each frequency position using a discrete Fourier transform or FFT, and then the spectral intensity is calculated from the complex component using an operation such as √(I²+Q²). A window function may also be applied, as is commonly done during discrete Fourier transforms and fast Fourier transforms. As a result, the spectral intensity at each frequency position is obtained as a spectral intensity vector. This spectral intensity vector is then output for each sensor.

[0018] Each spectral intensity vector is input to the quantization unit 12. The quantization unit 12 reduces the expression word length for each element of the spectral intensity vector (i.e., the value of the frequency spectral intensity) to a length such as 8 bits, which is suitable for the bit depth of the pseudo-image described later, and outputs it. As a result, quantized spectral intensity vectors are output for each sensor.

[0019] Each of the quantized spectral intensity vectors is input to the pseudo-image generation unit 13. The pseudo-image generation unit 13 treats one quantized spectral intensity vector as one row and arranges the quantized spectral intensity vectors corresponding to each sensor vertically to generate a two-dimensional image. This image is not a normal image like a camera image, and is therefore a pseudo-image. The bit depth of this pseudo-image is equal to the expression word length reduced in the quantization unit 12. By reducing the bit depth of the pseudo-image in this way, the amount of computation in the next stage image analysis unit 14 can be reduced.

[0020] The pseudo-image is input to the image analysis unit 14. The image analysis unit 14 treats the pseudo-image as an image and classifies it using general image processing algorithms and AI processing for image recognition, and outputs the classification result. This provides a classification result for the state of the equipment being monitored by the sensor (for example, first normal state, second normal state, first abnormal state, second abnormal state, third abnormal state, etc.).

[0021] This embodiment will be explained in more detail with reference to Figure 1(b). As described above, the digitized time-series signals from each sensor are subjected to a Fast Fourier Transform (FFT) to obtain spectral intensity vectors.

[0022] Alternatively, the system may be equipped with one Fast Fourier Transform (FFT) engine for each sensor, with one engine assigned to each sensor for processing, or multiple sensors may share a FFT engine and perform FFTs in a time-division multiplexer manner.

[0023] A spectral intensity vector is data consisting of the values ​​of frequency spectral intensity at each frequency position arranged horizontally. The number of dimensions of the vector (i.e., the number of frequency positions) is equal to half the number of points in the discrete Fourier transform or fast Fourier transform. Figure 1(b) shows the case with 16 points, and the number of dimensions of the vector is 8, but any number of points can be chosen as appropriate.

[0024] The aforementioned number of points determines the time width of the sensor signal used to generate each pseudo-image. That is, this time width is equal to the product of the time interval of the digitized time-series signal and the aforementioned number of points. Furthermore, the frequency resolution (the interval between each frequency position) is equal to the reciprocal of the time width.

[0025] Figure 1(b) shows the spectral intensity vector quantized, for example, by 8 bits in the quantization unit 12. Since it is 8 bits, it is represented by integer values ​​from 0 to 255. Therefore, the frequency spectral intensity may be normalized before quantization (i.e., reduced to an 8-bit word length) to fit this range. In Figure 1(b), the quantized spectral intensity vector corresponding to the first sensor is (85, 188, 128, 254, 171, 218, 149, 104). The quantized spectral intensity vector corresponding to the second sensor is (64, 89, 41, 112, 194, 168, 235, 213).

[0026] The pseudo-image generation unit 13, when generating a pseudo-image, treats the quantized spectral intensity vectors as one row and arranges them vertically. However, as shown in Figure 1(b), the quantized spectral intensity vectors corresponding to one sensor may be arranged in multiple rows, repeating the process. In Figure 1(b), each row is repeated twice. That is, the quantized spectral intensity vectors corresponding to the first sensor are arranged in the first and second rows of the pseudo-image, and the quantized spectral intensity vectors corresponding to the second sensor are arranged in the third and fourth rows, and so on, completing the pseudo-image. The number of repetitions can be selected as appropriate, including one repetition (i.e., no repetition).

[0027] As a result, the width (number of pixels) of the generated pseudo-image is equal to the number of dimensions of the quantized spectral intensity vector (8 pixels in Figure 1(b)). On the other hand, the height (number of pixels) is equal to the product of the number of sensors and the number of repetitions.

[0028] Furthermore, the pixel values ​​of the pseudo-image correspond to the quantized frequency spectral intensity after normalization. Thus, since the pixel values ​​of the pseudo-image correspond to the frequency spectral intensity of the time-series signal output from the sensor, they are not affected by the phase of the time-series signal.

[0029] When the vertical size of the pseudo-image is defined, the number of sensors that can be handled in a single pseudo-image decreases inversely proportional to the number of repetitions. On the other hand, the amount of information per sensor within the pseudo-image increases in proportion to the number of repetitions, making classification by the image analysis unit 14 easier.

[0030] Figure 1(c) provides further details regarding the generation of pseudo-images in this embodiment. In Figure 1(c), as in Figure 1(b), normalized and quantized spectral intensity vectors are generated for each sensor. For example, the quantized spectral intensity vector corresponding to the first sensor is (85,188,128,254,171,218,149,104), and the quantized spectral intensity vector corresponding to the second sensor is (64,89,41,112,194,168,235,213).

[0031] However, in Figure 1(c), the width of the target pseudo-image is set to 4 pixels, which is half the width in Figure 1(b). Therefore, the dimensionality of each quantized spectral intensity vector is twice the width of the target pseudo-image.

[0032] In such cases, each of the quantized spectral intensity vectors is divided into multiple spectral intensity subvectors to fit the width of the pseudo-image. When the pseudo-image generation unit 13 generates the pseudo-image, it treats the spectral intensity subvectors as one row and arranges each spectral intensity subvector vertically.

[0033] For example, in Figure 1(c), the quantized spectral intensity vector is split into two spectral intensity subvectors, and the dimension of the spectral intensity subvectors is halved (i.e., 4) to fit the width of the pseudo-image (i.e., 4).

[0034] For example, the quantized spectral intensity vector corresponding to the first sensor is divided into a first spectral intensity subvector (85, 188, 128, 254) and a second spectral intensity subvector (171, 218, 149, 104). Similarly, the quantized spectral intensity vector corresponding to the second sensor is divided into a third spectral intensity subvector (64, 89, 41, 112) and a fourth spectral intensity subvector (194, 168, 235, 213).

[0035] The pseudo-image generation unit 13 generates a pseudo-image by arranging the first spectral intensity subvector in the first row, the second spectral intensity subvector in the second row, the third spectral intensity subvector in the third row, and the fourth spectral intensity subvector in the fourth row, and similarly arranging the spectral intensity subvectors corresponding to the third sensor and subsequent sensors.

[0036] This allows for the generation of a pseudo-image with a width (number of pixels) of 4, as intended. The height (number of pixels) of the generated pseudo-image is the product of the number of divisions and the number of sensors.

[0037] Furthermore, the repetitions described in Figure 1(b) may be combined. That is, each of the spectral intensity subvectors may be repeated in multiple rows. For example, if the number of repetitions is 4, the first spectral intensity subvector is repeated in rows 1 through 4, the second spectral intensity subvector is repeated in rows 5 through 8, the third spectral intensity subvector is repeated in rows 9 through 12, and the fourth spectral intensity subvector is repeated in rows 13 through 16 to generate a pseudo-image. In this case, the vertical size (number of pixels) of the generated pseudo-image is the product of the number of repetitions, the number of divisions, and the number of sensors.

[0038] In Figures 1(b) and 1(c), when generating a pseudo-image by arranging the quantized spectral intensity vector and the spectral intensity sub-vector, they may be arranged in any order other than those described above, as appropriate.

[0039] Furthermore, if the number of sensors is small, the digitized time-series signal of one sensor may be reused as the output of other sensors, and a pseudo-image may be generated using the procedure described above. For example, the digitized time-series signal of the first sensor may be treated as the digitized time-series signal of the first sensor and the second sensor respectively, and the digitized time-series signal of the second sensor may be treated as the digitized time-series signal of the third sensor and the fourth sensor respectively.

[0040] Each of the above operations may be performed using a dedicated circuit (accelerator), or it may be performed by software processing using a general-purpose processor such as a CPU.

[0041] As described above, according to this embodiment, the state of the equipment being monitored by the sensor can be classified by image analysis without being affected by the phase of the time-series signal output from the sensor. Various image analysis methods have been known to date, and by selecting the appropriate method, even lightweight algorithms can perform highly accurate classification. Furthermore, there is a wealth of open-source software (OSS) available for image analysis. Therefore, it is believed that this embodiment makes it possible to easily and reliably realize equipment state detection based on sensors by applying edge devices.

[0042] <Second Embodiment> Figure 2(a) shows the configuration of the second embodiment of the present invention. In this embodiment, a convolutional neural network 21 is applied as the image analysis unit 14 in the first embodiment. The operation up to the pseudo-image generation unit 13 is as described in Figures 1(a), 1(b), and 1(c), so the explanation is omitted.

[0043] The pseudo-image output from the pseudo-image generation unit 13 is input to the convolutional neural network 21. The configuration of the convolutional neural network 21 is shown in Figure 2(b). The convolutional neural network 21 applies a classification type.

[0044] Similar to a typical convolutional neural network for classification, it includes a feature extraction unit 22 and a subsequent classification unit 23. The feature extraction unit 22 consists of multiple layers and performs feature extraction on the input pseudo-image.

[0045] The classification unit 23 consists of one or more layers and, upon receiving the results of the feature extraction, calculates the probability that the pseudo-image input to the convolutional neural network 21 belongs to each class and outputs it as a classification result. For example, in Figure 2(b), the convolutional neural network 21 performs classification into five classes: first normal state, second normal state, first abnormal state, second abnormal state, and third abnormal state, and outputs the probability of belonging to each class. In this way, the state of the equipment being monitored by the sensor can be classified based on the digitized time-series signal from the sensor.

[0046] As described above, the pseudo-image input to the convolutional neural network 21 has a reduced bit depth. Accordingly, the convolutional neural network 21 also performs calculations with a reduced word length, thereby reducing power consumption and implementation area. It is generally known that convolutional neural networks can maintain high-precision classification capabilities even when the word length is reduced.

[0047] The parameters of the convolutional neural network 21, such as weights, use pre-trained values. The training method is explained in Figure 2(c).

[0048] For example, if you want to classify a system into five classes—first normal state, second normal state, first abnormal state, second abnormal state, and third abnormal state—you would need to acquire digitized time-series signal data from each sensor for each of these five states beforehand.

[0049] For example, when using four sensors, for the first normal state, the following data is obtained: DATA[First Normal][1] from the first sensor, DATA[First Normal][2] from the second sensor, DATA[First Normal][3] from the third sensor, and DATA[First Normal][4] from the fourth sensor. Similarly, for the second normal state, the data from the first sensor DATA[Second Normal][1], the data from the second sensor DATA[Second Normal][2], the data from the third sensor DATA[Second Normal][3], and the data from the fourth sensor DATA[Second Normal][4] are obtained. Similarly, for the first abnormal state, the data DATA[First Abnormality][1] from the first sensor, DATA[First Abnormality][2] from the second sensor, DATA[First Abnormality][3] from the third sensor, and DATA[First Abnormality][4] from the fourth sensor are obtained. Similarly, for the second abnormal state, the data DATA[Second Abnormality][1] from the first sensor, DATA[Second Abnormality][2] from the second sensor, DATA[Second Abnormality][3] from the third sensor, and DATA[Second Abnormality][4] from the fourth sensor are obtained. Similarly, for the third abnormal state, the data DATA[Third Abnormality][1] from the first sensor, DATA[Third Abnormality][2] from the second sensor, DATA[Third Abnormality][3] from the third sensor, and DATA[Third Abnormality][4] from the fourth sensor are obtained.

[0050] Each of these data sets is divided into groups of samples equal to the number of points in the Discrete Fourier Transform or Fast Fourier Transform. For example, if a pseudo-image is generated using a 1024-point Discrete Fourier Transform or Fast Fourier Transform, each data set is divided into groups of 1024 samples from the beginning. For example, the k-th divided group of 1024 samples in DATA[First Normal][1] will be called DATA[First Normal][1][k]. The same applies to each other DATA set.

[0051] As shown in Figure 2(c), during training, DATA[X][1][Y], DATA[X][2][Y], DATA[X][3][Y], and DATA[X][4][Y] are retrieved and output from the training data storage unit 24. X is randomly selected from five possible states: first normal, second normal, first abnormal, second abnormal, and third abnormal. The classification number Y is also randomly selected. These four DATAs are data acquired simultaneously from four sensors.

[0052] In this way, the digitized time-series signal data of sensors 1, 2, 3, and 4 corresponding to the selected state and the selected classification number are simultaneously output from the learning data storage unit 24. The selected state X is also simultaneously output as label data. The digitized time-series signal data of each sensor is input to the spectral intensity conversion unit 11, and thereafter, the spectral intensity conversion unit 11, quantization unit 12, and pseudo-image generation unit 13 operate as described above, and a pseudo-image for state X is output from the pseudo-image generation unit 13.

[0053] The convolutional neural network 21 classifies the pseudo-images as described above and outputs the classification results. These classification results are numerical probabilities of belonging to the first normal state, the second normal state, the first abnormal state, the second abnormal state, and the third abnormal state, as described above.

[0054] The classification result is input to the learning unit 25. The label data indicating state X is also input to the learning unit 25. Furthermore, the values ​​of the internal nodes of the convolutional neural network 21 necessary for learning are also supplied to the learning unit 25. Based on these inputs, the learning unit 25 calculates values ​​such as weight parameters used in the convolutional neural network 21 using conventionally known algorithms.

[0055] The selection of state X and classification number Y, and the output of data from the learning data storage unit 24 are performed continuously and sequentially, and the generation of pseudo-images, classification of pseudo-images, and calculation of weight parameters, etc., based on the classification results are performed continuously for a predetermined period of time.

[0056] The calculation of weight parameters, etc., may be performed each time a pseudo-image is generated and classified (i.e., online learning), or it may be performed based on the results of multiple pseudo-image generation and classifications (i.e., batch learning).

[0057] After the predetermined period has ended, the learning unit 25 supplies the final obtained weight parameter values ​​to the convolutional neural network 21. This completes the learning process. The convolutional neural network 21 uses the supplied weight parameters when performing the operations described in Figure 2(a). As a result, the equipment status is classified as described above.

[0058] During the learning process, some or all of the operations shown in Figure 2(c) may be performed on a device different from the device performing the operations shown in Figure 2(a). For example, the operations shown in Figure 2(a), i.e., the normal operations of classifying the state of the equipment, may be performed using an edge device with limited hardware resources, while the learning operations shown in Figure 2(c) may be performed on a server or other device with abundant hardware resources. Furthermore, the implementation and execution are not limited to these examples and may be carried out in various ways.

[0059] As described above, according to this embodiment, the state of equipment being monitored by a sensor can be classified using a convolutional neural network that executes image recognition AI, without being affected by the phase of the time-series signal output from the sensor. By using a convolutional neural network, an image analysis algorithm corresponding to the sensor is automatically acquired through learning from the sensor data. Furthermore, convolutional neural networks are easier to learn and require less learning time compared to neural networks for time-series signals. In addition, by using a convolutional neural network with a reduced word count, power consumption and implementation area can be reduced, making it possible to implement it on edge devices. Therefore, it is believed that this embodiment makes it possible to easily and reliably realize sensor-based equipment state detection by applying edge devices.

[0060] <Third Embodiment> Figure 3(a) shows the configuration of a third embodiment of the present invention. In this embodiment, a nonlinear quantization unit 31 is applied as the quantization unit 12 in the second embodiment. The operation of the other parts is as described in the first and second embodiments, so the explanation is omitted. The reason why the application of the nonlinear quantization unit 31 is effective will be explained below using Figure 3(b).

[0061] As mentioned above, quantization is necessary to reduce the length of the expression words for each element of the spectral intensity vector. To achieve very high-precision equipment state detection, the quantization step size needs to be finer to avoid losing minute information. As a result, a long expression word length becomes necessary.

[0062] Therefore, in this embodiment, a nonlinear quantization unit 31 is introduced. The nonlinear quantization unit 31 first performs a nonlinear transformation on the value of each element of each input spectral intensity vector, and then performs quantization (i.e., expresses it with a reduced expression word length). As shown in Figure 3(b), the saturating characteristics of the nonlinear transformation allow for a significant reduction in the range of values, making it possible to apply a reduced expression word length.

[0063] Furthermore, this nonlinear quantization is only possible in combination with the preceding discrete Fourier transform or fast Fourier transform (FFT). That is, as shown in Figure 3(b), if nonlinear quantization is directly applied to the digitized time-series signal of the sensor, minute components are lost due to the saturation characteristics of the nonlinear transformation. On the other hand, as in the embodiment, by converting the time-series signal into frequency-domain information using the discrete Fourier transform or fast Fourier transform (FFT), minute components are separated, and the information of these minute components is maintained even if a nonlinear transformation is performed afterward.

[0064] Similar to the quantization unit 12, the nonlinear quantization unit 31 outputs quantized spectral intensity vectors for each sensor. This implementation enables even higher accuracy detection of the equipment's state in addition to the effects of the second embodiment.

[0065] <Fourth Embodiment> Figure 4(a) shows the configuration of the fourth embodiment of the present invention. In this embodiment, the strength processing unit 41 is inserted after the nonlinear quantization unit 31, as in the third embodiment. The operation of the other parts is as described in the first, second, and third embodiments, so a description is omitted.

[0066] The operation of the intensity processing unit 41 will be explained using Figure 4(b). The intensity processing unit 41 receives quantized spectral intensity vectors from the nonlinear quantization unit 31, one for each sensor. In order to remove the influence of unwanted components contained in the sensor signal, the intensity processing unit 41, for example, sets the elements at the frequency positions corresponding to the unwanted components in the quantized spectral intensity vectors to zero.

[0067] For example, in Figure 4(b), the fourth element, corresponding to the frequency of the unwanted component, is set to zero in order to remove the influence of unwanted components in the time-series signal of the first sensor. Similarly, the sixth and eighth elements, corresponding to the frequencies of the two unwanted components in the time-series signal of the second sensor, are set to zero in order to remove the influence of these components. Furthermore, the time-series signal of the last sensor contains broadband unwanted components, and to remove this influence, the second and third elements, corresponding to the occupied bandwidth of the unwanted components, are set to zero. The intensity processing unit 41 outputs a quantized spectral intensity vector (processed spectral intensity vector) after performing the above zero substitutions.

[0068] Furthermore, a filter is necessary to directly remove unwanted components from a time-series signal. In particular, when unwanted components are scattered across various frequencies, as in the time-series signal of the second sensor, designing and implementing the filter is not easy.

[0069] In contrast, in this embodiment, since the time-series signal is converted into frequency-domain information using the Discrete Fourier Transform or Fast Fourier Transform (FFT), as described above, the influence of unwanted components can be removed simply by replacing the values ​​of the elements of the quantized spectral intensity vector with zero. As shown in Figure 4(b), the zeros are reflected in the pseudo-image generated in the subsequent stage as a result of this operation. Therefore, the influence of unwanted components is removed from the pseudo-image. As a result, the convolutional neural network 21 can classify the pseudo-image without being affected by unwanted components.

[0070] Figure 4(c) illustrates another example of the operation of the strengthening section 41. In the operation described in Figure 4(b), it is necessary to know the frequency of the unwanted components in advance by some means. In contrast, Figure 4(c) shows an example of the operation of the strengthening section 41 when the presence or frequency of unwanted components is unknown.

[0071] Similar to Figure 4(b), the intensity processing unit 41 receives quantized spectral intensity vectors from the nonlinear quantization unit 31, one vector for each sensor. If the upper limit of the frequency spectral intensity of the required signal included in the time-series signal of each sensor is known, the intensity processing unit 41 replaces any elements of the quantized spectral intensity vector that exceed that upper limit with the upper limit. This does not remove unwanted components, but it does limit their influence.

[0072] In the example shown in Figure 4(c), the upper limit of the frequency spectral intensity of the required signal included in the time-series signal of the first sensor (however, the upper limit after normalization) is set to 150, and all element values ​​exceeding this are replaced with 150. Similarly, the upper limit for the second sensor is set to 120, and the upper limit for the third sensor is set to 100, and elements exceeding these values ​​are replaced. The intensity processing unit 41 outputs a quantized spectral intensity vector (processed spectral intensity vector) after performing the above replacements.

[0073] As shown in Figure 4(c), the above operation reflects the replacement result in the pseudo-images generated in subsequent stages. Therefore, the contribution of unwanted components in the pseudo-images can be reduced. As a result, the convolutional neural network 21 can classify the pseudo-images under conditions where the influence of unwanted components is reduced.

[0074] In this embodiment, the strength processing unit 41 is located after the nonlinear quantization unit 31, but similar operation is possible and effective even if it is located before the nonlinear quantization unit 31.

[0075] As described above, according to this embodiment, in addition to the effects of the third embodiment, even if various unwanted components are included in each sensor signal, the status of the equipment can be detected without being affected by them, or with reduced effects.

[0076] <Fifth Embodiment> Figure 5(a) shows the configuration of the fifth embodiment of the present invention. In this embodiment, the pseudo-image generation unit 13 is replaced with multiple pseudo-image generation units 51 as in the fourth embodiment. The operation of the other parts is as described in the first, second, third, and fourth embodiments, so a description is omitted.

[0077] As explained in Figures 1(b) and 1(c), if the width and height of the pseudo-image are fixed, increasing the number of iterations to make the pseudo-image easier to classify, or increasing the number of points in the discrete Fourier transform or fast Fourier transform to improve frequency resolution, reduces the number of sensors that can be handled in a single pseudo-image. Therefore, especially when there are many sensors, it becomes necessary to generate multiple pseudo-images.

[0078] Therefore, in this embodiment, the multiple pseudo-image generation unit 51 generates multiple pseudo-images. As described above, the multiple pseudo-image generation unit 51 receives quantized spectral intensity vectors equal to the number of sensors.

[0079] An example of the operation of the multiple pseudo-image generation unit 51 will be explained using Figure 5(b). In this example, the multiple pseudo-image generation unit 51 generates three pseudo-images from the time-series signals of 12 sensors. To monitor the status of the equipment, four types of sensors are placed at three locations on or near the equipment. These four types of sensors detect physical quantities A, B, C, and D, respectively. Examples of physical quantities include displacement, velocity, acceleration, strain, magnetism, and flow rate in the x, y, and z axes, as well as angles, angular velocity, and angular acceleration related to roll, pitch, and yaw rotations, current, voltage, concentrations of various gases and fluids, heat, temperature, and humidity.

[0080] The multiple pseudo-image generation unit 51 generates one pseudo-image for each of the three locations. Specifically, it generates a first pseudo-image based on the time-series signals of the four sensors placed at inspection location 1. Similarly, it generates a second pseudo-image based on the time-series signals of the four sensors placed at inspection location 2, and a third pseudo-image based on the time-series signals of the four sensors placed at inspection location 3. By analogy with RGB images, the first pseudo-image, second pseudo-image, and third pseudo-image are referred to as pseudo-R image, pseudo-G image, and pseudo-B image, respectively.

[0081] Specifically, the multiple pseudo-image generation unit 51 follows the following rules when arranging the quantized spectral intensity vectors corresponding to each sensor in the vertical direction. That is, in the generation of each pseudo-image, the quantized spectral intensity vectors corresponding to the sensor of physical quantity A are arranged in the top quarter region of the pseudo-image. At that time, the repetition and division explained in Figure 1(b) and Figure 1(c) may be applied.

[0082] Similarly, the quantized spectral intensity vectors corresponding to the sensor for physical quantity B are placed in the second quarter region from the top of the pseudo-image, the quantized spectral intensity vectors corresponding to the sensor for physical quantity C are placed in the third quarter region from the top of the pseudo-image, and the quantized spectral intensity vectors corresponding to the sensor for physical quantity D are placed in the bottom quarter region of the pseudo-image.

[0083] The three pseudo-images generated by the multiple pseudo-image generation unit 51 are input to the convolutional neural network 21. The convolutional neural network 21 processes these simultaneously as pseudo-RGB images and outputs the classification result.

[0084] Generally speaking, convolutional neural networks can classify not only images represented by three channels (R, G, B), but also images represented by numbers of channels other than three. In addition to RGB, other color space representation methods such as CMY, HSV, and HLS are commonly known, as are broadcast-specific formats such as YUV, YCbCr, and YPbPr. In this embodiment, an example in which three pseudo-images are generated is shown, but other numbers of pseudo-images may be generated. Furthermore, this embodiment may employ color spaces other than RGB as described above. In these variations as well, the convolutional neural network 21 processes these pseudo-images simultaneously and outputs the classification result.

[0085] According to the pseudo-image generation method described above, the region in the pseudo-image corresponding to the sensor of physical quantity A is the top quarter region in all three pseudo-images. Similarly, the region in the pseudo-image corresponding to the sensor of physical quantity B is the second quarter region from the top in all three pseudo-images, the region corresponding to the sensor of physical quantity C is the third quarter region from the top, and the region corresponding to the sensor of physical quantity D is the bottom quarter region.

[0086] Thus, the regions within the pseudo-images corresponding to the same type of sensor are identical in all three pseudo-images. Generally, when a convolutional neural network receives multiple channel (multiple image) inputs, it performs a sum-of-products operation on the pixel values ​​of the same region within the multiple channel images to fuse the information from the multiple channels, and then performs feature extraction and classification based on this fusion.

[0087] Therefore, by inputting multiple pseudo-images generated by the method shown in Figure 5(b) into the convolutional neural network 21, the convolutional neural network 21 fuses the signal information from similar sensors located at different locations and performs feature extraction and classification based on this fusion. This fuses information acquired at different locations, improving the accuracy of classifying the equipment status.

[0088] The multiple pseudo-image generation unit 51 may generate multiple pseudo-images in a manner different from that shown in Figure 5(b). Another method will be explained using Figure 5(c). In this example as well, the multiple pseudo-image generation unit 51 generates three pseudo-images from the time-series signals of 12 sensors. Three types of sensors are placed at four locations on or near the equipment to monitor the equipment's status. These three types of sensors detect physical quantities A, B, and C, respectively.

[0089] The multiple pseudo-image generation unit 51 generates one pseudo-image for each of the three types of sensors. Specifically, it generates a first pseudo-image based on the time-series signals of the physical quantity A sensors located at four locations (inspection location 1, inspection location 2, inspection location 3, and inspection location 4). Similarly, it generates a second pseudo-image based on the time-series signals of the physical quantity B sensors located at four locations, and a third pseudo-image based on the time-series signals of the physical quantity C sensors located at four locations. As in the case of Figure 5(b), by analogy with RGB images, the first pseudo-image, second pseudo-image, and third pseudo-image are called pseudo-R image, pseudo-G image, and pseudo-B image, respectively.

[0090] Specifically, the multiple pseudo-image generation unit 51 follows the following rules when arranging the quantized spectral intensity vectors corresponding to each sensor in the vertical direction. That is, in the generation of each pseudo-image, the quantized spectral intensity vectors corresponding to the sensors installed at inspection location 1 are arranged in the top quarter of the pseudo-image. At that time, the repetition and division explained in Figure 1(b) and Figure 1(c) may be applied.

[0091] Similarly, the quantized spectral intensity vectors corresponding to the sensor installed at inspection site 2 are placed in the second quarter region from the top of the pseudo-image, the quantized spectral intensity vectors corresponding to the sensor installed at inspection site 3 are placed in the third quarter region from the top of the pseudo-image, and the quantized spectral intensity vectors corresponding to the sensor installed at inspection site 4 are placed in the bottom quarter region of the pseudo-image.

[0092] According to the pseudo-image generation method described above, the area in the pseudo-image corresponding to the sensor installed at inspection location 1 is the top quarter of the image in all three pseudo-images. Similarly, the area in the pseudo-image corresponding to the sensor installed at inspection location 2 is the second quarter from the top in all three pseudo-images, the area corresponding to the sensor installed at inspection location 3 is the third quarter from the top, and the area corresponding to the sensor installed at inspection location 4 is the bottom quarter. Thus, the area in the pseudo-image corresponding to the sensor installed at the same inspection location is the same in all three pseudo-images.

[0093] As described above, generally, when a convolutional neural network receives images from multiple channels (multiple images), it performs a sum-of-accumulate operation on the pixel values ​​of the same region within the multiple-channel images to fuse the information from multiple channels, and then performs feature extraction and classification based on that fusion. Therefore, by inputting multiple pseudo-images generated by the method in Figure 5(c) into the convolutional neural network 21, the convolutional neural network 21 fuses the signal information from different types of sensors placed at the same inspection location, and then performs feature extraction and classification based on that fusion. This fuses the information acquired from different types of sensors, improving the accuracy of the classification of the equipment status.

[0094] As described above, according to this embodiment, in addition to the effects of the fourth embodiment, even when there are many sensors, the status of the equipment can be classified by utilizing their information. Furthermore, the status of the equipment can be classified by fusing information from sensors installed in different locations or by fusing information from different types of sensors. Therefore, the status of the equipment can be detected with higher accuracy.

[0095] <Sixth Embodiment> Figure 6(a) shows the configuration of the sixth embodiment of the present invention. In this embodiment, a detection control unit 61 is added compared to the second embodiment. The detection control unit 61 performs adaptive control based on the classification result. The operation of the other parts is as described in the first and second embodiments, so a description is omitted.

[0096] The operation of the detection control unit 61 will be explained using Figure 6(b). As explained in the first embodiment, the time width of the sensor signal used to generate each pseudo-image (the "FFT time width" in Figure 6(b)) is determined by the number of points of the discrete Fourier transform and fast Fourier transform (FFT). The frequency resolution (the interval between each frequency position) is equal to the reciprocal of the time width. In this embodiment, pseudo-images are generated and classified at a constant time interval (the "FFT period" in Figure 6(b)).

[0097] The detection control unit 61 adaptively controls the FFT time width and FFT period according to the classification results output by the convolutional neural network 21. For example, if the convolutional neural network 21 outputs probabilities of belonging to a first normal state, a second normal state, a first abnormal state, a second abnormal state, or a third abnormal state, and the highest probability value is lower than a predetermined value, the FFT time width is increased to improve the accuracy of the classification and enhance the frequency resolution of the discrete Fourier transform and fast Fourier transform. In this way, by updating the FFT time width as needed while classifying the state of the equipment, an appropriate set value for the FFT time width can be automatically reached.

[0098] Alternatively, if the classification result output by the convolutional neural network 21 shows that the probability of belonging to the first abnormal state, the second abnormal state, or the third abnormal state is higher than a predetermined value, it may be considered that an abnormality has occurred in the equipment. In this case, the FFT time width may be set to a longer value to increase the frequency resolution of the discrete Fourier transform or fast Fourier transform, and the equipment status may be monitored. Furthermore, in order to increase the frequency of state detection, the FFT period may be set to a shorter value, and the equipment status may be monitored.

[0099] The detection control unit 61 communicates the set values ​​for the FFT time width and FFT period to the spectral intensity conversion unit 11, the quantization unit 12, the pseudo-image generation unit 13, and the convolutional neural network 21. The spectral intensity conversion unit 11 periodically performs discrete Fourier transforms and fast Fourier transforms (FFTs) on the number of points corresponding to the communicated FFT time width at a period equal to the communicated FFT period.

[0100] Furthermore, the quantization unit 12 reduces the expression word length for the spectral intensity vector with a number of dimensions corresponding to the communicated FFT time width, as described above. This operation is performed periodically at a period equal to the communicated FFT period.

[0101] Furthermore, the pseudo-image generation unit 13 generates a pseudo-image as described above using a quantized spectral intensity vector with a number of dimensions corresponding to the communicated FFT time width. This operation is performed periodically at a period equal to the communicated FFT period.

[0102] Furthermore, the convolutional neural network 21 is configured to process pseudo-images corresponding to the communicated FFT time width. It has weight parameters and the like that have been pre-trained assuming the FFT time width, and it performs classification of the pseudo-images at a period equal to the communicated FFT period. Output the results.

[0103] Figure 6(b) shows an example of the operating timing. The time-series signal from the sensor during a certain period (period n) is converted into a pseudo-image by the spectral intensity conversion unit 11, the quantization unit 12, and the pseudo-image generation unit 13 during that period, and is supplied to the convolutional neural network 21 during the following period (period n+1). The convolutional neural network 21 performs classification of the supplied pseudo-image during period n+1 and outputs the classification result. The detection control unit 61 operates as described above based on this classification result.

[0104] Thus, in this embodiment, the FFT time width and FFT period can be set to the necessary values ​​depending on the situation. This makes it possible to avoid unnecessarily lengthening the FFT time width or unnecessarily shortening the FFT period, thereby reducing power consumption.

[0105] As described above, according to this embodiment, in addition to the effects of the second embodiment, the adaptability described above makes it possible to reduce the power consumption of the system that monitors the status of the equipment.

[0106] <Seventh Embodiment> A seventh embodiment of the present invention will be described with reference to Figure 7. In this embodiment, the second embodiment is extended to be able to handle general camera images as well. To this end, a selection unit 71 is inserted before the convolutional neural network 21, and an integrated decision unit 72 is added after it. The spectral intensity conversion unit 11, quantization unit 12, and pseudo-image generation unit 13 are the same as in the first and second embodiments, so their description will be omitted.

[0107] As described above, a pseudo-image is output from the pseudo-image generation unit 13. The selection unit 71 receives the pseudo-image and a camera image (RGB image, RGBD image, grayscale image, monochrome image, etc., acquired by an imaging device such as a camera) as input. Based on the input selection signal, the selection unit 71 selects and outputs either the pseudo-image or the camera image. The selection signal is also input to the convolutional neural network 21.

[0108] If the selection signal selects a pseudo-image, the selection unit 71 selects and outputs the pseudo-image. Furthermore, based on the selection signal's instructions, weight parameters and other settings for classifying the pseudo-images are set in the convolutional neural network 21. These weight parameters and other settings are assumed to have been acquired in advance using the learning method described in Figure 2(c). As a result, the convolutional neural network 21 performs the classification of the pseudo-images and outputs the classification results to the subsequent integrated decision unit 72.

[0109] On the other hand, if the selection signal selects a camera image, the selection unit 71 selects and outputs the camera image. In addition, based on the instructions of the selection signal, the convolutional neural network 21 is set with pre-trained weight parameters for classifying the camera image. As a result, the convolutional neural network 21 performs the classification of the camera image and outputs the classification result to the subsequent integrated decision unit 72.

[0110] The classification results, similar to those for pseudo-images, represent the probability of belonging to each appearance state. These appearance states include, for example, a normal appearance, a state with minor cracks, a state with minor discoloration, a state with obvious cracks, and a state with obvious discoloration. The weight parameters are pre-trained to enable classification of these states.

[0111] The selection signal may switch between the pseudo-image and the camera image each time, or it may switch after a certain period of time has elapsed. As a result, the integrated decision unit 72 receives the classification results of the pseudo-image and the camera image alternately.

[0112] The integrated judgment unit 72 determines and outputs a final classification result regarding the equipment's status based on the classification results of the simulated image and the camera image. For example, if the classification results of the simulated image show the highest probability of belonging to the second normal state, and the classification results of the camera image show the highest probability of belonging to a normal appearance state, the final classification result output is that the equipment is in the second normal state.

[0113] Furthermore, for example, if the classification result of the simulated image has the highest probability of belonging to the first abnormal state, and the classification result of the camera image has the highest probability of belonging to a state with obvious cracks or obvious discoloration, the final classification result will output that the equipment is in the first abnormal state. These are cases where the classification result of the simulated image and the classification result of the camera image clearly support the same conclusion.

[0114] Furthermore, for example, if the classification result of the simulated image has the highest probability of belonging to the first normal state, and the classification result of the camera image has the highest probability of belonging to a state other than a normal appearance, and the value of the probability of belonging to the first normal state is higher than a predetermined value, then the classification result of the simulated image will be given more weight, and the first normal state will be output as the final classification result.

[0115] On the other hand, if the probability of belonging to the first normal state is lower than a predetermined value, and considering that an abnormality is suggested from the classification result of the camera image, the abnormal state with the highest value among the probabilities of belonging to the first abnormal state, the second abnormal state, and the third abnormal state of the pseudo-image classification result (for example, the second abnormal state) is output as the final classification result.

[0116] Furthermore, for example, if the classification result of the simulated image shows the highest probability of belonging to the third abnormal state, and the classification result of the camera image shows the highest probability of belonging to a normal appearance state, then if the probability value of belonging to the third abnormal state is higher than a predetermined value, the classification result of the simulated image will be given more weight, and the third abnormal state will be output as the final classification result.

[0117] On the other hand, if the probability of belonging to the third abnormal state is lower than a predetermined value, and considering that no abnormality has been detected in the classification result of the camera image, the normal state with the higher value (for example, the second normal state) among the probabilities of belonging to the first normal state and the probabilities of belonging to the second normal state in the classification result of the pseudo-image is output as the final classification result.

[0118] The integrated determination unit 72 may determine and output the final classification result using various methods other than those described above.

[0119] As described above, this embodiment allows for the selection of images to be analyzed. Furthermore, in addition to the effects of the second embodiment, by using both the state detection result from a simulated image (i.e., a sensor) and the appearance detection result from a camera image, the state of the equipment can be classified with higher accuracy.

[0120] <Eighth Embodiment> An eighth embodiment of the present invention will be described with reference to Figure 8. Using FFT in the pseudo-image generation process has been advantageous because it allows for obtaining accurate frequency spectra, enabling accuracy improvements through frequency axis processing such as the removal of specific interference frequencies and filtering.

[0121] On the other hand, processing pseudo-images can be done using methods other than FFT to perform basis transformations and obtain spectra, which can yield different effects. Specifically, using the Walsh-Hadamard transform (WHT) to generate spectral intensity eliminates multiplication during spectral intensity transformation, thus reducing circuit size. Also, using the discrete cosine transform (DCT) eliminates the need for window function processing, further reducing circuit size. When using the WHT, the spectrum obtained from the multiplier transformation tends to distribute power more evenly across the entire basis compared to the FFT, requiring high classification performance in subsequent image analysis. When using the DCT, power is concentrated in the low-frequency components, so the image size can be reduced by omitting the high-frequency components to generate pseudo-images.

[0122] Figure 8(a) shows an example of converting a time-series sensor signal to spectral intensity. Depending on the conversion method, the resulting basis and spectral intensity will differ.

[0123] Figure 8(b) shows an example of generating pseudo-images using these methods. Different spectral intensities are obtained with FFT and WHT, resulting in the generation of different images. Different spectra are obtained by using different basis transformations (orthogonal transformations), and by mixing multiple types of these, the number of local features used for classification can be increased, thereby improving classification accuracy. Figure 8(c) shows an example of generating pseudo-images using FFT and WHT. Different accuracy and computational loads can be achieved by mixing multiple types in different proportions, mixing them in the same proportion and arranging them in close proximity, or arranging them grouped by transformation. Although examples using FFT and WHT have been explained, other combinations of types may also be used. Furthermore, three or more types of orthogonal transformations may be used.

[0124] <Ninth Embodiment> A ninth embodiment of the present invention will be described with reference to Figure 9. In the ninth embodiment, classification performance and feature extraction functionality can be improved by generating pseudo-images using multiple basis transformations (orthogonal transformations). On the other hand, if the signal from the sensor is noisy, or if power is concentrated on a specific component after a particular basis transformation, or if the classification performance of the image processing unit is not high, using only a specific transformation may improve the overall classification performance of the process rather than using multiple transformations. Since these depend on the application and sensor, it is possible to optimize performance according to multiple types of applications and image processing by adaptively switching the preprocessing ratio using an allocation control table (allocation table) that specifies which basis transformations of the preprocessing unit to use and in what proportion, depending on the image processing and application.

[0125] Figure 9(a) shows a block diagram of a method in which the allocation of basis transformation processing is switched using an allocation control table 91 in the preprocessing unit. The allocation control table 91 stores allocation ratios of basis transformation combinations according to the application and the algorithm of the image analysis unit, based on prior offline evaluation. The allocation ratios are set based on the loss and classification accuracy when processing test data.

[0126] Figure 9(b) shows a block diagram of the image analysis unit when a neural network is used. In convolutional neural networks, there is a trade-off between classification performance and computational complexity depending on the number of layers, the number of parameters, and the type of network, and the optimal preprocessing ratio differs accordingly. For example, if the classification performance of the neural network is low, a high feature extraction function is required in the pseudo-image generation unit, so a method using FFT or a method mixing multiple orthogonal transforms is more suitable for improving overall processing accuracy than a method using only the Walsh-Hadamard transform. In this embodiment, the allocation control table 91 registers an allocation ratio that maximizes classification accuracy based on test data, network learning loss, and accuracy for at least one of the application, sensor type, and neural network type.

[0127] Figure 9(c) is a block diagram illustrating the method for updating the allocation control table in a system with a learning unit. In Figure 9(c), the allocation ratio is updated to a pre-registered allocation control table 91 using input sensor data as learning data. The system monitors the loss and accuracy degradation of the input sensor data during learning, and if the loss or accuracy degradation exceeds a certain level, it adds table data with the pre-processed allocation ratio changed. In other words, the system performs learning and processing in real time, and the allocation control table 91 is updated by on-chip learning. The change in the allocation ratio involves decreasing the ratio of orthogonal transformations with low classification accuracy on their own, and increasing the ratio of methods with high orthogonal transformation ratios on their own. If there is no change in classification accuracy after the change, the number of orthogonal transformation types is increased.

[0128] <Tenth Embodiment> A tenth embodiment of the present invention will be described with reference to Figure 10. In the anomaly detection device (state detection device) of the present invention, the preferred combination of preprocessing and image processing used for anomaly detection varies depending on the application using anomaly detection, the hardware used for anomaly detection, and the algorithms available on that hardware. Therefore, advice software that configures or sends instructions to configure a suitable preprocessing unit and an arithmetic processing unit that performs image analysis based on the accuracy required for anomaly detection and hardware resources is useful.

[0129] In Figure 10, the anomaly detection device is mounted on or connected to the sensor terminal, and software (advice software) for setting or instructing the setting of the algorithms of at least one anomaly detection device's pre-processing unit and calculation processing unit is installed on a PC (personal computer). The PC with the advice software may be located via a network, and an appropriate system configuration is adopted depending on the detection accuracy sent from the anomaly detection device, the frequency of hardware resource updates, and the data size of the parameters sent from the advice software.

[0130] <Hardware Configuration> Next, an example of the hardware configuration of the state detection device will be described with reference to Figure 11. The state detection device shown in Figure 11 comprises a plurality of sensors (81a, 81b), a camera 82, a plurality of A / D converters (83a, 83b), a calculation unit, an input device 89, and an output device 90.

[0131] As described above, sensors are placed on or near the equipment to monitor the equipment's condition and detect predetermined physical quantities. In this example, two sensors (81a, 81b) are placed, but more sensors may be placed. The camera 82 is placed to monitor the equipment's condition, similar to the sensors, and acquires camera images by capturing images. A / D converters (83a, 83b) are used to convert the analog output of the sensors into digital signals. If the sensors (81a, 81b) output digital signals, the A / D converters (83a, 83b) may be omitted.

[0132] Next, the arithmetic unit will be described. The arithmetic unit performs calculations based on the input data, including spectral intensity conversion, quantization, pseudo-image generation, and image analysis (processing by the convolutional neural network 21), as described above. In this example, the arithmetic unit is equipped with a general-purpose processor 85 and a dedicated circuit 86 as the main configuration for data processing (processing unit 84), and uses these as appropriate to perform data processing.

[0133] In this example, the general-purpose processor 85 is a CPU (Central Processing Unit), but it is not limited to this, and other semiconductor devices may be used, for example. The dedicated circuit 86 is used as an accelerator to improve data processing. The dedicated circuit 86 can be configured as appropriate. For example, the dedicated circuit 86 may be a GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), etc. The configuration of the processing unit 84 may be changed as appropriate, as long as it can properly execute the predetermined processing.

[0134] The processing unit includes a storage unit 87 and a memory unit 88. The storage unit 87 stores appropriate data such as programs used for processing. The storage unit 87 stores, for example, a spectral intensity conversion unit 11, a quantization unit 12, a pseudo-image generation unit 13, an image analysis unit 14, a nonlinear quantization unit 31, an intensity processing unit 41, a multiple pseudo-image generation unit 51, a detection control unit 61, a selection unit 71, and an integrated judgment unit 72 as programs to be executed by the processing unit 84. The storage unit 87 also stores data related to the convolutional neural network 21 used for image analysis. In addition, the storage unit 87 may store data input from sensors (81a, 81b) and a camera 82, as well as data of generated pseudo-images.

[0135] Furthermore, the storage 87 may also store a learning unit 25, which is a program used for processing related to the convolutional neural network 21, and a learning data storage unit 24, which is a memory area for storing learning data. Additionally, the storage 87 may also store an allocation control table 91.

[0136] The storage 87 can be configured using, for example, an HDD (Hard Disk Drive). The memory 88 can be configured using, for example, RAM (Random Access Memory), and the processing unit 84 performs processing (predetermined processing such as spectral intensity conversion, quantization, pseudo-image generation, and image analysis) based on the programs and data loaded into the memory 88.

[0137] The input device 89 is used by the user to make various settings related to the operation of the device, and is configured using a keyboard, mouse, touch panel, etc. as appropriate. The output device 90 is used to display the classification results of the equipment status and the user's input, and can be configured using a display, etc. as appropriate.

[0138] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and various design modifications can be made without departing from the spirit of the invention as described in the claims. For example, the embodiments described above are described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations. Furthermore, unless otherwise specified, each component may be singular or plural.

[0139] Although examples of multiple sensors being deployed have been specifically described, even when only one sensor is deployed, the state detection device may perform the same processing as described above to detect the state of the equipment. That is, the state detection device may convert the time-series sensor signal obtained from a single sensor into a spectral intensity vector, generate a pseudo-image, and perform image analysis. Here, even when processing data from a single sensor, the state detection device may perform the same processing as described above (for example, generating a pseudo-image by dividing the spectral intensity vector, generating a pseudo-image by repeating the spectral intensity vector (or spectral intensity sub-vector), nonlinear quantization, etc.).

[0140] The selection signal input to the selection unit 71 may be made by a user using the input device 89. The selection unit 71 may then switch images according to the user's input. In this case, the image can be switched to the image intended by the user.

[0141] The PC (computer) shown in Figure 11 can be configured as appropriate using known components such as a CPU, memory, and interfaces. Alternatively, a computer other than a PC may be used. [Explanation of Symbols]

[0142] 11: Spectral intensity conversion unit 12: Quantization section 13: Pseudo image generation section 14: Image Analysis Department 21: Convolutional Neural Networks 22: Feature extraction unit 23: Classification section 24: Training data storage unit 25: Learning Department 31: Nonlinear Quantization Unit 41: Strength processing section 51: Multiple pseudo-image generation unit 61: Detection Control Unit 71: Selection Section 72: Integrated Judgment Department 81a, 81b: Sensors 82: Camera 83a, 83b: A / D converter 84: Processing Unit 85: General-purpose processor 86: Dedicated circuit 87: Storage 88: Memory 89: Input device 90: Output device 91: Allocation control table

Claims

1. Multiple sensors, A calculation unit that outputs data detected by the aforementioned multiple sensors, Equipped with, The aforementioned arithmetic unit, A processing unit that performs data processing is provided, The aforementioned processing apparatus is The digitized time-series signal data from the aforementioned multiple sensors is converted into data relating to spectral intensity. Based on the aforementioned spectral intensity data, a pseudo-image is generated. The aforementioned simulated image is analyzed to output a classification result of the equipment's condition. A state detection device characterized by the following features.

2. A state detection device according to claim 1, The aforementioned processing apparatus is The above transformation is performed by the Discrete Fourier Transform or the Fast Fourier Transform. A state detection device characterized by the following features.

3. A state detection device according to claim 1, The aforementioned processing apparatus is The aforementioned pseudo-image is analyzed using a convolutional neural network. A state detection device characterized by the following features.

4. A state detection device according to claim 2, The aforementioned processing apparatus is The aforementioned pseudo-image is analyzed using a convolutional neural network. A state detection device characterized by the following features.

5. A state detection device according to claim 4, The aforementioned processing apparatus is In the above conversion, a process is performed to nonlinearly convert the value corresponding to the spectral intensity. A state detection device characterized by the following features.

6. A state detection device according to claim 5, The aforementioned processing apparatus is In the above conversion, a process is performed to replace some of the values ​​of the data relating to spectral intensity. A state detection device characterized by the following features.

7. A state detection device according to claim 5, The aforementioned multiple observations Multiple sensor groups are configured, each containing a different type of sensor. The aforementioned group of observations is They are each placed in different inspection locations. The aforementioned processing apparatus is Multiple of the aforementioned pseudo-images are generated, In this process, the spectral intensity data for the same type of sensor is placed in the same region among the multiple pseudo-images being generated. A state detection device characterized by the following features.

8. A state detection device according to claim 5, The aforementioned multiple observations Multiple sensor groups are configured, each containing a different type of sensor. The aforementioned group of observations is They are each placed in different inspection locations. The aforementioned processing apparatus is Multiple of the aforementioned pseudo-images are generated, In this process, the spectral intensity data for sensors placed at the same inspection location is placed in the same region among the multiple pseudo-images generated. A state detection device characterized by the following features.

9. A state detection device according to claim 4, The aforementioned processing apparatus is Based on the output of the aforementioned convolutional neural network, the time width or period of the Discrete Fourier Transform or Fast Fourier Transform is set. A state detection device characterized by the following features.

10. A state detection device according to claim 4, The system further comprises a camera for acquiring camera images and an input device for the user, The aforementioned processing apparatus is The convolutional neural network processes the camera image or pseudo-image selected by the user. A state detection device characterized by the following features.

11. Sensors and, A calculation unit that outputs the data detected by the aforementioned sensor, Equipped with, The aforementioned arithmetic unit, A processing unit that performs data processing is provided, The aforementioned processing apparatus is The digitized time-series signal data from the aforementioned sensor is converted into data relating to spectral intensity. Based on the aforementioned spectral intensity data, a pseudo-image is generated. The aforementioned simulated image is analyzed to output a classification result of the equipment's condition. A state detection device characterized by the following features.

12. A state detection device according to claim 11, The aforementioned processing apparatus is The pseudo-image is generated by arranging multiple rows of data relating to the spectral intensity corresponding to the aforementioned sensor. A state detection device characterized by the following features.

13. A state detection device according to claim 11, The aforementioned processing apparatus is The above transformation is performed by the Discrete Fourier Transform or the Fast Fourier Transform. A state detection device characterized by the following features.

14. A state detection device according to claim 11, The aforementioned processing apparatus is The aforementioned pseudo-image is analyzed using a convolutional neural network. A state detection device characterized by the following features.

15. A state detection device according to claim 13, The aforementioned processing apparatus is The aforementioned pseudo-image is analyzed using a convolutional neural network. A state detection device characterized by the following features.

16. A state detection device according to claim 1, The aforementioned processing apparatus is The aforementioned transformation is performed using at least one of the following: discrete cosine transform, discrete sine transform, discrete Fourier transform (which is slower than the Fast Fourier Transform), or Walsh-Hadamard transform. A state detection device characterized by the following features.

17. A state detection device according to claim 1, The aforementioned processing apparatus is The aforementioned transformation is performed by combining at least two of the following: discrete cosine transform, discrete sine transform, discrete Fourier transform, fast Fourier transform, and Walsh-Hadamard transform. A state detection device characterized by the following features.

18. A state detection device according to claim 1, It has an allocation table that stores the allocation of orthogonal transformations used in the aforementioned transformation, The aforementioned processing apparatus is Based on the aforementioned allocation table, the transformation is performed using at least one of the following: discrete cosine transform, discrete sine transform, discrete Fourier transform, fast Fourier transform, and Walsh-Hadamard transform. A state detection device characterized by the following features.

19. A state detection device according to claim 18, The aforementioned processing apparatus is The allocation table is updated based on the information input during on-chip learning. A state detection device characterized by the following features.

20. A system using the state detection device described in claim 1, The system includes a computer that determines the combination of orthogonal transformation and image analysis processing algorithm to be used for the transformation based on the detection accuracy and resource information of the state detection device, and configures the state detection device based on the result, or sends an instruction to configure the state detection device based on the result. A system characterized by the following features.

Images