State detection apparatus and state detection method

Abstract

To provide a state detection apparatus, and the like, that are applied to an edge device and allow detection of a state of a facility, the detection being based on a sensor, by utilizing high resolution of a sensor signal without being affected by a phase of the sensor signal.SOLUTION: A state detection apparatus includes a computation unit that acquires data detected by a sensor. The computation unit includes a processing device that executes data processing. The processing device converts data of a digitized time-series signal from a sensor, into data on frequency spectrum intensity. The processing device converts a partial area in an overall area in which values of data on frequency spectrum intensity are distributed, into data expressed in low bits. The processing device generates a pseudo image, based on the data expressed in low bits. The processing device classifies the pseudo image, based on image recognition, and outputs a result of classification of a state of a facility.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a condition detection device and a condition detection method. [Background technology]

[0002] Infrastructure and plant facilities are aging, and their maintenance and management is an important social issue. In response to this, there is a growing demand for automatic equipment monitoring technology. In recent years, systems that detect external abnormalities (scratches, cracks, etc.) in equipment by applying image recognition AI processing to camera images and satellite images are being put into practical use.

[0003] However, methods that use only images have issues with the limitations of image resolution, such as the detection limit and the fact that only external abnormalities can be detected. Therefore, there is a need for technology that can detect the condition of equipment with high accuracy and reliability by analyzing the time-series signals of various sensors attached to or placed near the equipment.

[0004] However, in the past, analysis of time-series signals required the manual creation of dedicated analysis algorithms according to the type of sensor, the characteristics of the signal, the type of state to be detected, etc. As a result, it was not easy to build a system in a short period of time and at low cost.

[0005] On the other hand, there are also known efforts to identify conditions by processing time-series signals from sensors using AI such as deep learning. However, deep learning, which can generally handle time-series signals, generally has the problem that it is not easy to train a neural network and requires a lot of time and effort.

[0006] Patent Document 1 also discloses a technology for detecting abnormalities in equipment by arranging time-series signals from sensors, converting them into pseudo RGB images, and analyzing these images with an image recognition AI that uses a convolutional neural network and a support vector machine. The convolutional neural network extracts features from the above images, and the support vector machine judges the presence or absence of abnormalities (binary judgment) based on these features.

[0007] Patent Document 1 enables the application of image recognition AI that is easy to learn by replacing anomalies in a time series signal with anomalies in an image. However, in the case of a sensor such as a vibration sensor, the phase (time delay) of the sensor signal and the phase difference (time difference) between the sensor signals can have countless variations, which results in countless variations in the converted image. This creates new challenges in ensuring the accuracy of learning and inference.

[0008] Furthermore, to promote widespread use in actual fields, it is necessary to be able to build a condition detection system based on low-cost edge devices with limited hardware resources. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2020-144619 A Summary of the Invention [Problem to be solved by the invention]

[0010] Recent advances in A / D conversion circuit technology have made it possible to obtain high-resolution sensor signals. However, in order to implement image recognition AI on edge devices, it is necessary to reduce the representation word length (resolution) of images, which creates the problem of not being able to fully utilize the high resolution of sensor signals.

[0011] In view of the above circumstances, an object of the present invention is to provide a status detection device or the like that can be applied to an edge device and can realize sensor-based equipment status detection without being affected by the phase of the sensor signal and by taking advantage of the high resolution of the sensor signal. [Means for solving the problem]

[0012] According to a first aspect of the present invention, there is provided a condition detection device as follows. The condition detection device includes a calculation unit that acquires data detected by a sensor. The calculation unit includes a processing unit that executes data processing. The processing unit converts digitized time-series signal data of the sensor into data related to frequency spectrum intensity. The processing unit converts a portion of an entire area in which values ​​of the data related to frequency spectrum intensity are distributed into low-bit represented data. The processing unit generates a pseudo image based on the low-bit represented data. The processing unit classifies the pseudo image based on image recognition, and outputs a classification result of the equipment condition.

[0013] According to a second aspect of the present invention, there is provided a condition detection method as follows. This method includes a step in which a processing device that performs data processing converts digitized time-series signal data of a sensor into data related to frequency spectrum intensity. This method includes a step in which the processing device converts a part of an entire area in which values ​​of the data related to frequency spectrum intensity are distributed into low-bit represented data. This method includes a step in which the processing device generates a pseudo image based on the low-bit represented data. This method includes a step in which the processing device classifies the pseudo image based on image recognition and outputs a classification result of the equipment condition. Effect of the Invention

[0014] According to the present invention, a status detection device is provided that can be applied to an edge device and can realize sensor-based facility status detection without being affected by the phase of the sensor signal and by taking advantage of the high resolution of the sensor signal. Note that problems, configurations, and effects other than those described above will become clear from the following description of the embodiment of the invention. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram showing an example of the configuration of a status detection device according to a first embodiment. [Diagram 2] 4 is a diagram for explaining an example of a pseudo image generation process in the first embodiment. [Figure 3(a)] 11 is a diagram for explaining an example of operation when a range shift is applied; [Figure 3(b)] 11 is a diagram for explaining an example of operation when a range shift is not applied. [Figure 4(a)] 11 is a diagram for explaining an example of an operation relating to a range shift. [Figure 4(b)] 4 is a diagram showing a configuration example of a range shift unit. [Figure 5(a)] 13 is a diagram showing an example of the configuration of a status detection device according to a second embodiment. [Figure 5(b)] 13 is a diagram for explaining a search for an optimal partial region in relation to an example of an operation in the second embodiment. [Figure 6] 13 is a diagram showing an example of the configuration of a status detection device according to a third embodiment. [Figure 7] 13 is a diagram showing an example of the configuration of a status detection device according to a fourth embodiment. [Figure 8(a)] 13 is a diagram showing an example of the configuration of a status detection device according to a fifth embodiment. [Figure 8(b)] 11 is a diagram for explaining an example of operation when a range compression shift is applied; [Figure 8(c)] 11 is a diagram showing an example of the configuration of a range compression shift unit. [Figure 9(a)] 13 is a diagram showing an example of the configuration of a status detection device according to a sixth embodiment. [Figure 9(b)] 11 is a diagram for explaining an example of operation when a range shift is applied; [Figure 10] 13 is a diagram for explaining a seventh embodiment. [Figure 11(a)] 13 is a diagram for explaining an eighth embodiment. [Figure 11(b)] 13 is a diagram for explaining an eighth embodiment. [Figure 11(c)] 13 is a diagram for explaining an eighth embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of a hardware configuration of a status detection device. [Figure 13] FIG. 2 is a diagram illustrating an example of a hardware configuration of a status detection device. [Figure 14] FIG. 2 is a diagram illustrating an example of a hardware configuration of a status detection device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment is an example for explaining the present invention, and for the sake of clarity, some details have been omitted or simplified. The present invention can be implemented in various other forms. When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts, or when there is no need to distinguish between these multiple components, the subscripts may be omitted. In the embodiments, the processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., CPU, GPU), and performs processing defined by the program while using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the subject of the processing performed by executing the program may be the processor. Similarly, the subject of the processing performed by executing the program may be a controller, device, system, computer, or node having a processor. The program may be installed in the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] First Embodiment The present invention relates to a technology for automatically detecting the state of equipment and the like from a sensor signal, and can contribute to solving social issues such as the maintenance and management of infrastructure equipment. According to the present invention, it is possible to realize image recognition-based state detection that is not affected by the phase of the sensor signal and is suitable for automation and hardware weight reduction. A configuration example of a first embodiment of the present invention is shown in FIG. 1. In order to monitor the state of the equipment, one or more sensors are placed on or near the equipment. The time series signal output by each sensor is assumed to have already been digitized at the stage of sensor output, or to have been digitized by an A / D converter downstream of the sensor. This digitized time series signal (hereinafter, sensor time series signal) is input to a spectrum intensity conversion unit 11.

[0018] The spectrum intensity transform unit 11 transforms the sensor time series signal into a vector related to frequency spectrum intensity, for example, by discrete Fourier transform or fast Fourier transform (FFT). That is, by discrete Fourier transform or FFT, the signal is transformed into information of complex components I+j*Q at each frequency position, and further, from the complex components, for example, √(I 2 +Q 2) to calculate the spectral intensity. Note that a window function may be applied, as is commonly done in discrete Fourier transform and fast Fourier transform. In this way, the spectral intensity at each frequency position is obtained as a spectral intensity vector. The spectral intensity conversion unit 11 outputs this spectral intensity vector for each sensor.

[0019] Each of the spectral intensity vectors is input to the range shift unit 12. The range shift unit 12 reduces the representation word length for each element value of the spectral intensity vector (i.e., the value of the frequency spectral intensity) to, for example, 8 bits, so as to match the representation word length of a pseudo image (described later), and outputs the reduced length. As a result, the number of spectral intensity vectors represented in low bits is output, the number of which is the number of sensors.

[0020] Each of the low-bit-represented spectral intensity vectors is input to the pseudo image generating unit 13. The pseudo image generating unit 13 treats one low-bit-represented spectral intensity vector as one row, and vertically arranges the low-bit-represented spectral intensity vectors corresponding to each sensor to generate a two-dimensional image. This image is a pseudo image (pseudo image) because it is not a normal image such as a camera image. Therefore, the representation word length of this pseudo image is equal to the representation word length reduced by the range shifting unit 12. By reducing the representation word length of the pseudo image in this way, the amount of calculation in the image recognition AI unit 14 can be significantly reduced, making it possible to implement it on an edge device with limited hardware resources.

[0021] The pseudo image is input to the image recognition AI unit 14. The image recognition AI unit 14 is, for example, a convolutional neural network, which regards the pseudo image as an image, classifies the image, and outputs the classification result. This allows a classification result to be obtained for the state of the equipment monitored by the sensor (for example, a first normal state, a second normal state, a first abnormal state, a second abnormal state, a third abnormal state, etc.). The convolutional neural network is trained in advance using pseudo images corresponding to these states.

[0022] The image recognition AI unit 14 may be a neural network other than a convolutional neural network, or may execute a machine learning algorithm other than a neural network. Similarly, the image recognition AI unit 14 performs pre-learning and then classifies images.

[0023] This embodiment will be described in more detail with reference to Fig. 2. As described above, each sensor time series signal is subjected to a discrete Fourier transform or a fast Fourier transform (FFT) to obtain a spectral intensity vector. The spectral intensity vector is data in which the values ​​of frequency spectral intensity at each frequency position are arranged horizontally. The number of dimensions of the vector (i.e., the number of frequency positions) is equal to half the number of points of the discrete Fourier transform or fast Fourier transform. Fig. 2 shows a case in which the number of transform points is 16, and the number of dimensions of the vector is 8, but any number of points may be appropriately selected.

[0024] The number of points of the transformation determines the time width of the sensor signal used to generate each pseudo image. That is, this time width is equal to the product of the time interval of the sensor time-series signal and the number of points. Also, the frequency resolution (the interval between each frequency position) is equal to the reciprocal of the time width.

[0025] Fig. 2 shows the spectral intensity vectors when the expression word length is reduced to 8 bits in the range shift unit 12. Since it is an 8-bit expression, it is expressed by integer values ​​from 0 to 255. In Fig. 2, the spectral intensity vector expressed in low bits (8 bits) corresponding to the first sensor is (85, 188, 128, 254, 171, 218, 149, 104). Moreover, the spectral intensity vector expressed in low bits (8 bits) corresponding to the second sensor is (64, 89, 41, 112, 194, 168, 235, 213).

[0026] When generating a pseudo image, the pseudo image generating unit 13 treats the spectral intensity vectors expressed in low bits as one row and arranges them vertically, but the spectral intensity vectors expressed in low bits corresponding to one sensor may be arranged repeatedly in multiple rows as shown in Fig. 2. In Fig. 2, the vectors are arranged twice. That is, the spectral intensity vectors expressed in low bits corresponding to a first sensor are arranged in the first and second rows of the pseudo image, and the spectral intensity vectors expressed in low bits corresponding to a second sensor are arranged in the third and fourth rows, and so on until the pseudo image is completed.

[0027] The horizontal size (number of pixels) of the pseudo image generated by the above is equal to the number of dimensions of the low-bit representation of the spectral intensity vector (8 pixels in Fig. 2), while the vertical size (number of pixels) is equal to the product of the number of sensors and the number of repetitions.

[0028] Furthermore, the number of points for the discrete Fourier transform or fast Fourier transform may be, for example, 32. In this case, the number of dimensions of each spectral intensity vector is 16. Therefore, if it is desired to set the width of the pseudo image to, for example, 8 pixels, each spectral intensity vector is divided into two spectral intensity subvectors (a first spectral intensity subvector and a second spectral intensity subvector) each having a dimension of 8, and each of these is used as one line of the pseudo image. Furthermore, the above repetitions may be combined.

[0029] For example, the first row of the pseudo image may contain the first spectral intensity subvector of the first sensor, the second row may repeat this, the third row may contain the second spectral intensity subvector of the first sensor, the fourth row may repeat this, the fifth row may contain the first spectral intensity subvector of the second sensor, the sixth row may repeat this, the seventh row may contain the second spectral intensity subvector of the second sensor, the eighth row may repeat this, etc. This results in an 8x8 pixel pseudo image being generated from the data from the two sensors.

[0030] Other arrangements may also be used. For example, the first spectral intensity subvector of the first sensor may be arranged in the first row of the pseudo image, the first spectral intensity subvector of the second sensor may be arranged in the second row, the first spectral intensity subvector of the second sensor may be arranged in the third row, the second spectral intensity subvector of the first sensor may be arranged in the fourth row, the second spectral intensity subvector of the first sensor may be arranged in the fifth row, the second spectral intensity subvector of the second sensor may be arranged in the sixth row, the second spectral intensity subvector of the second sensor may be arranged in the seventh row, and the second spectral intensity subvector of the second sensor may be arranged in the eighth row. In this way, a pseudo image of 8×8 pixels is generated from the data of the two sensors. Note that pseudo images of sizes other than 8×8 pixels may be generated, and pseudo images of various sizes can be generated by the above repetition and division of the spectral intensity vector. The number of repetitions and the number of divisions of the spectral intensity vector can be appropriately selected according to the size of the pseudo image required. The number of repetitions may be 1 (no repetition).

[0031] As described above, the pixel values ​​of the pseudo image are expressed as frequency spectrum intensities in low bits, and are therefore not affected by the phase of the sensor time-series signal. This makes it easier for the image recognition AI unit 14 to learn, and is believed to improve inference accuracy.

[0032] The operation of the range shift unit 12 will be described in more detail with reference to FIG. 3(a). In many cases, the noise level of the sensor itself is sufficiently low, and the effective resolution of the A / D converter that digitizes the sensor output is also sufficiently high, so the output value of the spectrum intensity conversion unit 11 (i.e., the frequency spectrum intensity at each frequency position) is also expressed with a high number of bits. For example, in FIG. 3(a), it is expressed with a word length of 12 bits, and 4096 (=2000) from 0 to 4095 is expressed. 12 ) values. On the other hand, the representation word length of the pseudo image is reduced to, for example, 8 bits in FIG. 3(a) in order to reduce the amount of calculation in the image recognition AI unit 14. Therefore, the representation word length can be set to 256 (=2 8 ) values. Therefore, the range shift unit 12 focuses on a 256-wide partial region out of the 4096-wide (0 to 4095) region in which the frequency spectrum intensity values ​​of the sensor signal exist, and expresses the frequency spectrum intensity values ​​existing in that partial region in 8 bits (0 to 255). For example, in FIG. 3(a), the partial region (3840 to 4095) located at the top is focused on and converted to an 8-bit representation. Specifically, the conversion is performed by subtracting the lower limit value of the focused partial region (3840 in FIG. 3(a)) from each frequency spectrum intensity value.

[0033] Also, the partial region of interest may be changed as shown by the dashed line in Fig. 3(a). The dashed line in Fig. 3(a) focuses on the partial region (0 to 255) located at the bottom. Note that Fig. 3(a) shows frequency spectrum intensities at eight frequency positions included in one spectrum intensity vector, which is converted to an 8-bit word length as shown in the figure and used for one line of the pseudo image.

[0034] In the above, the conversion to 8-bit word length is performed based on the lower limit value of the partial region, but the conversion may be performed based on any reference value that indicates the partial region. For example, the upper limit value of the partial region may be the reference value, or the value at the center position or any other position within the partial region may be the reference value.

[0035] FIG. 3(b) shows an example when the range shift is not applied. As in FIG. 3(a), the frequency spectrum intensity at each frequency position is expressed with a word length of 12 bits and can take values ​​from 0 to 4095. The representation word length of the pseudo image is 8 bits and can only express values ​​from 0 to 255. In FIG. 3(b), conversion from 12-bit representation to 8-bit representation is performed by range compression. Specifically, 4080 to 4095 in the 12-bit representation is converted to 255 in the 8-bit representation. Furthermore, 4064 to 4079 in the 12-bit representation is converted to 254 in the 8-bit representation, and 4048 to 4063 in the 12-bit representation is converted to 253 in the 8-bit representation. In the same manner, 16 consecutive values ​​in the 12-bit representation are converted to one value in the 8-bit representation, and finally, 0 to 15 in the 12-bit representation is converted to 0 in the 8-bit representation.

[0036] In FIG. 3(b), the spectral intensity vector of the sensor signal is the same as in FIG. 3(a). However, due to the range compression, all four frequency spectral intensity values ​​near the upper limit are rounded to 255. Similarly, all four frequency spectral intensity values ​​near the lower limit are rounded to 0. In this way, when range compression is applied, the resolution of the pseudo image is reduced compared to the resolution originally held by the sensor signal. As a result, slight changes that appear in the sensor signal are not reflected in the pseudo image, and cannot be identified even by image recognition AI. Therefore, the sensitivity of equipment status detection is reduced.

[0037] In contrast, with the range shift of the present invention (FIG. 3(a)), no rounding error occurs during conversion to 8-bit representation. This is because, instead of converting the entire region into a pseudo image as in range compression, a partial region of the same size (256 values) as the 8-bit representation is cut out and converted into a pseudo image. Therefore, with the range shift of this embodiment, slight changes that appear in the sensor signal can be reflected as changes in the pseudo image, making them distinguishable by image recognition AI. As a result, the sensitivity of equipment status detection can be improved.

[0038] The operation of the range shift unit 12 will be described in more detail with reference to FIG. 4. FIG. 4(a) shows a method for handling frequency spectrum intensity values ​​that exist outside the partial region of interest. In FIG. 4(a), frequency spectrum intensity values ​​that exceed the partial region of interest are converted to a first fixed value (255 in FIG. 4(a)) when converted to an 8-bit representation. Also, frequency spectrum intensity values ​​that fall below the partial region of interest are converted to a second fixed value (0 in FIG. 4(a)) when converted to an 8-bit representation. As a result, even if the value of the frequency spectrum intensity that falls outside the partial region of interest changes, it is not reflected on the pseudo image. As a result, only the change in frequency spectrum intensity within the partial region of interest can be reflected on the pseudo image, and the accuracy of learning and inference in the image recognition AI unit 14 can be improved.

[0039] In order to realize the above operation, as shown in Fig. 4(b), the range shift unit 12 includes a subtraction unit 41 and a subsequent outside-range processing unit 42. Each frequency spectrum intensity value is input to the subtraction unit 41, and the lower limit value of the partial region of interest is subtracted as described above. As a result, if the frequency spectrum intensity value is within the partial region of interest, the subtraction result will be between 0 and 255. On the other hand, if the frequency spectrum intensity value is above the partial region of interest, the subtraction result will be greater than 255. Also, if the frequency spectrum intensity value is below the partial region of interest, the subtraction result will be less than 0.

[0040] The outside-range processing unit 42 following the subtraction unit 41 outputs the first fixed value (255 in FIG. 4(a)) if the inputted subtraction result is greater than 255, and outputs a second fixed value (0 in FIG. 4(a)) if the inputted subtraction result is less than 0. Moreover, if the inputted subtraction result is greater than 0 and less than 255, it outputs it as is. This makes it possible to realize the above-mentioned operation. The first fixed value may be a value other than 255 (i.e., 0 to 254), and the second fixed value may be a value other than 0 (i.e., 1 to 255).

[0041] In the above, the lower limit value of the partial region is subtracted, but any reference value indicating the partial region may be subtracted. For example, when the upper limit value of the partial region is subtracted, the frequency spectrum intensity values ​​within the partial region are converted to a value between -255 and 0. Furthermore, frequency spectrum intensity values ​​above the partial region are converted to a value of 1 or more, and frequency spectrum intensity values ​​below the partial region are converted to a value of -256 or less. Therefore, the sign of this conversion result is inverted (i.e., multiplied by -1) before the above-mentioned processing is performed by outside-range processing unit 42.

[0042] In this way, no matter what reference value is selected to indicate the partial region, the main operation of the invention remains the same, so in the following description of the embodiment, only the case where the reference value is the lower limit value will be described.

[0043] In the above explanation, the representation word length for the frequency spectrum intensity of the sensor signal is 12 bits, and the representation word length for the pseudo image is 8 bits, but the present invention can be applied to representation word lengths other than these.

[0044] The operations of the spectrum intensity conversion unit 11, the range shift unit 12, the pseudo image generation unit 13, and the image recognition AI unit 14 may be performed by using an accelerator (dedicated hardware), or may be performed by software processing using a general-purpose processor such as a CPU. Also, internal memory (SRAM, etc.) mounted on an accelerator chip or a CPU chip may be used, or external memory (DRAM, etc.) may be used.

[0045] As described above, according to this embodiment, the equipment status can be classified with high reliability without being affected by the phase of the sensor time-series signal. Furthermore, even when an edge device is applied, the high resolution of the sensor signal can be utilized, so that even slight differences in status can be classified. Therefore, a low-cost, highly reliable, and highly sensitive equipment status detection system using edge devices can be realized.

[0046] <Second embodiment> The configuration of the second embodiment of the present invention is shown in FIG. 5(a). As described above, in the present invention, it is necessary to extract an appropriate partial region from a wide region of the frequency spectrum intensity of the sensor signal. Therefore, in this embodiment, a method for searching for an optimal partial region based on the learning results of an image recognition AI is shown. Note that explanations similar to those already explained may be omitted.

[0047] As shown in Fig. 5(a), similarly to the first embodiment, the sensor time series signal is processed in the spectral intensity conversion unit 11, range shift unit 12, pseudo image generation unit 13, and image recognition AI unit 14 in that order. This embodiment further includes a range designation unit 51, a storage unit 52, and a learning unit 53. The operation timing of this embodiment is shown in Fig. 5(b). The range designation unit 51 designates the lower limit value of the partial region to be focused on to the range shift unit 12 in that order.

[0048] 5(b), for example, 0, 256, 512, 768, and 1024 are specified in that order as lower limit values. As a result, the range shifter 12 converts into 8-bit representation word lengths by focusing on partial regions of 0 to 255, 256 to 511, 512 to 767, 768 to 1023, and 1024 to 1279 in that order, as in the first embodiment. Furthermore, the pseudo image generator 13 generates a pseudo image, as in the first embodiment, based on each spectral intensity vector converted into 8-bit representation word length by the range shifter 12.

[0049] With each of the lower limit values ​​specified, a sensor time-series signal for learning is input to the spectrum intensity conversion unit 11, and corresponding annotation data (correct state) is input to the learning unit 53. For example, a sensor time-series signal acquired in advance in state 1, state 2, etc. of the equipment is input to the spectrum intensity conversion unit 11, and correct states such as "state 1" and "state 2" are input to the learning unit 53 as the corresponding annotation data. As a result, with attention focused on each partial region, pseudo images corresponding to state 1, state 2, etc. are generated as described above, and supplied to the image recognition AI unit 14.

[0050] The image recognition AI unit 14 is, for example, a convolutional neural network, which classifies the supplied pseudo image and outputs a state classification result. The state classification result is expressed as a probability, for example, 70% for state 1 and 30% for state 2. Furthermore, the state classification result is supplied to a learning unit 53. The learning unit 53 compares the supplied annotation data (correct states such as "state 1" and "state 2") with the state classification result, updates parameters such as weights based on the difference, and supplies the result to the convolutional neural network.

[0051] With each lower limit value specified (i.e., with attention focused on each partial region), the above operation repeatedly updates parameters such as weights for many learning sensor time-series signals and corresponding annotation data. After the parameter updates have converged, the learning unit 53 calculates a learning error based on the difference between the state classification result and the annotation data. The learning error is transmitted to the range specification unit 51, which instructs the storage unit 52 on the value to be stored based on the learning error.

[0052] In FIG. 5(b), when the lower limit is 0 (focusing on the partial region from 0 to 255), the learning error is 20, and when the lower limit is changed to 256 (focusing on the partial region from 256 to 511), the learning error increases to 30. The holding unit 52 stores the minimum value of the learning error at that time and the corresponding lower limit. First, 20 is stored as the learning error corresponding to the first lower limit value 0. When the next lower limit value is 256, the learning error increases to 30, so the holding unit 52 maintains the stored contents. As shown in FIG. 5(b), when the next lower limit value is 512 (focusing on the partial region from 512 to 767), the learning error is 10, which is a smaller value than the held 20. Therefore, the holding unit 52 updates the held value and holds 10 as the learning error and 512 as the corresponding lower limit. Furthermore, in the next state where the lower limit value is 768 (focusing on the partial region from 768 to 1023), the learning error becomes 5, which is a smaller value than the held 10. Therefore, the holding unit 52 updates the held value, and holds 5 as the learning error and 768 as the corresponding lower limit value. Finally, in the state where the lower limit value is 1024 (focusing on the partial region from 1024 to 1279), the learning error becomes 15, which is a larger value than the held 5. Therefore, the holding unit 52 maintains the held value. With the above, the learning period ends, and the lower limit value of 768 corresponding to the smallest learning error is finally held in the holding unit 52.

[0053] After the learning period ends, it is time to execute equipment status detection. Prior to execution, parameters such as weights of the convolutional neural network of the image recognition AI unit 14 are re-learned. For this purpose, the sensor time series signal for learning is input to the spectrum intensity conversion unit 11, and the corresponding annotation data is input to the learning unit 53. In addition, the range instruction unit 51 instructs the range shift unit 12 to use 768, which is the lower limit value (the lower limit value that minimizes the learning error) searched for during the learning period, as the lower limit value of the optimal partial region.

[0054] Thereby, as described above, a pseudo image is generated focusing on the partial region corresponding to the lower limit value 768 (partial region from 768 to 1023), and a state classification result is obtained by the convolutional neural network. The learning unit 53 repeatedly updates parameters such as weights based on the state classification result and annotation data as described above, and after the updates converge, the convolutional neural network holds the final parameters such as weights, and becomes capable of executing state detection.

[0055] Therefore, an actual (not for learning) sensor time-series signal is input to the spectrum intensity conversion unit 11, and based on that, a pseudo image focusing on the optimal partial region (in this case, the partial region from 768 to 1023) is generated and classified by the convolutional neural network. The convolutional neural network outputs the state classification result as described above, and this makes it possible to know the probability that the target equipment is in each state.

[0056] In the above, the image recognition AI unit 14 may be a neural network other than a convolutional neural network, and may perform learning and inference using a machine learning algorithm other than a neural network.

[0057] The operations of each block in FIG. 5(a) may be performed on the same edge device. Also, the operations during the execution period in FIG. 5(b) may be performed on the edge device, and the operations during the learning period may be performed on another device (an external computer such as a server) with higher computing power. In that case, the operations of each block in FIG. 5(a) are performed on the other device, and the operations of the spectrum intensity conversion unit 11, range shift unit 12, pseudo image generation unit 13, and image recognition AI unit 14 are performed on the edge device. Parameters such as the lower limit value of the optimal partial region and the final weighting obtained by the operations on the other device during the learning period may be transmitted to the edge device prior to the execution period. Also, the other device may perform only a part of the operations in FIG. 5(a).

[0058] According to this embodiment, an optimal partial region can be automatically searched for from a wide region of the frequency spectrum intensity of the sensor signal, so that an equipment status detection system can be easily constructed.

[0059] <Third embodiment> A third embodiment of the present invention will be described with reference to FIG. 6. Note that the same description as that already described may be omitted. In this embodiment, as in the second embodiment, an optimal partial region is automatically searched for. However, while in the second embodiment, the search is based on the learning results of the image recognition AI unit, in this embodiment, the search is performed by analyzing a pseudo image.

[0060] 6, similarly to the first embodiment, the sensor time series signal is processed in order by the spectrum intensity conversion unit 11, the range shift unit 12, the pseudo image generation unit 13, and the image recognition AI unit 14. Also, similarly to the second embodiment, the present embodiment includes a range designation unit 51, a storage unit 52, and a learning unit 53. Furthermore, in this embodiment, an image comparison unit 61 is provided.

[0061] The range designation unit 51 designates the lower limit values ​​of the partial regions to be focused on in order to the range shift unit 12. For example, as in FIG. 5(b), 0, 256, 512, 768, and 1024 are designated as the lower limit values ​​in order. As a result, as in the second embodiment, the range shift unit 12 focuses on partial regions of 0 to 255, 256 to 511, 512 to 767, 768 to 1023, and 1024 to 1279 in order and converts them into 8-bit representation word lengths. Furthermore, the pseudo image generation unit 13 generates a pseudo image as in the first embodiment based on each spectral intensity vector converted into 8-bit representation word length by the range shift unit 12.

[0062] With each of the lower limit values ​​specified, a sensor time-series signal for learning is input to the spectrum intensity conversion unit 11. For example, sensor time-series signals acquired in advance in state 1, state 2, etc. of the equipment are input to the spectrum intensity conversion unit 11. As a result, with attention focused on each partial region, pseudo images corresponding to state 1, state 2, etc. are generated by the pseudo image generation unit 13 as described above.

[0063] In this embodiment, the generated pseudo image is also supplied to an image comparison unit 61. The image comparison unit 61 calculates, for example, an "image difference amount" between a pseudo image corresponding to state 1 and a pseudo image corresponding to state 2. The image difference amount may be any index that quantitatively indicates the difference between images, such as a value obtained by squaring the difference in pixel values ​​at the same position in two images and accumulating the result for all pixels.

[0064] Image comparison unit 61 calculates the image difference amount for an average pseudo image of state 1 obtained by calculating an average value for each pixel of a plurality of pseudo images corresponding to state 1, and an average pseudo image of state 2 obtained by calculating an average value for each pixel of a plurality of pseudo images corresponding to state 2. The image difference amount is transmitted to range designation unit 51, which designates a value to be stored to storage unit 52 based on the image difference amount.

[0065] It is expected that the greater the image difference between the pseudo images in different states, the easier it will be to classify the image in the image recognition AI unit 14 and the higher the classification accuracy will be. Therefore, in this embodiment, a partial region in which the image difference between the states is the largest is searched for.

[0066] In each of the lower limit value states, the image difference amount is obtained as described above. The storage unit 52 first stores the first lower limit value 0 and the corresponding image difference amount. In the subsequent lower limit value state of 256, if the image difference amount is greater than the stored image difference amount (i.e., the image difference amount in the state where the lower limit value is 0), the storage unit 52 updates the stored value and stores the lower limit value 256 corresponding to the image difference amount. On the other hand, if the image difference amount is equal to or less than the stored image difference amount, the storage unit 52 maintains the stored value. Similarly, in the subsequent lower limit value states, if the image difference amount is greater than the stored image difference amount (the maximum image difference amount up to that point), the storage unit 52 updates the stored value, and otherwise maintains the stored value. As a result, after calculations at each lower limit value, the lower limit value that maximizes the image difference amount is stored in the storage unit 52. With the above, the search period for the optimal partial region ends.

[0067] After the search period for the optimal partial region ends, it is time to execute equipment state detection. The image recognition AI unit 14 is, for example, a convolutional neural network, and learns parameters such as weights prior to execution. For this purpose, the sensor time series signal for learning is input to the spectrum intensity conversion unit 11, and the corresponding annotation data is input to the learning unit 53. In addition, the range instruction unit 51 instructs the range shift unit 12 to use the lower limit value held in the holding unit 52 (i.e., the lower limit value that maximizes the image difference amount) as the lower limit value of the optimal partial region.

[0068] As a result, a pseudo image is generated focusing on that partial region, and a state classification result is obtained by the convolutional neural network. The learning unit 53 repeatedly updates parameters such as weights based on the state classification result and annotation data, and after the updates converge, the convolutional neural network holds the final parameters such as weights, and becomes capable of executing state detection.

[0069] Therefore, actual (not for learning) sensor time series signals are input to the spectrum intensity conversion unit 11, and based on them, a pseudo image focusing on the optimal partial region is generated and classified by the convolutional neural network. The convolutional neural network outputs the state classification results as described above, and this makes it possible to know the probability that the target equipment is in each state.

[0070] In the above, the image recognition AI unit 14 may be a neural network other than a convolutional neural network, and may perform learning and inference using a machine learning algorithm other than a neural network.

[0071] The operations of each block in FIG. 6 may be performed on the same edge device. Also, the operations during the execution period may be performed on the edge device, and the search for the optimal partial region and the learning of the image recognition AI may be performed on another device (an external computer such as a server) with higher computing power. In that case, the operations of each block in FIG. 6 are performed on the other device, and the operations of the spectrum intensity conversion unit 11, the range shift unit 12, the pseudo image generation unit 13, and the image recognition AI unit 14 are performed on the edge device. Parameters such as the lower limit value of the optimal partial region and the final weighting obtained by the operations on the other device may be transmitted to the edge device prior to the execution period. Also, the other device may perform only a part of the operations in FIG. 6.

[0072] According to this embodiment, similarly to the second embodiment, an optimal partial region can be automatically searched for from a wide region of the frequency spectrum intensity of the sensor signal, and therefore, an equipment status detection system can be easily constructed.

[0073] Furthermore, in the second embodiment, the image recognition AI needs to be trained for many partial regions (lower limit values) each time, but in this embodiment, it is sufficient to train the searched partial region only once. Therefore, the learning period can be shortened. On the other hand, since the partial region where the learning error is actually minimized is not searched for as in the second embodiment, the learning error of the searched partial region may not be minimized. From the above, when shortening the learning period is prioritized, the third embodiment is considered to be suitable, and when the accuracy of the equipment status classification is prioritized, the second embodiment is considered to be suitable.

[0074] <Fourth embodiment> A fourth embodiment of the present invention will be described with reference to FIG. 7. Note that the same description as that already described may be omitted. In this embodiment, an analog signal (voltage signal or current signal) of a sensor is processed by an AFE unit (analog front-end circuit) that can be configured using, for example, an amplifier, a filter, an A / D converter, etc. The signal is digitized by the last A / D converter in the AFE unit to become a sensor time-series signal.

[0075] As shown in FIG. 7, an analog signal of the sensor is input to the selection unit 71. A pilot signal is also input to the selection unit 71. One of these is selected by the selection unit 71, and the AFE unit 72 performs the above-mentioned processing. The sensor time-series signal digitized by the AFE unit 72 is processed in order by the spectrum intensity conversion unit 11, the range shift unit 12, the pseudo image generation unit 13, and the image recognition AI unit 14, as in the previous embodiments. Also, as in the second embodiment, the range designation unit 51, the holding unit 52, and the learning unit 53 are provided. As in the second embodiment, during the learning period, after the lower limit value of the optimal partial region is searched for, re-learning is performed with the partial region being focused on, and the equipment state detection can be performed.

[0076] In this embodiment, a range adjustment unit 73 is provided between the range indication unit 51 and the range shift unit 12. The range adjustment unit 73 adjusts the lower limit value of the partial region supplied from the range indication unit 51, and transmits the adjusted lower limit value to the range shift unit 12. Note that in the operation until the equipment status detection can be executed, the range adjustment unit 73 transmits the lower limit value of the partial region supplied from the range indication unit 51 to the range shift unit 12 as is.

[0077] This embodiment shows a method for dealing with gain fluctuations occurring in the AFE unit 72 after equipment status detection starts. The gain of analog circuits such as the amplifier, filter, and A / D converter included in the AFE unit 72 fluctuates due to fluctuations in temperature and power supply voltage. The amplitude of the digitized sensor time-series signal fluctuates in proportion to this gain fluctuation. Therefore, the value of each element of the spectrum intensity vector output by the spectrum intensity conversion unit 11 (i.e., the value of the frequency spectrum intensity at each frequency position) also fluctuates uniformly in proportion to the gain fluctuation.

[0078] The range adjustment unit 73 has a role of absorbing the influence of fluctuations in the spectral intensity vector. That is, when the gain of the AFE unit 72 increases and the values ​​of each element of the spectral intensity vector increase, the range adjustment unit 73 increases the lower limit value after the adjustment accordingly. On the other hand, when the gain of the AFE unit 72 decreases and the values ​​of each element of the spectral intensity vector decrease, the range adjustment unit 73 decreases the lower limit value after the adjustment accordingly.

[0079] In order to monitor the gain fluctuation of the AFE unit 72, the selection unit 71 periodically selects the pilot signal. The pilot signal is, for example, a signal of a known single frequency located within the passband of a filter in the AFE unit 72. This pilot signal is affected by the gain fluctuation in the AFE unit 72 in the same manner as the analog signal of the sensor, and then converted into a frequency spectrum intensity value in the spectrum intensity conversion unit 11. The frequency spectrum intensity value is input to the range adjustment unit 73, which estimates the gain fluctuation of the AFE unit 72 from the change in this frequency spectrum intensity value. Furthermore, based on the estimation result, the lower limit value after adjustment is determined as described above, and transmitted to the range shift unit 12.

[0080] The range shift unit 12 focuses on the partial region corresponding to the adjusted lower limit value and generates a spectral intensity vector expressed in low bits. This minimizes the change in the spectral intensity vector expressed in low bits (e.g., 8 bits). Therefore, the change in the pseudo image generated by the pseudo image generation unit 13 is also minimized, so that the deterioration of the classification accuracy in the image recognition AI unit 14 can be minimized.

[0081] When there are a plurality of sensors, the analog signal of each sensor is selected in turn by the selection unit 71 and supplied to the AFE unit 72. At that time, the pilot signal may be selected periodically as described above.

[0082] The operations of each block in FIG. 7, or the operations of each block (i.e., digital circuit unit) after the spectrum intensity conversion unit 11, may be performed on the same edge device. Also, the operations during the execution period may be performed on the edge device, and the operations during the learning period may be performed on another device (an external computer such as a server) with higher computing power. In that case, the operations of each block after the spectrum intensity conversion unit 11 in FIG. 7 are performed on the other device, and the operations of the spectrum intensity conversion unit 11, range shift unit 12, pseudo image generation unit 13, image recognition AI unit 14, and range adjustment unit 73 are performed on the edge device. Parameters such as the lower limit value of the optimal partial region and the final weighting obtained by the operations on the other device during the learning period may be transmitted to the edge device prior to the execution period. Also, the other device may perform only a part of the operations in FIG. 7.

[0083] According to this embodiment, even if the gain of the AFE unit (analog front-end circuit) that processes the analog signal of the sensor varies over time due to changes in temperature and power supply voltage, the deterioration of the classification accuracy of the equipment status can be minimized. Therefore, the implementation of the AFE unit becomes easy.

[0084] <Fifth embodiment> A fifth embodiment of the present invention will be described with reference to FIG. 8. Note that the same description as that already described may be omitted. This embodiment shows a method for reducing the total number of partial regions. By reducing the total number of partial regions, it is possible to shorten the search time for the optimal partial region in the second and third embodiments. In this embodiment, a range compression shift unit 81 is used instead of the range shift unit 12 in order to reduce the total number of partial regions.

[0085] As shown in FIG. 8(a), the sensor time series signal is processed in the order of the spectrum intensity conversion unit 11, the range compression shift unit 81, the pseudo image generation unit 13, and the image recognition AI unit 14. The range compression shift unit 81 is based on the operation of the range shift unit 12, but for the upper partial area, the range is set wider than the expression word length of the pseudo image (8 bits in the figure). For example, in FIG. 8(b), the range of the partial area located at the top is 512 (3584 to 4095), which is twice the expression range of the pseudo image, 256 (0 to 255). Therefore, in addition to the range shift, the uppermost partial area also requires range compression with a compression rate of 1 / 2. As a result, when the frequency spectrum intensity values ​​of the sensor signal are 4094 and 4095, the pixel value of the pseudo image is converted to 255. Also, when the frequency spectrum intensity values ​​are 4092 and 4093, the pixel value of the pseudo image is 254. Similarly, when the frequency spectrum intensity values ​​are 3584 and 3585, the pixel value of the pseudo image is 0. In this way, two different frequency spectrum intensity values ​​are converted into the same pixel value, resulting in a rounding error.

[0086] On the other hand, as shown in FIG. 8(b), the lowermost partial region is set to 0 to 255, which is the same as the representation range of the pseudo image. Therefore, range compression is not necessary. In this embodiment, the total number of partial regions can be reduced by making the other upper partial regions wider. Instead, as described above, rounding errors occur when converting the upper partial region into a pseudo image. However, unlike FIG. 3(b), it is not necessary to fit the entire range of the frequency spectrum intensity of the sensor signal into the representation range of the pseudo image. Therefore, compression is minor, and rounding errors are also minor. Furthermore, since the frequency spectrum components present in the upper partial region have a large intensity, it is expected that the intensity change due to the change in the equipment state is also large. Therefore, it is expected that there will be no influence of minor rounding errors.

[0087] 8(c) also shows the configuration of range compression shift unit 81. In order to achieve the above operation, in addition to the configuration of range shift unit 12, a multiplication unit 82 that multiplies the compression rate is provided after subtraction unit 41. That is, in subtraction unit 41, the lower limit value of the partial region of interest is subtracted from the frequency spectrum intensity value, and further, multiplication unit 82 multiplies the value by the compression rate. Thereafter, as in the first embodiment, outside-range processing unit 42 performs processing on the frequency spectrum intensity values ​​that fall outside the partial region of interest.

[0088] For example, in the example of Fig. 8(b), when focusing on the uppermost partial region, the lower limit is set to 3584, the compression rate is set to 1 / 2, the subtraction and multiplication are performed, and the multiplication result is truncated to an integer value. When focusing on the lowermost partial region, the lower limit is set to 0, and the compression rate is set to 1. The other partial regions can be processed in the same manner.

[0089] In this embodiment, the range of the upper partial region is set to twice the representation range of the pseudo image, but in order to further reduce the total number of partial regions, the range of the partial region may be set to be wider. Also, the range of the partial region may be gradually increased toward the top.

[0090] As described above, in this embodiment, by reducing the total number of partial regions, it is possible to shorten the time required to search for the optimal partial region compared to the second and third embodiments.

[0091] Sixth embodiment A sixth embodiment of the present invention will be described with reference to FIG. 9. Note that the same description as that already described may be omitted. As in the sixth embodiment, this embodiment can also reduce the total number of partial regions, thereby shortening the search time for the optimal partial region. In this embodiment, a nonlinear conversion unit 91 is inserted before the range shift unit 12 in order to reduce the total number of partial regions.

[0092] As shown in FIG. 9(a), the sensor time series signal is processed in the order of the spectrum intensity conversion unit 11, the nonlinear conversion unit 91, the range shift unit 12, the pseudo image generation unit 13, and the image recognition AI unit 14. The nonlinear conversion unit 91 performs nonlinear conversion on the values ​​of each element of each spectrum intensity vector output from the spectrum intensity conversion unit 11. Specifically, a conversion such as logarithmic conversion is performed so that the larger the value, the more compressed it becomes. As a result, for example, the frequency spectrum intensity of the sensor signal originally expressed in 12 bits (4096 values) can be expressed in 11 bits (2048 values) as shown in FIG. 9(b). Due to the compression, as shown in FIG. 9(b), the expression accuracy becomes coarser toward the top and closer to the original expression accuracy toward the bottom. Since the frequency spectrum components present in the upper part have a large intensity, it is expected that the intensity change due to the change in the state of the equipment is also large. Therefore, it is expected that the coarse expression accuracy in the upper part is tolerable.

[0093] The range shift unit 12 operates in the same manner as in the first embodiment. That is, the frequency spectrum intensity after the nonlinear transformation is transformed by focusing on a partial region having the same size as the representation range of the pseudo image (0 to 255 in FIG. 9(b)). In FIG. 9(b), the uppermost partial region is in the range of 1792 to 2047, and the lowermost partial region is in the range of 0 to 255. Each focused partial region is transformed into the representation range of the pseudo image (0 to 255 in FIG. 9(b)) by the configuration and operation of the range shift unit 12 described in FIG. 4(b).

[0094] As described above, in the fifth embodiment, the representation range is compressed by the range compression shift unit 81, whereas in this embodiment, the representation range is compressed by the nonlinear conversion unit 91. The essence of the operation is the same in both embodiments, but the implementation forms are different. In this embodiment, too, by reducing the total number of partial regions, it is possible to shorten the search time for the optimal partial region in the second and third embodiments.

[0095] Seventh embodiment A seventh embodiment of the present invention will be described with reference to Fig. 10. Note that the same description as that already described may be omitted. In this embodiment, range shift is performed on one sensor time series signal with attention to a plurality of partial regions, and a pseudo image is generated by reflecting the results.

[0096] In the example of FIG. 10, the range shift unit 12 focuses on the partial region corresponding to the first lower limit value, and converts the pixel values ​​into the 8-bit expression range (0 to 255) in the same manner as in the first embodiment. As a result, a first spectral intensity vector expressed in low bits (8 bits) is generated. The range shift unit 12 further focuses on the partial region corresponding to the second lower limit value, and generates a second spectral intensity vector expressed in low bits (8 bits). Similarly, the range shift unit 12 focuses on other partial regions, and generates spectral intensity vectors expressed in low bits (8 bits). The pseudo image generation unit 13 generates a pseudo image using these spectral intensity vectors.

[0097] For example, in FIG. 10, the pseudo image generating unit 13 generates one pseudo image by repeatedly arranging the first spectral intensity vector in the first and second rows and repeatedly arranging the second spectral intensity vector in the third and fourth rows.

[0098] The partial regions used for generating the pseudo image are searched for in the same manner as in the second and third embodiments. In these embodiments, one partial region with the smallest learning error or the largest image difference amount is searched for. In the present embodiment, similarly, multiple partial regions with small learning error or multiple partial regions with large image difference amount are searched for, and the obtained multiple partial regions are used for generating the pseudo image.

[0099] Prior to the execution of equipment state classification, a pseudo image is generated using the sensor time-series signals for learning as shown in Fig. 10, and the corresponding annotation data (correct state) is used to retrain the image recognition AI unit 14. The image recognition AI unit 14 may apply a convolutional neural network or another neural network, or a machine learning algorithm other than a neural network.

[0100] According to this embodiment, a single pseudo image can be generated using multiple partial regions of the sensor signal that are promising for classifying the equipment condition, making it possible to detect changes in condition that would be overlooked using only a single partial region.

[0101] Eighth embodiment The eighth embodiment of the present invention will be described with reference to Fig. 11(a) to (c). Note that the same description as that already described may be omitted. In this embodiment, range shift is performed on one sensor time series signal with attention to multiple partial regions, and a set of pseudo images of multiple channels (e.g., R image, G image, B image) is generated by reflecting the results.

[0102] The range shift unit 12 focuses on the partial region corresponding to the first lower limit value and converts the pixel values ​​into the 8-bit expression range (0 to 255) in the same manner as in the first embodiment. As a result, a first spectral intensity vector expressed in low bits (8 bits) is generated. The range shift unit 12 further focuses on the partial region corresponding to the second lower limit value and generates a second spectral intensity vector expressed in low bits (8 bits). Similarly, the range shift unit 12 focuses on the partial region corresponding to the third lower limit value and generates a third spectral intensity vector expressed in low bits (8 bits). The pseudo image generation unit 13 generates a pseudo image using these spectral intensity vectors.

[0103] For example, as shown in Fig. 11(a), the pseudo image generating unit 13 repeatedly arranges the first spectral intensity vector in the first and second rows to generate a pseudo R image. As shown in Fig. 11(b), the pseudo image generating unit 13 repeatedly arranges the second spectral intensity vector in the first and second rows to generate a pseudo G image. As shown in Fig. 11(c), the pseudo image generating unit 13 repeatedly arranges the third spectral intensity vector in the first and second rows to generate a pseudo B image. Note that similar processing can be performed on other sensor time series signals to reflect them in the remaining areas of the pseudo R, G, and B images.

[0104] Each partial area used for generating the pseudo image can be searched in the same manner as in the seventh embodiment. Furthermore, as in the seventh embodiment, prior to the execution of equipment state classification, a pseudo image is generated using a sensor time-series signal for learning as shown in Fig. 11(a) to (c), and the image recognition AI unit 14 is retrained using the corresponding annotation data (correct state). The image recognition AI unit 14 may apply a convolutional neural network or another neural network, or a machine learning algorithm other than a neural network.

[0105] According to this embodiment, a multi-channel pseudo image can be generated using multiple partial regions of the sensor signal that are promising for classifying the equipment condition, making it possible to detect changes in condition that would be overlooked using only a single partial region.

[0106] In the above, the case where the range shift is performed by applying the same lower limit value to the data from different sensors has been described. As a result, a uniform lower limit value is selected by the method shown in the second and third embodiments so that an appropriate partial area of ​​a sensor with high state detection sensitivity is included. Note that range shift may be performed by applying different lower limit values ​​to data from different sensors. In that case, in the second and third embodiments, the lower limit value for each sensor is changed independently, and the optimal lower limit value for each sensor is searched for and determined. In this case, the search time is increased, but since the optimal partial area can be selected for the data from all sensors, more accurate state detection is possible. Note that in the seventh embodiment, which handles one sensor time series signal, a uniform lower limit value is selected by the method shown in the second and third embodiments so that an appropriate partial area with high state detection sensitivity is included, but based on the same viewpoint, the optimal lower limit value for each of the partial areas may be searched for and determined while changing the lower limit values ​​for each of the partial areas independently.

[0107] <Hardware configuration> Next, an example of the hardware configuration of the status detection device will be described with reference to Fig. 12. The status detection device shown in Fig. 12 includes a plurality of sensors (101a, 101b), a camera 102, a plurality of A / D converters (103a, 103b), a calculation unit, an input device 109, and an output device 110. Note that this hardware configuration is an example and may be changed as appropriate.

[0108] As described above, the sensors are placed on or near the equipment to monitor the equipment status and detect a predetermined physical quantity. In this example, two sensors (101a, 101b) are placed, but more sensors may be placed. Also, a single sensor may be placed. As in the case of the sensors, the camera 102 is placed to monitor the equipment status and captures a camera image by imaging. The A / D converters (103a, 103b) are used to convert the analog output of the sensor into a digital signal. If the sensors (101a, 101b) output digital signals, the A / D converters (103a, 103b) may be omitted.

[0109] Next, the calculation unit will be described. The calculation unit performs calculation processing based on input data, and performs the above-described spectral intensity conversion, range shift, pseudo image generation, image analysis, etc. In this example, the calculation unit includes a general-purpose processor 105 and a dedicated circuit 106 as a main component (processing device 104) for data processing.

[0110] In this example, the general-purpose processor 105 is a CPU (Central Processing Unit), but is not limited thereto and may be configured using, for example, other semiconductor devices. The dedicated circuit 106 is used as an accelerator for improving data processing. The dedicated circuit 106 may be configured as appropriate. For example, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc. may be used for the dedicated circuit 106. The configuration of the processing device 104 may be changed as appropriate as long as it can appropriately execute a predetermined process.

[0111] In this example, the calculation unit includes a storage 107 and a memory 108. The storage 107 stores appropriate data such as a program used in processing. The storage 107 may store, for example, programs used to realize the spectrum intensity conversion unit 11, the range shift unit 12, the pseudo image generation unit 13, the image recognition AI unit 14, the range instruction unit 51, the learning unit 53, the image comparison unit 61, the range adjustment unit 73, the range compression shift unit 81, and the nonlinear conversion unit 91. The holding unit 52 can be realized using an appropriate storage device such as the storage 107. The storage 107 may store, for example, data (weights, models, etc.) used in image analysis. The storage 107 may store data input from the sensors (101a, 101b) and the camera 82, data of the generated pseudo image, and the like.

[0112] Furthermore, the storage 107 may include a learning data storage unit that is a storage area for storing learning data.

[0113] The storage 107 can be configured using, for example, a hard disk drive (HDD). The memory 108 can be configured using, for example, a dynamic random access memory (DRAM), and the processing device 104 can execute processes (predetermined processes such as spectrum intensity conversion, range shift, pseudo image generation, and image analysis) based on the programs and data loaded into the memory 108.

[0114] The input device 109 is used by the user to perform various settings related to the operation of the device, and is configured appropriately using a keyboard, a mouse, a touch panel, etc. The output device 110 is used to display the classification results of the equipment status and the contents of the user's input, and can be configured appropriately using a display, etc.

[0115] Next, an example of a hardware configuration of a status detection device different from that of Fig. 12 will be described with reference to Fig. 13. Note that descriptions similar to those already described may be omitted.

[0116] The status detection device in Fig. 13 differs from that in Fig. 12 in that it includes a communication device 111 and is capable of communicating with an external computer 201. In this example, a process requiring a large amount of calculation, such as a learning process, is executed by an external computer 201 having large calculation resources, and the status detection device obtains the resulting data.

[0117] The communication device 111 is an interface device used for communication, and the status detection device can transmit and receive data to and from an external computer 201 via the communication device 111. Note that the external computer 201 may be located, for example, on-premise or in the cloud. Furthermore, wired communication or wireless communication may be performed.

[0118] With the configuration of Fig. 13, for example, the condition detection device may transmit data used for learning a model or a machine learning algorithm used in image recognition to an external computer 201 via the communication device 111. Then, the condition detection device may acquire learned parameters of the model or the machine learning algorithm calculated by the external computer 201 via the communication device 111. Here, the external computer 201 performs learning for each partial region as described with reference to Fig. 5(b). In addition, the condition detection device may also acquire a value for focusing on a partial region when executing condition detection.

[0119] Also, for example, the condition detection device may transmit pseudo images showing different states of the equipment to the external computer 201 via the communication device 111. Then, the condition detection device may acquire learned parameters in the area where the image difference amount calculated by the external computer 201 is the largest via the communication device 111. Here, as described using FIG. 6, the external computer 201 searches for a partial area where the image difference amount is the largest from the pseudo images showing different states of the equipment, and learns the partial area. Also, the condition detection device may acquire a value for focusing on the partial area when performing condition detection.

[0120] Next, an example of a hardware configuration of a status detection device different from those in Figures 12 and 13 will be described with reference to Figure 14. Note that descriptions similar to those already described may be omitted. In Figure 14, the status detection device includes an input / output device 112 to which a portable storage device is connected, and inputs and outputs data to and from a storage device such as a USB memory via the input / output device 112.

[0121] In this configuration, a user may connect a storage device connected to the input / output device 112, in which data used for learning a model for image recognition or a machine learning algorithm is stored by the status detection device, to an external computer 201, and the external computer 201 may acquire the data from the storage device. The external computer 201 may also store the calculated model or learned parameters of the machine learning algorithm in a connected storage device, and the user may connect the storage device to the input / output device 112 of the status detection device, and the status detection device may acquire the learned parameters from the storage device. Here, the external computer 201 performs learning for each partial region, as described with reference to FIG. 5(b). The external computer 201 may also store a value for focusing on a partial region in the execution of status detection in a storage device, and the status detection device may acquire the value from the storage device together with the learned parameters.

[0122] Also, for example, a user may connect a storage device connected to the input / output device 112, in which data of pseudo images showing different states of the equipment are stored by the status detection device, to the external computer 201, and the external computer 201 may acquire the data from the storage device. Also, the external computer 201 may store learned parameters in the area where the calculated image difference amount is the largest in the connected storage device, and the user may connect the storage device to the input / output device 112 of the status detection device, and the status detection device may acquire the learned parameters from the storage device. Here, as described with reference to FIG. 6, the external computer 201 searches for a partial area where the image difference amount is the largest from pseudo images of different states of the equipment, and learns the partial area. Also, the external computer 201 may store a value for focusing on the partial area in the execution of status detection in the storage device, and the status detection device may acquire the value from the storage device together with the learned parameters.

[0123] Although the embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various design changes can be made within the scope of the spirit of the present invention described in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. In addition, unless otherwise specified, each component may be singular or plural.

[0124] The operations of the range designation unit 51, learning unit 53, image comparison unit 61, range adjustment unit 73, range compression shift unit 81, and nonlinear conversion unit 91 may be performed using an accelerator (dedicated hardware), or may be performed by software processing using a general-purpose processor such as a CPU. Also, an internal memory (SRAM, etc.) mounted on an accelerator chip or CPU chip may be used, or an external memory (DRAM, etc.) may be used. The holding unit 52 may be realized using a memory.

[0125] As described above, the program used for the processing may be stored in the storage device (storage 107) of the status detection device. On the other hand, the program used for the processing of the status detection device may be stored in a storage device connected via the input / output device 112, for example. [Explanation of symbols]

[0126] 11 Spectral Intensity Conversion Section 12 Range shift section 13 Pseudo image generation unit 14 Image Recognition AI Department 41 Subtraction section 42 Out-of-range processing unit 51 Range indicator 52 Holding part 53 Learning Department 61 Image Comparison Section 71 Selection section 72 AFE Department 73 Range adjustment section 81 Range compression shift section 82 Multiplication section 91 Nonlinear Transformation Section 101a, 101b Sensor 102 Camera 103a, 103b A / D converter 104 Processing equipment 105 General Purpose Processors 106 Dedicated Circuit 107 Storage 108 Memory 109 Input Device 110 Output Device 111 Communication device 112 Input and output device 201 Calculator

Claims

1. A calculation unit is provided to acquire data detected by the sensor, The calculation unit is A processing device for performing data processing, The pretreatment device comprises: converting the digitized time series signal data of the sensor into data related to frequency spectrum intensity; converting a part of the entire range in which the data values ​​relating to the frequency spectrum intensity are distributed into low-bit representation data; generating a pseudo image based on the low-bit representation of the data; classifying the pseudo image based on image recognition and outputting a classification result of the equipment state; A condition detection device characterized by:

2. The condition detection device according to claim 1 , The processing device includes: Perform image recognition using neural network models or machine learning algorithms. A condition detection device characterized by:

3. The condition detection device according to claim 1 , The processing device includes: converting the value of the data relating to the frequency spectrum intensity into the low-bit represented data using a reference value of the partial region; A condition detection device characterized by:

4. The condition detection device according to claim 3, The processing device includes: during the conversion, replacing values ​​outside the range of the low-bit representation with a predetermined value within the range of the low-bit representation to obtain the low-bit represented data; A condition detection device characterized by:

5. The condition detection device according to claim 1 , The partial area when performing the classification is determined based on the accuracy of a model or a machine learning algorithm used for the image recognition during learning. A condition detection device characterized by:

6. The condition detection device according to claim 1 , The part of the area for the classification is determined based on the amount of difference in the pseudo image between different states of the equipment. A condition detection device characterized by:

7. The condition detection device according to claim 3, An analog front-end unit for processing a signal from the sensor is further provided, the digitized time series signal of the sensor is obtained by processing the analog signal of the sensor by the analog front end unit; A selection unit that periodically selects a pilot signal having a known frequency and outputs the pilot signal to the analog front unit, The processing device includes: adjusting the reference value based on a frequency spectrum intensity of the pilot signal; A condition detection device characterized by:

8. The condition detection device according to claim 1 , The partial area occupying the side of the larger value of the data on the frequency spectrum intensity within the entire area is set wider than the range of the low bit representation. A condition detection device characterized by:

9. The condition detection device according to claim 1 , The processing device includes: converting the data related to the frequency spectrum intensity into the low-bit represented data based on the nonlinearly converted data; A condition detection device characterized by:

10. The condition detection device according to claim 1 , The processing device includes: converting the plurality of partial regions into a plurality of low-bit represented data; generating a pseudo image based on the plurality of low-bit representation data; A condition detection device characterized by:

11. The condition detection device according to claim 1 , The processing device includes: converting the plurality of partial regions into a plurality of low-bit represented data; generating a multi-channel pseudo image based on the plurality of low-bit representation data; A condition detection device characterized by:

12. The condition detection device according to claim 1 , Further comprising a communication device, The processing device includes: acquiring, via the communication device, trained parameters of a model or a machine learning algorithm used for the image recognition; A condition detection device characterized by:

13. The condition detection device according to claim 12, The processing device includes: acquiring, via the communication device, learned parameters of an area having a minimum learning error among the learning for each of the partial areas across the entire area; A condition detection device characterized by:

14. The condition detection device according to claim 12, The processing device includes: acquiring, via the communication device, learned parameters of the partial area in which the amount of difference in the pseudo image between different states of the equipment is the largest; A condition detection device characterized by:

15. The condition detection device according to claim 1 , an input / output device to which a portable storage device can be connected and which enables data exchange between the processing device and the storage device connected thereto; The processing device includes: Acquiring, via the input / output device, trained parameters of the model or machine learning algorithm used for the image recognition, which are calculated by an external computer and stored in the storage device; A condition detection device characterized by:

16. The condition detection device according to claim 15, The processing device includes: acquiring, via the input / output device, learned parameters of a region having a minimum learning error among the partial region across the entire region; A condition detection device characterized by:

17. The condition detection device according to claim 15, The processing device includes: acquire, via the input / output device, learned parameters of the partial area in which the amount of difference in the pseudo image between different states of the equipment is the largest; A condition detection device characterized by:

18. A state detection method performed using a processing device, comprising: A processing device that performs data processing converts the digitized time series signal data of the sensor into data related to frequency spectrum intensity; a step of converting a part of an entire range in which the values ​​of the frequency spectrum intensity data are distributed into low-bit representation data by the processing device; generating a pseudo image based on the low-bit representation data by the processing device; The processing device classifies the pseudo image based on image recognition, and outputs a classification result of the equipment state. A condition detection method comprising:

19. A program for causing a processor to execute the state detection method according to claim 18.

20. A storage device that stores a program that is read and executed by a computer to implement the condition detection method according to claim 18.

Images