Status detection device and status detection method

Abstract

To detect equipment status using low-cost and small computation hardware in a status detection device.SOLUTION: A status detection device performs convolutional neural network operation on an image using different bit widths for representation, and determines the true state of the equipment from among a plurality of equipment statuses on the basis of the computation results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a condition detection device and a condition detection method. [Background technology]

[0002] Infrastructure and plant facilities are aging, and their maintenance and management is an important social issue. Therefore, automatic equipment monitoring technology is desired.

[0003] In recent years, systems have been developed that detect external anomalies (scratches, cracks, etc.) in equipment by applying image recognition AI processing to camera images and satellite images. However, methods that use only images have issues with detection limitations due to image resolution restrictions and the fact that they can only detect external anomalies. Therefore, there is a need for technology that can detect the condition of equipment with high accuracy and reliability by analyzing the time-series signals of various sensors attached to the equipment.

[0004] However, conventional time-series signal analysis requires manually creating dedicated analysis algorithms depending on the type of sensor, signal characteristics, and type of condition to be detected, making it difficult to build a system in a short period of time and at low cost.

[0005] On the other hand, there are efforts to analyze time-series signals from sensors using deep neural networks for time-series signals, such as recurrent neural networks (RNNs) and long short-term memories (LSTMs). However, deep neural networks for time-series signals are generally not easy to train.

[0006] Recently, Transformer networks have been used for natural language processing. Although it is possible to analyze time-series signals using Transformer networks, the amount of calculation and power consumption is large, which leads to high costs and large sizes of the calculation hardware.

[0007] In response to this, Patent Documents 1 and 2 disclose a technology for detecting abnormalities in equipment by generating a pseudo-image based on the time-series signal of a sensor and analyzing this image using a convolutional neural network.

[0008] Convolutional neural networks are easier to train than deep neural networks for time series signals, and they require less computational effort and consume less power than Transformer-based networks. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 2020-144619 [Patent Document 2] Japanese Patent Application Publication No. 2023-171228 Summary of the Invention [Problem to be solved by the invention]

[0010] To enable widespread application in a variety of facilities, it is necessary to reduce the cost and size of the computing hardware as much as possible. In addition, reducing power consumption also means reducing the size of the battery and enabling fanless operation.

[0011] Therefore, in order to further reduce the amount of calculation and power consumption of the convolutional neural network, the number of bits (resolution) used to represent the image (input image and intermediate image) is reduced in Patent Document 2. Meanwhile, A / D conversion technology for digitizing sensor output continues to advance, making it possible to obtain high-resolution time-series signals.

[0012] However, the resolution is not fully utilized due to the reduction in the number of bits used for representation, and as a result, the equipment to which it can be applied is still limited.

[0013] An object of the present invention is to enable a status detection device to detect the status of equipment using low-cost, small-sized computing hardware. [Means for solving the problem]

[0014] A condition detection device according to one aspect of the present invention is a condition detection device that detects the condition of equipment using at least one sensor, and includes an image generation unit that converts a time series signal from the sensor into an image, an arithmetic circuit that performs a convolutional neural network operation on the image using different numbers of representation bits, and a judgment unit that determines a true state from among a plurality of states of the equipment based on the calculation results of the arithmetic circuit, wherein in a first calculation on the image, the arithmetic circuit performs a first calculation on all image regions of the image using a first number of representation bits from the different numbers of representation bits, and outputs first class probabilities for each of a plurality of states, and A plurality of states with high probability values ​​are extracted as state candidates from the class probabilities, and in a second calculation on the image, a partial image region in the image with a high contribution is selected for each of the plurality of state candidates based on the weight value of the trained final layer of the convolutional neural network, and a second calculation is performed on the partial image region with a second number of representation bits higher than the first number of representation bits from the different number of representation bits to output second class probabilities for the plurality of state candidates, and the determination unit determines the state candidate with the highest probability value among the plurality of second class probabilities as the true state of the equipment. [Effects of the Invention]

[0015] According to one aspect of the present invention, in a status detection device, it is possible to perform equipment status detection using low-cost and small-sized computing hardware. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating a configuration of a status detection device according to a first embodiment. [Figure 2A] FIG. 10 is a diagram showing the effect of the first embodiment. [Figure 2B] FIG. 10 is a diagram showing the effect of the first embodiment. [Figure 3] FIG. 1 is a diagram illustrating a configuration of a neural network according to a first embodiment. [Figure 4A] FIG. 2 is a diagram illustrating the operation of the first embodiment. [Figure 4B] FIG. 2 is a diagram illustrating the operation of the first embodiment. [Figure 5] FIG. 10 is a diagram showing the configuration of an image area contribution table according to the first embodiment. [Figure 6A] FIG. 10 is a diagram illustrating a configuration of a status detection device according to a second embodiment. [Figure 6B] FIG. 10 is a diagram illustrating the operation of the second embodiment. [Figure 7A] FIG. 10 is a diagram illustrating a configuration of a status detection device according to a third embodiment. [Figure 7B] FIG. 10 is a diagram illustrating the operation of the third embodiment. [Figure 8A] FIG. 10 is a diagram illustrating the configuration of a fourth embodiment. [Figure 8B] FIG. 10 is a diagram illustrating the configuration of a fourth embodiment. [Figure 9A] FIG. 10 is a diagram illustrating the configuration of a status detection device according to a fifth embodiment. [Figure 9B] FIG. 10 is a diagram showing the effect of Example 6. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment will be described with reference to the drawings. [Example]

[0018] FIG. 1 is a diagram illustrating the configuration of a status detection device according to a first embodiment.

[0019] As shown in FIG. 1, the condition detection device of the first embodiment includes an image generation unit 11, a bit-number-switchable convolutional neural network calculation circuit 12, an image memory 13, a weight memory 14, a scheduler 15, a recalculation image area determination unit 16, an image area contribution table 17, and a determination unit 18.

[0020] In order to monitor the condition of the equipment, one or more sensors (not shown) are placed on or near the equipment. The time-series signal output by each sensor is assumed to be already digitized at the sensor output stage or by an A / D converter (not shown) downstream of the sensor. This digitized time-series signal (hereinafter referred to as the sensor signal) is input to the image generation unit 11.

[0021] The image generation unit 11 converts the sensor signal into an image. The bit-number-switchable convolutional neural network operation circuit 12 performs neural network operation on the image and outputs the probability of each state (first-time class probability). At this time, the operation is performed while reading and writing the image and images (intermediate images) of each layer output generated during the neural network operation from and to an image memory 13. The operation is also performed while reading weight data for the neural network stored in a weight memory 14.

[0022] The neural network operation proceeds according to a control signal output from the scheduler 15 to the bit-number-switchable convolutional neural network operation circuit 12. The operation is performed while selecting pixels in the image and the intermediate image according to a pixel selection signal output from the scheduler 15.

[0023] The scheduler 15 further outputs a bit number selection signal to the bit number switchable convolutional neural network operation circuit 12 to instruct it to represent the pixel using a low bit number or a high bit number.

[0024] In the first neural network calculation, the bit number selection signal indicates a low representation bit number. This causes the image and the intermediate image to be represented and calculated using a low bit number. As a result, the first class probability output is a low-precision class probability.

[0025] The first class probability output (low-accuracy class probability) is input to the recalculation image region determination unit 16. Because the first class probability output is low-accuracy, it cannot be used to determine the true state, but it is accurate enough to extract multiple state candidates. Therefore, one of these state candidates is the true state. The recalculation image region determination unit 16 extracts multiple states with high probabilities in the first class probability output as state candidates.

[0026] Here, as the state with a high probability value, a plurality of states whose probability value exceeds a predetermined threshold are extracted as state candidates, or a plurality of states having a predetermined number of higher probability values ​​among the plurality of first class probabilities (first class probabilities) are extracted as state candidates as the state with a high probability value.

[0027] The recalculation image area determination unit 16 further determines an area (image area) in the image having a high contribution for each state of the state candidates based on the image area contribution table 17, and transmits this to the scheduler 15 as a recalculation image area.

[0028] The image region contribution table 17 holds information on image regions that have a high contribution to each state based on the learned weight data of the final layer of the neural network.

[0029] Next, the scheduler 15 uses the control signal to cause the bit-number-switchable convolutional neural network operation circuit 12 to execute a second neural network operation, similar to the first neural network operation. However, in the second operation, only the recalculation image area for each of the state candidates transmitted from the recalculation image area determination unit 16 is calculated. Therefore, the pixel selection signal output by the scheduler 15 selects pixels in the image or intermediate image that correspond to the recalculation image area.

[0030] Furthermore, in the second calculation, the bit-number selection signal indicates a higher number of representation bits, which causes each of the pixels to be represented and calculated using a higher number of bits. As a result, the bit-number-switchable convolutional neural network calculation circuit 12 outputs a class probability output with higher accuracy as a result of the second neural network calculation than the first calculation.

[0031] Therefore, the determining unit 18 determines that the state with the highest probability value in the highly accurate class probability output is the true state. As a result, the state of the equipment can be detected with high accuracy.

[0032] As shown in Figure 2A, this technology can reduce power consumption. Without applying this invention, calculations are performed only once for each image using a high number of representation bits (e.g., 16 bits), and a highly accurate class probability is output. The state with the highest probability is determined to be the true state. Because calculations are performed using a high number of representation bits, power consumption (power consumption for memory access and power consumption of the calculation circuit) increases in proportion to the number of bits.

[0033] In contrast, when applying the present invention, the first calculation is performed for each image using a low number of representation bits (for example, 4 bits). Multiple state candidates are extracted based on the low-precision class probability output obtained in the first calculation, and the second calculation is performed again only for image areas that have a high contribution to the extracted state. The state with the highest probability in the high-precision class probability output obtained is determined to be the true state. Because the calculation is performed using a low number of representation bits, power consumption can be reduced.

[0034] Furthermore, although a second calculation is required, it is performed only on a portion of the image area (partial image area), so it takes less time than the first calculation. Therefore, compared to when this technology is not applied, it is possible to reduce power consumption with almost no decrease in image classification speed (i.e., frequency of state detection).

[0035] 2B, the present invention can also reduce the mounting area. When the present invention is not applied, calculations are performed using a high number of representation bits (for example, 16 bits), so the memory capacity required for the image memory 13 increases in proportion to the number of bits, and the memory circuit area also increases.

[0036] On the other hand, when the present invention is applied, calculations are performed using a low number of representation bits (for example, 4 bits), thereby reducing the memory capacity and area required for image memory 13. The second calculation is performed using a high number of representation bits (for example, 16 bits), but since calculations are performed only on a portion of the image area, the small memory capacity reserved for the first calculation is sufficient. Generally, the circuit area of ​​a memory is larger than the area of ​​the calculation circuit, so reducing the circuit area of ​​the memory can sufficiently reduce the overall circuit area.

[0037] FIG. 3 shows the configuration of the neural network that is the target of calculation by the bit-number-switchable convolutional neural network calculation circuit 12.

[0038] The convolutional layer and pooling layer 31, which is the first stage of the neural network, is any appropriate combination of multiple convolutional layers and multiple pooling layers. The intermediate image at the end of the convolutional layer and pooling layer 31 is converted into one-dimensional data by the flattening layer 32. The one-dimensional data is input to the fully connected layer 33, which is the final layer of the entire neural network.

[0039] The fully connected layer 33 performs a product-sum operation using each output of the previous Flatten layer 32 and the weights associated with them, applies a Softmax function as an activation function to the product-sum operation result, and outputs the class probability for each state as the output value of the Softmax function.

[0040] Specific operation is shown in FIG. 4A and FIG. 4B.

[0041] Figure 4A shows the first neural network operation: the image is input and processed by two convolutional layers across all image regions.

[0042] In this case, as mentioned above, the calculation is performed using a low representation bit count (for example, 4 bits). The output of the second convolutional layer is converted into one-dimensional data by a flattening layer. The output of the flattening layer is input to a single fully connected layer. This fully connected layer is the final layer of the neural network and outputs the first class probability mentioned above.

[0043] The neural network in Figure 4A classifies four states: A, B, C, and D, with the probability of state A being 0.4 (40%), the probability of state B being 0 (0%), the probability of state C being 0.5 (50%), and the probability of state D being 0.1 (10%). Therefore, states A and C, which have high probabilities, are extracted as the candidate states.

[0044] Figure 4B shows the second neural network calculation. As mentioned above, states A and C were extracted as state candidates in the first neural network calculation. Therefore, calculations need to be performed only on image region 6 (the region marked "6" in the "Image" section of Figure 4B) that has a high contribution to state A and image region 13 (the region marked "13" in the "Image" section of Figure 4B) that has a high contribution to state C.

[0045] The reason why the image regions 6 and 13 are image regions with high contributions is as follows.

[0046] The magnitude of the contribution to each state is determined by the magnitude of the weights in the final layer. The final layer is a fully connected layer, and the weights of the connections between each output of the flattening layer (16 outputs in Figures 4A and 4B) and each state of the final layer (4 states in Figures 4A and 4B) use values ​​obtained in advance during training.

[0047] In the example of Figure 4B, the weight for the sixth output of the Flatten layer (shown in bold) is the largest among the weights related to state A. Therefore, of the 16 outputs of the Flatten layer, the sixth output has the highest contribution to state A. Similarly, the weight for the 13th output of the Flatten layer (shown in bold) is the largest among the weights related to state C. Therefore, of the 16 outputs of the Flatten layer, the 13th output has the highest contribution to state C.

[0048] On the other hand, each output of the Flatten layer is calculated by a convolution operation on the corresponding image region. In Figure 4B, the sixth output of the Flatten layer is calculated by a convolution operation on image region 6. Therefore, the image region with the highest contribution to state A is image region 6. Similarly, the thirteenth output of the Flatten layer is calculated by a convolution operation on image region 13. Therefore, the image region with the highest contribution to state C is image region 13.

[0049] Therefore, in the second neural network calculation, a convolution calculation is performed on image region 6 as follows, and the sixth output of the flattened layer is calculated. First, the first convolutional layer calculation is performed on image region 6. Next, the second convolutional layer calculation is performed on the region obtained by the first convolutional layer calculation (the region indicated by "6" in "Output of the first convolutional layer" in Figure 4B). This generates the pixel value indicated by "6" in "Output of the second convolutional layer" in Figure 4B. This pixel value is used to rewrite the sixth output of the flattened layer from the value obtained during the first neural network calculation.

[0050] Similarly, in the second neural network calculation, a convolution calculation is performed on image region 13 as follows, and the 13th output of the Flatten layer is calculated. First, the first convolution layer calculation is performed on image region 13. Next, the second convolution layer calculation is performed on the region obtained by the first convolution layer calculation (the region indicated as "13" in "Output of the First Convolution Layer" in Figure 4B). This generates the pixel value indicated as "13" in "Output of the Second Convolution Layer" in Figure 4B. This pixel value is used to rewrite the 13th output of the Flatten layer from the value obtained during the first neural network calculation.

[0051] The convolution operation in the second neural network operation is performed with a high representation bit count (for example, 16 bits) as described above. As a result, the 6th and 13th outputs of the Flatten layer in the second neural network operation are replaced with values ​​with higher precision than in the first neural network operation.

[0052] Furthermore, the final layer (fully connected layer) is recalculated using the Flatten layer output after the replacement, and the first class probability output is updated. This updates the class probabilities of states A and C. In Figure 4B, the probability of state A has been updated from 0.4 (40%) to 0.7 (70%), and the probability of state C has been updated from 0.5 (50%) to 0.2 (20%). Because these updated class probabilities are calculated with high accuracy, they can be used to determine the true state. Since the probability of state A is the highest among the updated class probabilities, state A is determined to be the true state.

[0053] Note that although Figures 4A and 4B show a case where the convolutional layer and pooling layer 31 are two convolutional layers, the same applies to a case where there are more layers or where a pooling layer is mixed in.

[0054] 5 shows an example of the image area contribution table 17. It stores the number of the image area with the highest contribution to each state (first contribution) and the number of the image area with the second highest contribution (second contribution).

[0055] As mentioned above, the image regions with the highest contributions are determined based on the magnitude of the weights between the final layer (fully connected layer) and the Flatten layer output. For each state of the final layer, the number of the image region corresponding to the largest weight is stored as the number 1 contributor, and the number of the image region corresponding to the second largest weight is stored as the number 2 contributor.

[0056] Fig. 5 corresponds to the example of Fig. 4B, in which image area 6 is stored as the highest contributor to state A, and image area 13 is stored as the highest contributor to state C. Therefore, the recalculation image area determination unit 16 can execute the operation shown in Fig. 4B by determining an image area corresponding to a state candidate with reference to image area contribution table 17 shown in Fig. 5.

[0057] In addition, the image area with the second highest contribution stored as shown in Figure 5 may also be used to perform a second neural network calculation on two image areas for each state candidate (the image area with the first highest contribution and the image area with the second highest contribution).

[0058] Similarly, based on the weight between the final layer (fully connected layer) and the Flatten layer output, image regions with contributions of 3rd or lower may also be stored and subjected to a second neural network calculation. By including image regions with lower contributions in the second neural network calculation, the accuracy of the resulting class probabilities can be further improved.

[0059] On the other hand, the second neural network calculation takes time, which reduces the image classification speed (i.e., the frequency of state detection).The number of image regions to be included in the second neural network calculation can be set according to each state detection application.

[0060] As described above, in this embodiment, the time-series signals of the sensors are converted into images, and the images are classified twice using a neural network based on a convolutional layer, thereby classifying the state of the equipment.

[0061] In the first neural network calculation, the image input to the neural network and intermediate images generated during the neural network calculation are represented using a low number of bits and then calculated. Based on the first class probability output, several state candidates with high probabilities are extracted.

[0062] In the second neural network calculation, only a portion of the image area (partial image area) that contributes highly to the extracted state candidate is recalculated with a higher number of bits.The state with the highest probability in the second class probability output is determined to be the final state.

[0063] The state candidate may be selected from states having a probability value equal to or greater than a predetermined threshold in the first class probability output, or may be selected from a predetermined number of states having higher probability values ​​among a plurality of first class probabilities in the first class probability output.

[0064] Furthermore, if the maximum probability value in the second class probability output is less than a predetermined threshold value, the number of image regions may be increased and a third neural network operation may be similarly performed to determine the true state.

[0065] The above configuration and operation reduce the circuit mounting area and power consumption of the computing hardware that executes sensor-based equipment status detection, thereby enabling the cost and size of the computing hardware to be reduced. [Example]

[0066] 6A is a diagram showing the configuration of a status detection device of Example 2. In this Example, a frequency conversion image generation unit 61 is used as the image generation unit 11 in Example 1. The other configurations and operations are the same as those in Example 1, and therefore a description thereof will be omitted.

[0067] The frequency conversion image generation unit 61 converts time-series signals (sensor signals) from one or more sensors into frequency spectrum data, and generates an image based on the frequency spectrum data. For example, the image may be generated by the method described in Patent Document 2.

[0068] In particular, by including frequency spectrum data from multiple sensors on a single image, sensor fusion can be achieved, improving the sensitivity and accuracy of condition detection. Figure 6B shows an example.

[0069] As shown in Figure 6B, the time series signals (sensor signals) from multiple sensors are each converted into one-dimensional frequency spectrum data using FFT (Fast Fourier Transform). Each one-dimensional frequency spectrum data is used as one line of the image, and by arranging them vertically, an image like that shown in Figure 6B is generated.

[0070] As described in Patent Document 2, each one-dimensional frequency spectrum data may be divided into multiple pieces and aligned vertically to increase the number of rows in the image. Alternatively, each one-dimensional frequency spectrum data may be aligned repeatedly for multiple rows to increase the number of rows in the image. These methods may also be combined. In this way, an image of a desired size is generated.

[0071] The reason why image generation based on the above-described frequency transformation is preferable is as follows: In many cases, specific frequency components of each sensor signal contribute significantly to detecting the equipment condition. For example, in Figure 6B, the frequency components indicated by the bold lines in each frequency spectrum contribute significantly.

[0072] These frequency components are transformed into the pixels enclosed in a bold frame in the "Image" in Figure 6B. In other words, unlike general images of dogs, cats, etc., images generated from sensor signals based on frequency transformation contain important information in specific pixels (more generally, specific image regions).

[0073] Therefore, by using such images to train the neural network in Figure 3 in advance, it is possible to train it so that the weights corresponding to the specific image region are significantly larger among the weights in the final layer (fully connected layer) (weights between the output of the final layer and the flattened layer). This increases the validity of performing the second neural network calculation only on the image regions with the highest contribution, as shown in Figures 4B and 5.

[0074] In other words, if there are no image regions with prominent contributions, many image regions must be included in the second neural network calculation, which reduces the image classification speed (i.e., the frequency of state detection), but this situation can be avoided.

[0075] As described above, this embodiment improves the validity of performing the second neural network calculation using only the image regions with the highest contribution, thereby achieving both high accuracy and high frequency (real-time) state detection. [Example]

[0076] 7A is a diagram showing the configuration of a status detection device according to Example 3. While Example 1 shows the configuration of the status detection device in the inference phase, Example 3 shows the configuration of the status detection device in the learning phase.

[0077] In this embodiment 3, the learning method is improved in the embodiment 1. Therefore, a learning unit with a regularization term 71 is added to the configuration of the embodiment 1.

[0078] As described above, in the present invention, neural network training is performed in advance. During training, as shown in FIG. 7A, a training sensor time-series signal (training sensor signal) is input to the image generation unit 11, which then generates a training image. The generated training image is input to the bit-number-switchable convolutional neural network operation circuit 12. The bit-number-switchable convolutional neural network operation circuit 12 performs calculations for the neural network configuration shown in FIG. 3 and outputs class probabilities. The class probability output is supplied to the regularization term-attached training unit 71. Training labels are also supplied to the regularization term-attached training unit 71.

[0079] The training label is the class probability correct answer data corresponding to the training sensor signal. For example, in the case of four-state classification as shown in Figures 4A and 4B, if the training sensor signal is for state A, the class probability for state A is 100% and the remaining states B, C, and D are 0%, so the training label is (1,0,0,0). Similarly, if the training sensor signal is for state B, the class probability for state B is 100% and the remaining states A, C, and D are 0%, so the training label is (0,1,0,0).

[0080] During learning, the learning sensor signals for all states and the corresponding learning labels are sequentially supplied, either sequentially or randomly, to the image generation unit 11 and the learning unit with regularization term 71. As a result, class probability outputs corresponding to the learning sensor signals are sequentially supplied from the bit-number-switchable convolutional neural network operation circuit 12 to the learning unit with regularization term 71, as described above, and at the same time, the corresponding learning labels are supplied to the learning unit with regularization term 71.

[0081] The regularization term-added learning unit 71 updates the weights used in the neural network (FIG. 3) from their current values ​​based on the difference between the supplied class probability output and the learning label (class probability correct data). The updated weights are stored in the weight memory 14. As a result, the bit-number-switchable convolutional neural network operation circuit 12 uses the updated weights stored in the weight memory 14 to perform neural network operation on the next learning sensor signal (learning image).

[0082] In this way, by updating the weights of the neural network while successively supplying learning sensor signals and corresponding learning labels, the weights eventually converge to appropriate values ​​and learning is completed.

[0083] Prior to learning, default weights learned using general images are stored in the weight memory 14. The weights to be updated may be the weights of all layers of the neural network or only some of the layers. These may be selected according to the state detection accuracy required for the state detection application and the properties of the sensor signals used. At least the weights of the final layer (fully connected layer) directly linked to image classification (the weights between the output of the final layer and the flattened layer) are to be updated.

[0084] After the learning is completed, of the weights finally obtained, the weight values ​​of the final layer ("learned final layer weight data" in FIG. 7A) are supplied to the image region contribution table 17. This allows the operation shown in the first embodiment to be executed.

[0085] During the above learning, the bit-number-switchable convolutional neural network operation circuit 12 performs highly accurate calculations with a high number of representation bits. To this end, the scheduler 15 instructs the bit-number-switchable convolutional neural network operation circuit 12 to perform calculations with a high number of representation bits using a bit-number selection signal.

[0086] During learning, neural network calculations are performed on all image regions. To this end, the scheduler 15 causes the bit-number-switchable convolutional neural network calculation circuit 12 to execute calculations on all pixels via a pixel selection signal.

[0087] The update of the weights performed by the regularization term-added learning unit 71 will be further described with reference to FIG. 7B.

[0088] In normal learning, the value of each weight is updated so that the normal loss function L based on the difference between the supplied class probability output and the learning label (class probability correct data) is minimized.

[0089] To achieve this, the gradient of the normal loss function L is derived in advance as a formula by differential calculation, and the weights are updated using a weight update formula based on the derived formula.

[0090] In contrast, the regularization term learning unit 71 of this embodiment uses the regularization term loss function L TOTAL Update the value of each weight so that it is minimized. Therefore, we first set the loss function L TOTAL The gradient of is derived as a formula by differential calculation, and each weight is updated by a weight update formula based on the derived formula.

[0091] Loss function L with regularization term TOTAL is the regularization term L for the normal loss function L. REG is added with a coefficient of λ. That is, it is as shown in (Equation 1).

[0092]

number

[0093]

number

[0094] For example, the weight W associated with state A 1A , W 2A , …, W 16A If all are equal to 1 / 16, the first term of (Equation 2) becomes 16. On the other hand, among the weights associated with state A, for example, W 1A Only the large W stands out. 1A =1, W 2A =0, …, W 16A = 0, the first term of (Equation 2) becomes 1. The same applies to the weights associated with states B, C, and D.

[0095] In this way, the regularization term L in (Equation 2) REG is a function that becomes smaller as the value of some of the weights mentioned above becomes significantly larger. REG is the loss function L TOTAL By including it in, not only the usual loss function L but also the regularization term L REGTherefore, learning not only reduces the difference between the class probability output and the learning label (class probability correct data), but also progresses so that some of the weights in the final layer become significantly larger, as shown by the bold line in Figure 7B.

[0096] As a result, similar to the second embodiment, the validity of performing the second neural network calculation only on the image regions with the highest contributions as shown in FIG. 4B and FIG. 5 is increased.

[0097] In addition, λ in (Equation 1) is the regularization term L REG The larger λ is, the larger the influence of the regularization term L REG Priority is given to reducing , and learning proceeds so that the weights of only a few final layers become significantly larger, but this weakens the effect of reducing the normal loss function L (i.e., the difference between the class probability output and the learning label), and the error during learning increases.

[0098] In other words, the larger the λ, the larger the error during learning, but on the other hand, the effect of the second neural network calculation when executing state detection (during AI inference) increases. Therefore, an appropriate λ should be selected so that the accuracy of the second class probability output is maximized.

[0099] Learning based on a normal loss function L is also possible. In this case, learning is performed as described in FIG. 7A. For example, as described in the second embodiment, if the frequency-transformed image generation unit 61 is used, learning based on the normal loss function L is performed so that only the weights of some of the final layers become significantly larger. It is more effective to apply both the frequency-transformed image generation unit 61 and the learning unit with regularization term 71 of this embodiment.

[0100] In the above explanation, the neural network learning is performed using the circuit that actually performs state detection (i.e., the bit-number-switchable convolutional neural network operation circuit 12), but the same applies when the learning is performed on other hardware.

[0101] As described above, according to this embodiment, it is possible to improve the validity of performing the second neural network calculation using only the image regions with the highest contribution, as in the second embodiment, thereby achieving both high accuracy and high frequency (real-time) of condition detection. [Example]

[0102] A fourth embodiment will be described with reference to Figures 8A and 8B. This embodiment shows an example of a method for switching the number of bits in the bit-number-switchable convolutional neural network operation circuit 12 of the first embodiment. In particular, this embodiment relates to a configuration related to a multiplication circuit for product-sum operations in neural network operations.

[0103] Figure 8A shows a multiplication circuit with a low representation bit count that is executed in the first neural network calculation. Specifically, it shows the case of 4-bit representation. The pixel values ​​of an image or intermediate image are input. With this configuration, four different pixel values ​​A1, A2, A3, and A4, each expressed in 4 bits, can be multiplied by their respective weights W1, W2, W3, and W4 in parallel.

[0104] As shown in Figure 8A, multiplication is performed using multipliers 81A, 81B, 81C, and 81D. Weights W1, W2, W3, and W4 are expressed using a bit length necessary to ensure accuracy, such as 8 bits. In this case, each multiplier 81A, 81B, 81C, and 81D has an input of 4 bits x 8 bits and an output of 4 bits. The outputs of the four parallel multiplications are multiplied by 1 (i.e., the values ​​remain unchanged) by the subsequent multipliers 82A, 82B, 82C, and 82D, and then pass through or adder 83 directly to be output as four independent 4-bit multiplication outputs O1, O2, O3, and O4.

[0105] Figure 8B shows a multiplication circuit with a high representation bit count that is executed in the second neural network calculation. Specifically, it shows the case of 16-bit representation. As mentioned above, pixel values ​​from a part of the image area of ​​an image or intermediate image are input. With this configuration, one pixel value A expressed in 16 bits is multiplied by a weight W (expressed in 8 bits, for example).

[0106] A feature of this method is that the four multipliers 81A, 81B, 81C, and 81D with 4-bit input x 8-bit output and 4-bit output used in multiplication with a low number of representation bits (Fig. 8A) are also used in multiplication with a high number of representation bits (Fig. 8B). Therefore, the 16-bit pixel value A is divided into four parts starting from the most significant bits and supplied to the multipliers 81A, 81B, 81C, and 81D, respectively.

[0107] The outputs of these multipliers are multiplied by 1, 1 / 16, 1 / 256, and 1 / 4096 in the subsequent multipliers 82A, 82B, 82C, and 82D, respectively, and the multiplied outputs are passed through or added in addition unit 83 to obtain the multiplication result O of the pixel value A expressed in 16 bits and the weight W.

[0108] As described above, the pass or add unit 83 has a circuit configuration in which, when multiplying with a low number of representation bits (FIG. 8A), it passes the four inputs as they are, and when multiplying with a high number of representation bits (FIG. 8B), it adds the four inputs and outputs the result.

[0109] As mentioned above, multiplier 82A multiplies the input by 1 both when multiplying with a low number of representation bits (FIG. 8A) and when multiplying with a high number of representation bits (FIG. 8B), so there is no change in the value and it does not actually need to be implemented as a circuit.

[0110] Furthermore, since multipliers 82B, 82C, and 82D multiply by 1 when multiplying with a low number of representation bits (FIG. 8A), and by 1 / 16, 1 / 256, and 1 / 4096, respectively, when multiplying with a high number of representation bits (FIG. 8B), they may be implemented as a circuit that switches between a pass-through function and a right 4-bit shift function (1 / 16), a right 8-bit shift function (1 / 256), or a right 12-bit shift function (1 / 4096).

[0111] As described above, according to this embodiment, the first neural network calculation with a low number of representation bits and the second neural network calculation with a high number of representation bits can be realized by sharing a circuit.

[0112] Therefore, the circuit mounting area of ​​the calculation hardware that executes the sensor-based detection of the equipment status can be further reduced, and the calculation hardware can be reduced in cost and size. [Example]

[0113] 9A is a diagram showing the configuration of a status detection device according to a fifth embodiment. In this embodiment, the recalculation image area determination unit 16 in the first embodiment is replaced with a recalculation judgment and image area determination unit 91. Other configurations and operations are the same as those in the first embodiment, and therefore descriptions thereof will be omitted.

[0114] If the maximum value of the class probability included in the first class probability output (low-accuracy class probability) exceeds a predetermined threshold, the recalculation judgment and image area determination unit 91 does not perform the second neural network calculation, and the judgment unit 18 judges the state corresponding to the maximum class probability value to be the true state.

[0115] Although the first class probability output is low accuracy, if the class probability value is high enough to exceed the predetermined threshold, the corresponding state is likely to be the true state. This reduces the power consumption required for the second neural network calculation, as shown in Figure 9B.

[0116] If the maximum value of the class probabilities included in the first class probability output is equal to or smaller than a predetermined threshold value, the recalculation judgment and image area determination unit 91 instructs the scheduler 15 to perform recalculation and transmits the recalculated image area, and the second neural network calculation is performed in the same manner as in Example 1. Other configurations and operations are also the same as in Example 1.

[0117] Furthermore, if the ratio between the maximum probability and the second largest probability value in the first class probability output exceeds a predetermined threshold value, the second neural network operation may not be performed.

[0118] According to this embodiment, the power consumption of the computing hardware that executes the sensor-based detection of the equipment state can be further reduced, thereby enabling the computing hardware to be made smaller.

[0119] In the above-described embodiments, a program may be executed to perform processing. In this case, the computer executes the program using a processor (e.g., a CPU or a GPU) and performs the processing defined by the program while using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity that executes the program and performs the processing may be the processor. Similarly, the entity that executes the program and performs the processing may be a controller, device, system, computer, or node that has a processor.

[0120] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0121] In the above example, the equipment state is classified by converting the time-series signal from the sensor into an image and classifying it twice using a convolutional neural network. In the first classification, the input image of the convolutional neural network and the input image of the intermediate layer are represented using a low number of bits and then calculated. From the obtained classification results, several state candidates with high class probabilities are extracted.

[0122] In the second classification, only the image regions that contribute most to the extracted state candidates are recalculated using a convolutional neural network, and the corresponding regions of the input image and intermediate layer input image are represented using a higher number of bits for calculation. The state with the highest class probability in the second classification result is determined to be the final state.

[0123] According to the above embodiment, sensor-based equipment status detection can be performed using low-cost, small-sized computing hardware, and highly sensitive status detection can be achieved by taking advantage of the high resolution of the sensor signals. [Explanation of symbols]

[0124] 11 Image generation unit 12-bit switchable convolutional neural network arithmetic circuit 13 Image memory 14 Weight Memory 15 Scheduler 16 Recalculation image area determination unit 17 Image Area Contribution Table 18 Judgment section 31 Convolutional and Pooling Layers 32 Flatten layers 33 Fully connected layer 61 Frequency conversion image generation unit 71 Learning unit with regularization term 81A~81D Multiplier 82A~82D Multiplier 83 Pass or Add Section 91 Recalculation judgment and image area determination unit

Claims

1. A status detection device that detects a status of equipment using at least one sensor, an image generating unit that converts the time series signal from the sensor into an image; an arithmetic circuit that performs a convolutional neural network operation on the image with different representation bit numbers; a determination unit that determines a true state from among a plurality of states of the equipment based on the calculation result of the calculation circuit, The arithmetic circuit comprises: In the first calculation on the image, performing a first calculation on all image regions of the image using a first number of representation bits among the different number of representation bits to output first class probabilities for a plurality of states, and extracting a plurality of states having high probability values ​​from the plurality of first class probabilities as state candidates; In a second operation on the image, For each of the plurality of state candidates, a partial image region in the image having a high contribution is selected based on a weight value of a trained final layer of the convolutional neural network; performing a second calculation on the partial image region using a second number of representation bits, which is higher than the first number of representation bits, among the different number of representation bits, to output second class probabilities for each of the plurality of state candidates; The determination unit A state detection device characterized by determining the state candidate with the highest probability value among the plurality of second class probabilities as the true state of the equipment.

2. The arithmetic circuit comprises: In the first calculation on the image, 2. The condition detection device according to claim 1, wherein a plurality of states having a probability value exceeding a predetermined threshold value are extracted as the state candidates as the states having a high probability value.

3. The arithmetic circuit comprises: In the first calculation on the image, 2. The state detection device according to claim 1, wherein a plurality of states having a predetermined number of higher probability values ​​among the plurality of first class probabilities are extracted as the state candidates as the states with high probability values.

4. The arithmetic circuit comprises: In a second operation on the image, 2. The condition detection device according to claim 1, wherein the second calculation is performed on the partial image region and a partial region of an intermediate layer corresponding to the partial image region.

5. an image area contribution table that stores information on the partial image areas that have high contributions to the plurality of states; The arithmetic circuit comprises: In a second operation on the image, 2. The condition detection device according to claim 1, wherein the partial image area having a high contribution in the image is selected for each of the plurality of condition candidates by referring to the image area contribution table.

6. The image region contribution table includes:

6. The condition detection device according to claim 5, wherein the condition detection device is created by referring to the weight values ​​of the trained final layer of the convolutional neural network.

7. The image generation unit converting the time series signal into frequency spectrum data; The condition detection device according to claim 1 , wherein the image is generated based on the frequency spectrum data.

8. The image generation unit converting the plurality of time series signals from the plurality of sensors into the frequency spectrum data, respectively; 8. The condition detection device according to claim 7, wherein a single image is generated using a plurality of pieces of frequency spectrum data.

9. a learning unit with a regularization term that performs learning using a regularization term; The regularization term-added learning unit Applying values ​​obtained by learning based on a loss function including the regularization term to weight values ​​of the final layer; The condition detection device according to claim 1 , wherein the learning is performed so that the regularization term becomes smaller as some of the weight values ​​of the final layer become relatively larger.

10. The learning The condition detection device according to claim 9 , wherein the condition detection device is executed using the arithmetic circuit.

11. The arithmetic circuit comprises: The condition detection device according to claim 1, wherein the operations using the different number of representation bits are performed using a common multiplication circuit.

12. The arithmetic circuit comprises: In the first calculation, If the largest probability value among the plurality of first class probabilities exceeds a predetermined threshold, the second calculation is not performed; The determination unit The condition detection device according to claim 1 , wherein the candidate condition having the maximum probability value is determined as the true condition of the equipment.

13. The arithmetic circuit comprises: In the first calculation, If a ratio between the largest probability value and the second largest probability value among the plurality of first class probabilities exceeds a predetermined threshold, the second calculation is not performed; The determination unit The condition detection device according to claim 1 , wherein the candidate condition having the maximum probability value is determined as the true condition of the equipment.

14. The arithmetic circuit comprises: In the second calculation, If the maximum probability value among the plurality of second class probabilities is less than a predetermined threshold, 2. The condition detection device according to claim 1, further comprising: increasing the number of partial image regions with high contributions in the image; and further executing a third calculation on the image.

15. A status detection method for detecting a status of equipment using at least one sensor, comprising: an image generation step of converting the time series signal from the sensor into an image by an image generation unit; a calculation step of executing a convolutional neural network calculation on the image with different representation bit numbers by a calculation circuit; a determination step of determining a true state from among a plurality of states of the equipment based on the calculation result of the calculation circuit by a determination unit, The calculation step includes: In the first calculation on the image, performing a first calculation on all image regions of the image using a first number of representation bits of the two different number of representation bits to output first class probabilities for a plurality of states, and extracting a plurality of states having high probability values ​​from the plurality of first class probabilities as state candidates; In a second operation on the image, For each of the plurality of state candidates, a partial image region in the image having a high contribution is selected based on a weight value of a trained final layer of the convolutional neural network; performing a second calculation on the partial image region using a second number of representation bits, which is higher than the first number of representation bits, among the different number of representation bits, to output second class probabilities for each of the plurality of state candidates; The determining step A state detection method, characterized in that the state candidate with the highest probability value among the plurality of second class probabilities is determined to be the true state of the equipment.

Images