State recognition device, state recognition method, and state recognition program

Abstract

To provide a technology capable of easily perform a response which increases the number of observation data used for recognizing a state of an observation target, and improves recognition accuracy of the state of the observation target.SOLUTION: Observation data of observation characteristics of an observation target observed by a sensor are inputted in an input unit. A recognition unit recognizes a state of the observation target, by a recognition model generated by machine learning, on the basis of the observation data inputted to the input unit. An output unit outputs the state of the observation target recognized by the recognition unit. Moreover, the input unit has a plurality of input channels to which the observation data are inputted, and a part of the channels is a non-operation channel to which the observation data has not been inputted.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This invention relates to a technique for recognizing the state of an observation target using a recognition model generated by machine learning.

Background Art

[0002] Conventionally, there has been a device that observes the observation characteristics of an observation target with a sensor and recognizes the state (for example, a normal state or an abnormal state) of the observation target with a recognition model generated by machine learning based on the observation data of this sensor. For example, the device described in Patent Document 1 utilizes a variational autoencoder (hereinafter referred to as VAE) as a recognition model for recognizing the state of an observation target.

[0003] As is well known, VAE is a model including an encoder that calculates a latent variable from observation data which is input data, and a decoder that restores observation data from the latent variable. VAE is mainly generated by machine learning using observation data during normal times of an observation target. Here, the observation data restored from the latent variable is referred to as restored data.

[0004] The device described in Patent Document 1 calculates the difference between the observation data and the restored data, and recognizes whether the state of the observation target is normal or abnormal based on this difference.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, with the conventional device configuration, there was a problem in that it was time-consuming and laborious to increase the number of observational data used to recognize the state of the observed object and thereby improve the accuracy of recognizing the state of the observed object. Specifically, it was time-consuming and laborious because it required creating new source code to accommodate the system with increased sensors, and then using this new source code to perform machine learning and generate a recognition model.

[0007] Furthermore, increasing the number of observation data points as described above may involve adding sensors of a type that is not currently being used, or adding sensors of the same type as those already being used, but at a different observation location.

[0008] The purpose of this invention is to provide a technology that allows for easy implementation of measures to improve the accuracy of recognizing the state of an observed object by increasing the number of observational data used to recognize the state of the observed object. [Means for solving the problem]

[0009] To achieve the above objective, the state recognition device of this invention is configured as shown below.

[0010] The recognition unit has a recognition model generated by machine learning using observation data of the observed characteristics of the observed object observed by the sensor, and recognizes the state of the observed object. The recognition model of the recognition unit may be, for example, a variational autoencoder (VAE), or it may be something other than a VAE. Furthermore, the recognition unit may recognize, for example, whether the state of the observed object is normal or abnormal, or it may recognize that it is one of three or more predetermined states.

[0011] The input unit receives observational data of the observed characteristics of the observed object, as observed by the sensor. The observed object is an object whose state is recognized, such as space, a person, an object, or a substance. The observational data is sensing data obtained by sensing the observed object with a sensor. The observational data may be, for example, a frame image of the observed object captured by a camera, audio data of the surrounding area of ​​the observed object collected by a microphone, the temperature of the observed object measured by a temperature sensor, or other data. In other words, the observational characteristics are determined by the type of sensor. Furthermore, the input unit may receive one or more observational data.

[0012] The output unit outputs the state of the observed object recognized by the recognition unit, based on the observation data input to the input unit.

[0013] Furthermore, the input unit has multiple input channels to which observation data is input. Among these multiple input channels is an inoperable channel to which no observation data of the observation characteristics of the object being observed by the sensor is input. For example, an inoperable channel is a channel to which no sensor is connected.

[0014] With this configuration, to improve the accuracy of recognizing the state of the observed object by increasing the number of observational data points used to recognize its state, the additional sensor should be connected to a non-operational channel. The non-operational channel to which the additional sensor is connected will then be switched to an operational channel. Therefore, when increasing the number of observational data points used to recognize the state of the observed object in order to improve the accuracy of recognizing its state, it is not necessary to create new source code. Furthermore, machine learning of the recognition model after increasing the number of observational data points used to recognize the state of the observed object can be efficiently performed using transfer learning that utilizes the recognition model before increasing the number of observational data points.

[0015] Therefore, it is easy to improve the accuracy of recognizing the state of the observed object by increasing the number of observational data used to recognize its state.

[0016] Further, for example, the recognition model may be generated by machine learning using random numbers that follow a normal distribution as observation data of a non-operating channel.

[0017] Also, for example, it may be configured to include an auxiliary input unit that inputs virtual observation data to a non-operating channel. In this case, for example, virtual observation data that does not affect the recognition result of the state of the observation target obtained during the machine learning of the recognition model may be input to the non-operating channel.

Advantages of the Invention

[0018] According to this invention, it is possible to easily increase the number of observation data used for recognizing the state of the observation target and improve the recognition accuracy of the state of the observation target.

Brief Description of the Drawings

[0019] [Figure 1] It is a diagram showing a monitoring system to which the state recognition device of this example is applied. [Figure 2] It is a diagram for explaining the input switching unit of the state recognition device of this example. [Figure 3] It is a schematic diagram for explaining a variational autoencoder. [Figure 4] It is a flowchart showing the state recognition process during the operation of the state recognition device of this example. [Figure 5] It is a flowchart showing the update process for updating the recognition model. [Figure 6] It is a flowchart showing the state recognition process during the operation of the state recognition device of Modification 1.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, embodiments of this invention will be described.

[0021] <1. Application Example> Figure 1 shows a monitoring system to which the state recognition device of this example is applied. The monitoring system 100 of this example has a state recognition device 1 and a plurality of sensors 2. This monitoring system 100 monitors the state of an object to be observed. The object to be observed is an object whose state is recognized, and can be, for example, a space, a person, an object, or a substance. For example, the object to be observed may be an office building, an apartment building, a bridge, a road on which vehicles travel, a train station or theme park used by an unspecified number of people, or something other than these.

[0022] The state recognition device 1 recognizes the state of the object being observed and outputs the recognized state to a higher-level device (not shown). In this example, the state recognition device 1 recognizes whether the state of the object being observed is an abnormal state.

[0023] The state recognition device 1 may also recognize which of three or more states the observed object is in.

[0024] The state recognition device 1 has n input channels to which sensors 2 are connected. n is 2 or greater. Figure 1 shows three sensors 2 (2a-2) connected to the state recognition device 1, but this does not mean that the number of sensors 2 connected to the state recognition device 1 is limited to three. In Figure 1, sensor 2a is connected to input channel ch1, sensor 2b is connected to input channel ch2, and sensor 2c is connected to input channel ch3.

[0025] Furthermore, Figure 1 shows an example where sensor 2 is not connected to input channel ch(n-1) and input channel ch(n). However, this does not mean that there are only two input channels ch to which sensor 2 is not connected. Here, input channels ch to which sensor 2 is connected are called operational channels, and input channels ch to which sensor 2 is not connected are called non-operational channels.

[0026] Figure 1 illustrates a state of the monitoring system 100 in which sensors 2 are connected to some of the n input channels of the state recognition device 1, and sensors 2 are not connected to the remaining input channels.

[0027] Sensor 2 outputs observation data, which is obtained by sensing the observation characteristics of the object being observed, to the state recognition device 1. Sensor 2 is a type of sensor that observes observation characteristics, and examples include temperature sensors, acceleration sensors, strain sensors, current sensors, voltage sensors, light sensors, sound sensors, and flow sensors. Multiple sensors of the same type may be included in Sensor 2 connected to input channel ch. For example, two temperature sensors may be used to observe the temperature at different observation locations of the object being observed, and these temperature sensors may be connected to different input channels.

[0028] The state recognition device 1 has a recognition model that recognizes the state of the object being observed based on observation data from sensor 2 connected to the input channel. This recognition model is, for example, a model generated by machine learning (e.g., deep learning) using observation data from sensor 2 connected to the operation channel. Furthermore, the machine learning of the recognition model is performed according to the source code (learning algorithm) for n input channels. When the state recognition device 1 recognizes the state of the object being observed based on observation data from sensor 2 connected to the operation channel, it outputs the recognition result of the state of the object being observed to a higher-level device.

[0029] Furthermore, in this example, the state recognition device 1 can increase the number of observational data points used to recognize the state of the observed object in order to improve the accuracy of recognizing the state of the observed object. Specifically, the user connects sensor 2, which provides additional observational data, to an inactive channel of state recognition device 1 at that time. The user then regenerates the recognition model using machine learning with the observational data, including the observational data from sensor 2 that was added this time. This machine learning for regenerating the recognition model is also performed according to the source code with n input channels. Moreover, this machine learning for regenerating the recognition model is performed using transfer learning that utilizes the recognition model at that time.

[0030] Thus, in this example, the state recognition device 1 does not require the creation of new source code when increasing the number of observational data used to recognize the state of the observed object in order to improve the accuracy of recognizing the state of the observed object. Furthermore, the machine learning required to regenerate the recognition model in response to an increase in the number of observational data used to recognize the state of the observed object can be efficiently performed using transfer learning. Therefore, the state recognition device 1 in this example can easily increase the number of observational data used to recognize the state of the observed object and improve the accuracy of recognizing the state of the observed object.

[0031] <2. Example Configuration> Referring to Figure 1, the configuration of the main parts of the state recognition device 1 in this example will be described. As shown in Figure 1, the state recognition device 1 includes a control unit 11, an input unit 12, a virtual observation data output unit 13, an input switching unit 14, and an output unit 15.

[0032] The control unit 11 controls the operation of each part of the main body of the state recognition device 1. The control unit 11 has a recognition unit 11a and a switching control unit 11b. The recognition unit 11a and the switching control unit 11b will be described later.

[0033] The input unit 12 is an input interface that receives observation data from the sensor 2 connected to each input channel. No observation data from the sensor 2 is input to input channels to which the sensor 2 is not connected (non-operational channels).

[0034] The virtual observation data output unit 13 outputs virtual observation data to be input to the non-operational channel.

[0035] The input switching unit 14 is a switch circuit that switches the input to the control unit 11 for each input channel. The input switching unit 14 switches the input channels m (where m is 1 to n) to which the sensor 2 is connected, inputting the observation data from the sensor 2 to the control unit (see Figure 2(A)), and inputting virtual observation data to the control unit for input channels m to which the sensor 2 is not connected (see Figure 2(B)).

[0036] The virtual observation data output unit 13 and the input switching unit 14 correspond to the auxiliary input unit in this invention.

[0037] The output unit 15 outputs the recognition result of the state of the observed object in the control unit 11 to the higher-level device.

[0038] Next, the recognition unit 11a and the switching control unit 11b of the control unit 11 will be described.

[0039] The recognition unit 11a has a recognition model generated by machine learning using observation data from the sensor 2 connected to the input unit 12. The recognition model may be, for example, a variational autoencoder (VAE), or it may be something other than a VAE. In this example, the recognition model will be described as a VAE.

[0040] Figure 3 is a schematic diagram illustrating a variational autoencoder (VAE). As is well known, the VAE20 is composed of a computer. The VAE20 is a model generated by machine learning (deep learning) using normal observation data of the observed object, and has an encoder 21 and a decoder 22. The encoder 21 and decoder 22 are neural networks. The encoder 21 extracts the latent variable z from the observation data input to each input channel. The latent variable z is a feature obtained from the observation data input to each input channel. The decoder 22 takes the latent variable z as input and reconstructs the observation data for each input channel. Here, the observation data reconstructed by the decoder 22 from the latent variable z is called the reconstructed data. The observation data is, for example, a sequence of numerical values ​​arranged in time.

[0041] In deep learning, the encoder 21 learns a conditional probability q(z|x) for extracting a latent variable z from the observed data input to each input channel, and the decoder 22 learns a conditional probability p(x|z) for reconstructing the observed data from the latent variable z.

[0042] The recognition unit 11a determines whether the difference between the observed data and the reconstructed data exceeds a predetermined threshold for each input channel. The difference between the observed data and the reconstructed data may be a difference, a ratio, or something else. The threshold may be set for each input channel or may be common to all input channels.

[0043] The recognition unit 11a determines that the state of the observed object is abnormal if there is an input channel where the difference between the observed data and the reconstructed data exceeds a predetermined threshold. The recognition unit 11a also determines that an abnormality has occurred in an item related to the observation characteristics observed by the sensor 2 connected to the input channel where the difference between the observed data and the reconstructed data exceeds a predetermined threshold.

[0044] Furthermore, for non-operational channels to which sensor 2 is not connected, the recognition unit 11a determines that the threshold has not been exceeded (i.e., it is normal) without determining the difference between the observed data (actually, virtual observed data) and the reconstructed data.

[0045] The switching control unit 11b controls the switching of the switch circuit in the input switching unit 14 for each input channel. Specifically, for operational channels to which sensor 2 is connected, the switching control unit 11b controls the switch circuit of the input switching unit 14 so that the observation data from sensor 2 is input to the control unit 11 (controlling it to the state shown in Figure 2(A)). For non-operational channels to which sensor 2 is not connected, the switching control unit 11b controls the switch circuit of the input switching unit 14 so that virtual observation data is input to the control unit 11 (controlling it to the state shown in Figure 2(B)).

[0046] The hardware CPU constituting the control unit 11 of the state recognition device 1 operates as a recognition unit 11a and a switching control unit 11b when it executes the state recognition program according to this invention. The memory also has an area for deploying the state recognition determination program according to this invention and an area for temporarily storing data generated when the state recognition program is executed. The control unit 11 may be an LSI integrating the hardware CPU, memory, etc. The hardware CPU is a computer that executes the state recognition method according to this invention.

[0047] <3. Example of operation> Figure 4 is a flowchart showing the state recognition process during operation of this example state recognition device. The switching control unit 11b controls the input switching unit 14 so that, during operation, observation data from the sensor 2 connected to the operational channel is input to the control unit 11, and virtual observation data from the virtual observation data output unit 13 is input to the control unit 11 for non-operational channels. The virtual observation data, for example, always has a value of 0.

[0048] The recognition unit 11a acquires observation data for each input channel (operational channel and non-operational channel) (s1). For example, in s1, for each input channel, data is acquired that consists of observation data (or virtual observation data) that was input to that input channel during the preceding predetermined time period, arranged in time series.

[0049] The encoder 21 of the VAE20 encodes the observed data for each input channel acquired in s1 (s2), and obtains the resulting latent variable z as a feature (s3). The decoder 22 of the VAE20 decodes the feature (latent variable z) acquired in s3 (s4), and obtains the reconstructed data for each input channel (s5).

[0050] The recognition unit 11a selects the operational channel to be processed (s6), and compares the observed data input to the operational channel selected in s6 with the reconstructed data (s7). Based on the comparison of the observed data and the reconstructed data in s7, the recognition unit 11a performs an individual determination to determine whether the state of the items related to the observation characteristics observed by the sensor 2 connected to the operational channel to be processed is normal or abnormal (s8).

[0051] The recognition unit 11a determines whether there are any unprocessed operational channels (operational channels that have not undergone the processing described in s6 to s8) (s9), and if there are any unprocessed operational channels, it returns to s6 and repeats the above processing.

[0052] If the recognition unit 11a determines in s9 that there are no unprocessed operational channels, it performs an overall determination to determine whether the state of the observed target is normal or abnormal (s10). In the overall determination in s10, for example, (1) If one or more operational channels are determined to be abnormal in s8, the state of the observed target may be determined to be abnormal. (2) If there is one or more operational channels among the predetermined judgment items that were determined to be abnormal in s8, the state of the observed object may be determined to be abnormal. (3) If the number of operational channels determined to be abnormal in s8 exceeds a predetermined number, the state of the observed object may be determined to be abnormal. Other criteria may be used to determine whether the observed object is in a normal or abnormal state.

[0053] The state recognition device 1 outputs the recognition result of the state of the observed object by the recognition unit 11a to the higher-level device (s11), and returns to s1. The recognition result output in s11 may indicate whether the state of the observed object is normal or abnormal, or, if it is abnormal, it may output which observation characteristic item is abnormal.

[0054] In this way, the state recognition device 1 outputs the recognition result to the higher-level device, determining whether the state of the observed object is normal or abnormal during operation.

[0055] Next, we will describe the process of updating the recognition model in the state recognition device 1 by increasing the number of observation characteristics observed for the observed object, that is, by adding a sensor 2 that observes the observation characteristics of the observed object, in order to improve the accuracy of recognizing the state of the observed object. Figure 5 is a flowchart of the update process for updating the recognition model. When updating the recognition model of the recognition unit 11a, multiple training data sets used in machine learning are prepared. These training data sets consist of observation data from each channel. In the training data sets, the operational channels consist of observation data from the sensor 2 connected to that operational channel, or data generated based on this observation data. In the training data sets, the non-operational channels are assigned random numbers generated according to a normal distribution.

[0056] Note that the channel to which sensor 2 is added is the operational channel.

[0057] The state recognition device 1 receives a supply of multiple prepared training data (s21), performs retraining to update the recognition model using machine learning with the supplied training data (s22), and then terminates this process. In s21, the state recognition device 1 is supplied with training data stored in a storage medium (for example, an HDD (Hard Disk Drive) or SSD (Solid State Drive)) that stores multiple training data.

[0058] Furthermore, in this example, the state recognition device 1 utilizes an unused channel as a channel to add a sensor 2 that observes the observation characteristics of the observed object in order to improve the accuracy of recognizing the state of the observed object. Therefore, in s22, the state recognition device 1 can be made to retrain the recognition model without creating new source code.

[0059] Furthermore, in s22, the recognition model is retrained using transfer learning that utilizes the previous recognition model. Therefore, the recognition model can be retrained efficiently, and the time required for retraining can be reduced. In other words, in the state recognition device 1, the period during which operation is stopped in order to increase the number of observation characteristics observed for the observed object in order to improve the recognition accuracy of the state of the observed object can be reduced.

[0060] Furthermore, for operational channels to which Sensor 2 is connected, if it is determined that this does not affect the accuracy of recognizing the state of the observed object, these operational channels can be switched to non-operational channels. In this case as well, the recognition model can be updated by retraining as shown in Figure 5.

[0061] <4. Variation> • Variation 1 The state recognition device 1 of the modified example 1 will now be described.

[0062] The state recognition device 1 in this modified example 1 has the configuration shown in Figure 1. Figure 6 is a flowchart of the state recognition process during operation of the state recognition device in modified example 1. In Figure 6, the same steps as those shown in Figure 4 are given the same step numbers.

[0063] In this modified version 1, the state recognition device 1, after executing the processes s1 to s5 described above, compares the distribution of features obtained in s3 with the distribution of features of the observed data when the observed object is in a normal state, and determines whether the state of the observed object is normal or abnormal (s31). If the state recognition device 1 in this modified version 1 determines in s31 that the observed object is in a normal state (s32), it outputs a recognition result of "the observed object is in a normal state" in s11 without executing the processes s6 to s9 described above. On the other hand, if the state recognition device 1 in this modified version 1 determines in s31 that the observed object is in an abnormal state (s32), it executes the processes s6 to s9 described above and outputs a recognition result of "the observed object is abnormal" in s11. If the state recognition device 1 in this modified version 1 determines in s31 that the observed object is in an abnormal state, it may also output which observation characteristic item is in an abnormal state.

[0064] In this modified example 1, the state recognition device 1 does not execute the processes described in s6 to s9 if the state of the observed object is normal, thus reducing the processing load.

[0065] • Variation 2 In the example above, the state recognition device 1 inputs virtual observation data, whose value is always 0, into the non-operation channel during operation. However, during the training of the recognition model, it is also possible to obtain a value that does not affect the recognition result of the state of the observed object, and input this value as virtual observation data into the non-operation channel.

[0066] It should be noted that this invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Also, the order of the steps shown in Figures 4 to 6 is just one example, and the order may be changed as appropriate. Moreover, components from different embodiments may be combined as appropriate.

[0067] Furthermore, the correspondence between the configuration of this invention and the configuration of the embodiment described above can be described as follows. <Note> A recognition unit (11a) has a recognition model generated by machine learning using observation data of the observation characteristics of the observed object observed by sensor (2), and recognizes the state of the observed object. An input unit (12) into which the observation data of the observation characteristics of the observed target observed by the sensor (2) is input, The system includes an output unit (15) that outputs the state of the observed object recognized by the recognition unit (11a) based on the observation data input to the input unit (12), The aforementioned input unit (12) is The system has multiple input channels (ch1~chn) into which the aforementioned observation data is input. The aforementioned plurality of input channels (ch1~chn) include non-operational channels in which no observation data has been input, in which the sensor (2) has observed the observation characteristics of the observed target. State recognition device (1). [Explanation of Symbols]

[0068] ch1~chn...Input Channels 1... State recognition device 2(2a~2c)...Sensor 11…Control Unit 11a...Recognition part 11b…Switching control unit 12...Input section 13…Virtual observation data output unit 14…Input switching section 15…Output section 20…VAE 21… Encoder 22… Decoder 100... Surveillance system

Claims

1. A recognition unit that recognizes the state of the observed object, having a recognition model generated by machine learning using observation data of the observed characteristics of the observed object observed by a sensor, An input unit into which the observation data of the observed object and its observed characteristics, observed by the sensor, is input. The system comprises an output unit that outputs the state of the observed object recognized by the recognition unit based on the observed data input to the input unit, The aforementioned input unit is The system has multiple input channels to which the aforementioned observation data is input, The aforementioned plurality of input channels include non-operational channels in which no observation data is input, in which the sensor has observed the observation characteristics of the observed target. It includes an auxiliary input unit that inputs virtual observation data to the non-operational channel, State recognition device.

2. The state recognition device according to claim 1, wherein the recognition model is generated by machine learning using random numbers following a normal distribution as observation data of the non-operational channel.

3. The aforementioned recognition model is generated using machine learning with random numbers following a normal distribution as the observation data of the non-operational channel. The state recognition device according to claim 1, further comprising an auxiliary input unit that inputs virtual observation data obtained during machine learning of the recognition model, which does not affect the recognition result of the state of the observed object in the recognition unit, to the non-operational channel.

4. The state recognition device according to any one of claims 1 to 3, wherein the recognition model includes a variational autoencoder generated by machine learning.

5. The state recognition device according to claim 4, wherein the recognition unit determines the state of the observed object for each input channel based on the difference between the observed data input to that input channel and the output data of the variational autoencoder relating to the observed data.

6. The state recognition device according to claim 5, wherein the recognition unit does not use the difference between the input data and the output data of the variational autoencoder for the non-operational channel to determine the state of the object being observed.

7. Computers A recognition step in which a recognition model generated by machine learning using observation data of the observed characteristics of the observed object observed by a sensor recognizes the state of the observed object, An input step in which the input unit receives the observation data of the observation characteristics of the observed target observed by the sensor, Based on the observation data input to the input unit, an output step is performed in which the state of the observed object recognized in the recognition step is output to the output unit. The aforementioned input unit is The system has multiple input channels to which the aforementioned observation data is input, The aforementioned plurality of input channels include non-operational channels in which no observation data is input, in which the sensor has observed the observation characteristics of the observed target. The computer performs an auxiliary input step in which it causes the auxiliary input unit to input virtual observation data to the non-operational channel. A method for recognizing the state.

8. On the computer, A recognition step in which a recognition model generated by machine learning using observation data of the observed characteristics of the observed object observed by a sensor recognizes the state of the observed object, An input step in which the input unit receives the observation data of the observation characteristics of the observed target observed by the sensor, Based on the observation data input to the input unit, the system performs an output step in which the state of the observed object recognized in the recognition step is output to the output unit. The aforementioned input unit is The system has multiple input channels to which the aforementioned observation data is input, The aforementioned plurality of input channels include non-operational channels in which no observation data is input, in which the sensor has observed the observation characteristics of the observed target. The computer is instructed to perform an auxiliary input step, which involves having the auxiliary input unit input virtual observation data to the non-operational channel. State recognition program.

Images