Abnormal determination device and abnormal determination method

Description

Technical Field

[0001] The disclosed technology relates to an abnormality determination technology for determining the state of a person to be determined.

Background Art

[0002] Among abnormality determination technologies, there are those that determine the state of a person to be determined using measurement results from a sensor device (such as a camera, radar, infrared sensor, etc.). Among such abnormality determination technologies, for example, in an abnormality determination technology for determining the state of a person to be determined such as a driver (passenger) of a vehicle (mobile body), since it is related to the safety of the person to be determined, etc., it is desirable to perform determination with high accuracy so that false determination does not occur. On the other hand, in recent years, research on technologies using deep learning has been progressing in abnormality determination technologies. In deep learning, a mathematical model called a neural network (Neural Network = NN) is used. A neural network outputs results through layers such as an input layer, a hidden layer (intermediate layer), and an output layer. Patent Document 1 shows an identification device that uses a convolutional neural network (Convolutional Neural Network = CNN), which is a type of neural network, to identify that the driver of a mobile body is in an abnormal state. Specifically, in the process of identifying the state of the driver (such as whether it is an abnormal state or not) based on the acquired image, the identification device of Patent Document 1 acquires the skeletal information of the driver from the image using a convolutional neural network. The identification device of Patent Document 1 improves the accuracy of the skeletal information of the driver using a neural network.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, in the abnormality determination technology for a moving body as described above, quick determination is often required along with higher accuracy. When attempting to achieve higher accuracy using the neural network described above in the abnormality determination technology, for example, in the neural network, a method of extracting and using a large number of determination indexes (for example, various types of indexes such as the opening and closing state of eyelids, the number of blinks, the position of the face, the state of brain waves, the heat dissipation state, driving performance (reaction time, etc.)) indicating the state of the person to be determined from an image can be considered. If, by the above method, an attempt is made to output a determination result of an abnormal state with high accuracy using a neural network, the number of layers in the neural network will increase, or the configuration will become complex, resulting in an increase in the amount of calculation and the calculation time. Therefore, in the abnormality determination technology, there has been a problem that it is difficult to quickly output a highly accurate determination result using a neural network. The identification device of Patent Document 1 only outputs skeletal information from an image via a neural network and cannot solve the above problems.

[0005] The present disclosure solves the above problems and aims to enable quick output of a highly accurate determination result using a neural network in the abnormality determination technology.

Means for Solving the Problems

[0006] The abnormality determination device of the present disclosure is an image acquisition unit that acquires and outputs an image, a neural network unit having a plurality of layers in a neural network, acquiring the image output by the image acquisition unit, and outputting a determination result indicating the state of the determination target included in the image. An output storage unit that stores the output data output by the neural network unit; A reference value that is a reference for the input value to the first layer, which is at least one layer among the plurality of layers of the neural network unit, is stored in advance, and the difference value between the reference value and the current value, which is the value to be newly input to the first layer, is used to determine the change in the state to be determined. A state comparison unit; When the state comparison unit determines that the state to be determined has not changed, the neural network unit is commanded not to execute the processing in the layers after the first layer, and the output data stored in the output storage unit is commanded to be output. A processing control unit; It is provided with.

Effect of the Invention

[0007] According to the present disclosure, in the abnormality determination technology, it is possible to quickly output a highly accurate determination result using a neural network.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0009] Hereinafter, in order to explain the present disclosure in more detail, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the following description, ordinal numbers such as "first", "second", "third", etc. may be used. These terms are used for convenience to facilitate understanding of the content of the embodiments, and are not limited to the order or the like that may be caused by these ordinal numbers.

[0010] Embodiment 1. Embodiment 1 describes one form of the basic configuration of the present disclosure.

[0011] An example of the configuration of the abnormality warning device 100A and the abnormality determination device 1000A according to Embodiment 1 will be described. FIG. 1 is a diagram showing an example of the configuration of the abnormality warning device 100A and the abnormality determination device 1000A according to Embodiment 1 of the present disclosure. The abnormality warning device 100A acquires an image, uses the image to determine the state of the determination target imaged in the image, and outputs an alarm when the state of the determination target indicates an abnormality. The abnormality warning device 100 shown in FIG. 1 includes an abnormality determination device 1000A and an alarm output unit 2000.

[0012] The abnormality determination device 1000A acquires an image and uses the image to output the state of the determination target imaged in the image. The state of the determination target is output data including values in forms such as, for example, a determination value and a probability value. The abnormality determination device 1000A includes an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, and an output storage unit 1050.

[0013] The image acquisition unit 1010 acquires and outputs an image. The image acquisition unit 1010 acquires an image, for example, when an image (moving image or still image) captured by a camera is input. The camera is, for example, an imaging device installed in the vehicle interior, which images a determination target (the determination target is, for example, a living body such as an occupant including a driver) existing in the vehicle interior and outputs the captured image. When the image is a moving image, an image obtained by dividing the moving image into still images (frames) at regular time intervals by a camera is input to the image acquisition unit 1010.

[0014] The neural network unit 1020 is composed of a neural network. The neural network unit 1020 has a plurality of layers in the neural network, acquires the image output by the image acquisition unit 1010, and outputs output data which is a determination result indicating the state of the determination target included in the image. The plurality of layers in the neural network are an input layer, a hidden layer (intermediate layer), and an output layer. More specifically, for example, they are layers such as a convolutional layer and a pooling layer. Here, a neural network with many (deep) layers is particularly called a DNN (Deep Neural Network). As derivatives, for example, there are CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), etc. CNN is widely used in the fields of object recognition and image recognition, and RNN is widely used in time series processing, speech recognition, natural language processing, etc. The neural network in the present disclosure is applicable regardless of the form of the neural network. In this description, the form of CNN will be described as an example.

[0015] The state comparison unit 1030 determines the change in the state of the determination target. The state comparison unit 1030 stores in advance a reference value which is a reference for the input value for at least one layer among the plurality of layers of the neural network unit 1020, and uses the difference value between the reference value and the current value which is the value to be newly input to the first layer to determine the change in the state of the determination target. The first layer may be a predetermined one layer among all the layers in the neural network unit 1020, or may be a plurality of predetermined partial layers among all the layers, or may be each of all the layers.

[0016] The reference value is the previous value stored as the input value of the first layer, which is at least one layer, among the plurality of layers of the neural network unit 1020, as the previous processing result, and is used to obtain the difference value between the previous value and the current value, which is the input value to be newly input to the first layer this time. In the description of the processing to be described later, the case where the previous value is used as the reference value will be described. However, the reference value may be stored and used as the following values.

[0017] As the reference value, a typical processing result learned and modeled in advance can be used. That is, the reference value may be the output value of a typical intermediate layer modeled in advance in the neural network. By using a typical processing result, comparison can be performed without being affected by noise or anomalies included in the actually observed image.

[0018] Also, as the reference value, some results determined by prior information can be used. Some results determined by prior information are assumed to be, for example, adopting pixels with a high probability of the presence of a head. Also, for example, it is assumed that pixels with large changes but low importance, such as the background, are not adopted. That is, the reference value may be the output value of predetermined nodes among the nodes included in the neural network. Thereby, memory saving and reduction of data to be handled can be achieved, and higher speed can be achieved.

[0019] The state comparison unit 1030 determines the change in the state to be determined based on the result of determining the magnitude of the difference between the reference value and the current value by using the difference value and a threshold value stored in advance. Specifically, for example, when the sum of the squares of the difference values is smaller than the threshold value stored in advance for each unit of the image, the state comparison unit 1030 determines that the state to be determined has not changed. Specifically, for example, when the sum of the absolute values of the difference values is smaller than a threshold value stored in advance for each unit of an image, the state comparison unit 1030 determines that the state of the determination target has not changed.

[0020] The process control unit 1040 issues a command to limit the current processing operation in the neural network unit 1020 according to the change in the state of the determination target, and also issues a command to output the output data output by the previous processing of the neural network unit 1020. When the state comparison unit 1030 determines that the state of the determination target has not changed (not changed includes cases where the change is small), the process control unit 1040 commands the neural network unit 1020 not to execute the processing in the layer after the first layer, which is at least one of the plurality of layers, and commands to output the output data stored in the output storage unit 1050. The output data is, for example, a determination value indicating the state of the determination target person. The determination value is also referred to as a probability value including a probabilistic element in the embodiment of the present disclosure.

[0021] The output storage unit 1050 stores the output data (determination value (probability value)) output by the neural network unit 1020. When receiving a command from the process control unit 1040, the output storage unit 1050 outputs the stored output data. The output storage unit 1050 outputs the output data to, for example, the alarm output unit 2000. The output storage unit 1050 holds the latest output data output by the processing of the neural network unit 1020. For example, each time the output storage unit 1050 acquires the output data output by the neural network unit 1020, it deletes the previously stored output data and stores the latest output data.

[0022] The alarm output unit 2000 acquires the output data and outputs an alarm signal based on the output data to an alarm device (not shown) or the like. The alarm output unit 2000 acquires output data which is a determination value indicating the drowsy state or the dozing state of the person to be determined, and outputs an alarm for the person to be determined according to the determination value.

[0023] Although the abnormality determination device 1000A shown in FIG. 1 is shown as a configuration not including the alarm output unit 2000, it may be configured to include the alarm output unit 2000. When configured in this way, the abnormality determination device 1000A is equivalent to the abnormality alarm device 100A shown in FIG. 14. In the following description, unless it is necessary to distinguish between the abnormality alarm device 100A and the abnormality determination device 1000A, the abnormality determination device 1000A will be described as being configured to include the alarm output unit 2000.

[0024] In addition to the above configuration, the abnormality determination device 1000A may include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire abnormality determination device 1000A and each component. The control unit (not shown) activates the abnormality determination device 1000A according to a command from the outside, for example. Also, the control unit (not shown) controls the state of the abnormality determination device 1000A (operation states such as startup, shutdown, sleep). The storage unit (not shown) stores each data used in the abnormality determination device 1000A. The storage unit (not shown) stores, for example, the output (output data) by each component in the abnormality determination device 1000A, and outputs the data requested for each component to the component that requested it. The communication unit (not shown) communicates with an external device. For example, communication is performed between the abnormality determination device 1000A and an imaging device such as an in-vehicle camera. Also, for example, when the abnormality determination device 1000A does not include a display unit or an audio output unit, communication is performed between the abnormality determination device 1000A and an external device such as a display device or an audio output device.

[0025] An example of the processing of the abnormality alarm device 100A and the abnormality determination device 1000A according to Embodiment 1 will be described. FIG. 2 is a flowchart showing an example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to Embodiment 1 of the present disclosure. When an image is input from, for example, a camera, the abnormality determination device 1000A starts the processing shown in FIG. 2.

[0026] The abnormality determination device 1000A executes image acquisition processing (step ST100). In the image acquisition processing, the image acquisition unit 1010 of the abnormality determination device 1000A acquires and outputs an image.

[0027] The abnormality determination device 1000A executes state storage and state comparison processing (step ST200). In the state storage and state comparison processing, the state comparison unit 1030 of the abnormality determination device 1000A stores the previous value, which is the input value of the first layer, which is at least one of all the layers of the neural network unit 1020 at least once after the start of processing. The state comparison unit 1030 only stores the previous value without performing the state comparison process for the first time, but determines the change in the state to be determined after the second time. The state comparison unit 1030 stores in advance a reference value that is a reference for the input value for at least one layer, which is the first layer, among the plurality of layers of the neural network unit 1020, and uses the difference value between the reference value and the current value, which is the value to be newly input to the first layer, to determine the change in the state to be determined.

[0028] The abnormality determination device 1000A executes processing control processing (step ST300). In the processing control processing, the processing control unit 1040 of the abnormality determination device 1000A issues a command to limit the current processing operation in the neural network unit 1020 according to the change in the state to be determined, and issues a command to output the output data output by the previous processing of the neural network unit 1020. Specifically, when the state comparison unit 1030 determines that the state to be determined has not changed (not changed includes cases where the change is small), the processing control unit 1040 instructs the neural network unit 1020 not to execute the processing in the layer after the first layer, which is at least one of the plurality of layers, and also instructs the output storage unit 1050 to output the output data stored therein.

[0029] The abnormality determination device 1000A executes state output processing (step ST400). In the state output processing, the neural network unit 1020 or the output storage unit 1050 of the abnormality determination device 1000A outputs output data. When the processing is performed in all layers, the neural network unit 1020 outputs the output data to the alarm output unit 2000. When receiving a command from the processing control unit 1040, the output storage unit 1050 outputs the latest stored output data to the alarm output unit 2000.

[0030] The abnormality determination device 1000A executes alarm output processing (step ST500). In the alarm output processing, the alarm output unit 2000 of the abnormality determination device 1000A acquires the output data and outputs an alarm signal to an alarm device (not shown) or the like based on the output data. The alarm output unit 2000 determines whether to output an alarm based on the determination value included in the output data, and if it is determined to output an alarm, outputs an alarm signal to an alarm device (not shown) or the like.

[0031] When the abnormality determination device 1000A executes the processing of step ST500, the series of processes shown in FIG. 2 is terminated, and the process of step ST100 is repeated. The abnormality determination device 1000A, for example, turns off the power in conjunction with the camera being turned off.

[0032] A detailed example of the processing of the abnormality alarm device 100A and the abnormality determination device 1000A according to Embodiment 1 will be described. FIG. 3 is a flowchart showing an example of more detailed processing of the abnormality warning device 100A and the abnormality determination device 1000A according to Embodiment 1 of the present disclosure. When the abnormality determination device 1000A starts the processing shown in FIG. 3, first, in the same manner as step ST100 described above, it executes image acquisition processing (step ST100).

[0033] The abnormality determination device 1000A executes storage processing (step ST201). In the storage processing, the state comparison unit 1030 of the abnormality determination device 1000A stores the previous value, which is the input value of the first layer, which is at least one layer, among the processing results in all layers of the neural network unit 1020 at least for the first time after the start of processing. Also, each time processing is performed in the first layer, the state comparison unit 1030 stores the input value of the first layer.

[0034] The state comparison unit 1030 of the abnormality determination device 1000A executes previous storage determination processing for determining whether it has been stored previously (step ST202).

[0035] The state comparison unit 1030 of the abnormality determination device 1000A executes comparison processing between the current time and the previous time (step ST203). The state comparison unit 1030 calculates the difference (difference value) between the current value, which is the value about to be newly input to the first layer, and the previous value (reference value).

[0036] The state comparison unit 1030 of the abnormality determination device 1000A determines the change in the state of the determination target using the difference value (step ST204). The state comparison unit 1030 determines the change in the state of the determination target based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a threshold value stored in advance. Specifically, for example, the state comparison unit 1030 determines that there is no change in the state of the determination target when the sum of the squares of the difference values is smaller than the threshold value stored in advance for each unit of the image. Specifically, for example, when the sum of the absolute values of the difference values is smaller than a threshold value stored in advance for each unit of the image, the state comparison unit 1030 determines that the state of the determination target has not changed.

[0037] When it is determined by the state comparison unit 1030 of the abnormality determination device 1000A that the difference is equal to or greater than the threshold value and it is determined that the state of the determination target has changed (step ST204 “NO”), the processing control unit 1040 of the abnormality determination device 1000A executes normal processing command processing (step ST301). The processing control unit 1040 outputs a command to execute the processing in the first layer of the neural network unit 1020.

[0038] The neural network unit 1020 of the abnormality determination device 1000A executes output processing (step ST401). The neural network unit 1020 outputs the output data to the alarm output unit 2000.

[0039] The output storage unit 1050 of the abnormality determination device 1000A executes output storage processing (step ST402). The output storage unit 1050 stores the output data output by the neural network unit 1020.

[0040] When it is determined by the state comparison unit 1030 of the abnormality determination device 1000A that the difference is less than the threshold value and it is determined that the state of the determination target has not changed (step ST204 “YES”), the processing control unit 1040 of the abnormality determination device 1000A executes omission processing command processing (step ST302). In the omission processing command processing, the processing control unit 1040 commands the neural network unit 1020 not to execute the processing in the layer after the first layer, which is at least one of the plurality of layers, and commands the output storage unit 1050 to output the output data stored therein.

[0041] The output storage unit 1050 of the abnormality determination device 1000A executes a stored data output process (step ST403). When the output storage unit 1050 receives a command from the process control unit 1040, it outputs the stored output data to the alarm output unit 2000.

[0042] The alarm output unit 2000 executes an alarm output process (step ST500). The alarm output unit 2000 determines whether to output an alarm based on the judgment value included in the output data, similar to the processing of step ST500 already described, and if it determines to output an alarm, outputs an alarm signal to an alarm device or the like (not shown).

[0043] When abnormality determining device 1000A executes the process of step ST500, it ends the series of processes shown in FIG. 3 and repeats the process from step ST100. For example, when the camera is turned off, the abnormality determination device 1000A is also turned off in conjunction with the camera.

[0044] The configuration and processing described above enable the abnormality determination device to omit processing of a layer in the neural network depending on the conditions.

[0045] The abnormality determination device of the present disclosure has the following configuration. "An image acquisition unit that acquires and outputs an image; a neural network unit having a plurality of layers in a neural network, acquiring an image output by the image acquisition unit, and outputting a judgment result indicating a state of a judgment target included in the image; an output storage unit that stores output data (judgment value (probability value)) output by the neural network unit; a state comparison unit that prestores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer; When the state comparison unit determines that the state of the determination target has not changed, the processing control unit commands the neural network unit not to execute the processing in the layers after the first layer, and commands to output the output data stored in the output storage unit. An abnormality determination device comprising the above. Accordingly, the present disclosure can provide an abnormality determination device that can quickly output a highly accurate determination result using a neural network in the abnormality determination technology, and has the effect of being able to do so.

[0046] The abnormality determination method of the present disclosure has the following configuration. "The image acquisition unit outputs the acquired image to the neural network unit. The neural network unit having a plurality of layers in the neural network acquires the image output by the image acquisition unit and outputs a determination result indicating the state of the determination target included in the image. The output storage unit stores the output data (determination value (probability value)) output by the neural network unit. The state comparison unit stores in advance a reference value that is a reference for the input value for at least one layer, the first layer, among the plurality of layers of the neural network unit, and uses the difference value between the reference value and the current value, which is the value to be newly input to the first layer, to determine the change in the state of the determination target. When the state comparison unit determines that the state of the determination target has not changed, the processing control unit commands the neural network unit not to execute the processing in the layers after the first layer, and commands to output the output data stored in the output storage unit. An abnormal state determination method. Accordingly, the present disclosure can provide an abnormality determination method that can quickly output a highly accurate determination result using a neural network in the abnormality determination method, and has the effect of being able to do so.

[0047] The abnormality determination device of the present disclosure is further configured as follows. "The reference value is one that stores the previous value for each layer as the previous processing result in the neural network. An abnormality determination device characterized by this." As a result, the present disclosure can provide an abnormality determination device that can quickly output a highly accurate determination result using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure has the same effect as the above effect.

[0048] The abnormality determination device of the present disclosure is further configured as follows. "The reference value is the output value of a pre-modeled typical intermediate layer in the neural network. An abnormality determination device characterized by this." As a result, the present disclosure can provide an abnormality determination device that can quickly output a highly accurate determination result using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure has the same effect as the above effect.

[0049] The abnormality determination device of the present disclosure is further configured as follows. "The reference value is the output value of a predetermined node among the nodes included in the neural network. An abnormality determination device characterized by this." As a result, the present disclosure can provide an abnormality determination device that can quickly output a highly accurate determination result using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure exhibits the same effect as the above effect.

[0050] The abnormality determination device of the present disclosure is further configured as follows. "The state comparison unit determines the change in the state of the determination target based on the result of determining the magnitude of the difference between the reference value and the current value by using the difference value and a threshold value stored in advance. An abnormality determination device characterized by the above." Accordingly, the present disclosure can provide an abnormality determination device that can output a highly accurate determination result promptly using a neural network in the abnormality determination technology, and has the effect of achieving the same effect as the above effect. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure exhibits the same effect as the above effect.

[0051] The abnormality determination device of the present disclosure is further configured as follows. "The state comparison unit determines that the state of the determination target has not changed when the sum of the squares of the difference values is smaller than a threshold value stored in advance for each unit of the image. An abnormality determination device characterized by the above." Accordingly, the present disclosure can provide an abnormality determination device that can output a highly accurate determination result promptly using a neural network in the abnormality determination technology, and has the effect of achieving the same effect as the above effect. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure exhibits the same effect as the above effect.

[0052] The abnormality determination device of the present disclosure is further configured as follows. "The state comparison unit When the sum of the absolute values of the difference values is smaller than a threshold value stored in advance for the unit of the image, it is determined that the state of the determination target has not changed. An abnormality determination device characterized by the above. Accordingly, the present disclosure can provide an abnormality determination device that can quickly output a highly accurate determination result using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, the present disclosure has the same effect as the above effect by applying the above configuration to the above abnormality determination method.

[0053] The abnormality determination device of the present disclosure is further configured as follows. "The output data, which is a determination value indicating the drowsy state or dozing state of the determination target person, is acquired, and an alarm output unit for outputting an alarm to the determination target person according to the determination value is further provided. An abnormality determination device characterized by the above. Accordingly, the present disclosure can provide an abnormality determination device that can quickly output a highly accurate determination result using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, the present disclosure has the same effect as the above effect by applying the above configuration to the above abnormality determination method.

[0054] Embodiment 2. Embodiment 2 describes a form in which the basic mechanism of the present disclosure is applied to each layer in the neural networks of the feature extraction unit and the abnormal state determination unit.

[0055] An example of the configuration of the abnormality alarm device 100B and the abnormality determination device 1000B according to Embodiment 2 will be described. FIG. 4 is a diagram showing an example of the configuration of the abnormality alarm device 100B and the abnormality determination device 1000B according to Embodiment 2 of the present disclosure. The abnormality warning device 100B includes an abnormality determination device 1000B and an alarm output unit 2000.

[0056] The abnormality determination device 1000B acquires an image and uses the image to output the state of the determination target imaged in the image. The abnormality determination device 1000B shown in FIG. 4 includes an image acquisition unit 1100B, a feature extraction unit 1300B, and an abnormal state determination unit 1500B. Here, in the abnormality determination device 1000B, the neural network unit already described is configured to include a first neural network unit 1320 and a second neural network unit 1520 as will be described later. The first neural network unit 1320 is included in the feature extraction unit 1300B, and the second neural network unit 1520 is included in the abnormal state determination unit 1500B. Also, in the abnormality determination device 1000B, the output storage unit already described is configured to include a first output storage unit 1390 and a second output storage unit. The first output storage unit 1390 is included in the feature extraction unit 1300B, and the second output storage unit is included in the abnormal state determination unit 1500B. Also, in the abnormality determination device 1000B, the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540. The first state comparison unit 1340 is included in the feature extraction unit 1300B, and the second state comparison unit 1540 is included in the abnormal state determination unit 1500B. Also, in the abnormality determination device 1000B, the processing control unit already described is configured to include a first processing control unit 1380 and a second processing control unit 1580. The first processing control unit 1380 is included in the feature extraction unit 1300B, and the second processing control unit 1580 is included in the abnormal state determination unit 1500B.

[0057] Since the image acquisition unit 1100B is the same as the image acquisition unit 1100A already described, a detailed description of the image acquisition unit 1100B here is omitted.

[0058] The feature extraction unit 1300B outputs a feature map representing the characteristic state of the determination target included in the image using the image. For example, the feature extraction unit 1300B extracts parts indicating signs of dozing, such as eyelids and eyeballs, included in the image, and generates a feature map representing the state of the determination target person characteristic of an abnormal state such as dozing.

[0059] An example of the internal configuration of the feature extraction unit 1300B will be described. FIG. 5 is a diagram showing an example of the internal configuration of the feature extraction unit 1300B in the abnormal alarm device 100B and the abnormal determination device 1000B. The feature extraction unit 1300B shown in FIG. 5 includes a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340B), a processing control unit (first processing control unit 1380B), and an output storage unit (first output storage unit 1390).

[0060] The first neural network unit 1320 has a plurality of layers that are part of the layers constituting the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map representing the characteristic state of the determination target included in the image. The first neural network unit 1320 shown in FIG. 5 includes an image branching unit 1321, a convolutional layer unit 1322, a pooling layer unit 1325, and an image combining unit 1328.

[0061] The image branching unit 1321 branches and inputs the image to a plurality of nodes in the layer of the neural network. The image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 5, it is branched into two.

[0062] The convolutional layer unit 1322 performs filtering for extracting characteristic parts of the determination target in the image. The convolutional layer unit 1322 shown in FIG. 5 includes a first convolutional layer 1323 and a second convolutional layer 1324. The first convolutional layer 1323 and the second convolutional layer 1324 perform filtering for extracting facial (body) parts for detecting a drowsy state by means of convolutional processing (cross-correlation processing) using, for example, pre-prepared convolutional filters sized 3×3 or 5×5.

[0063] The pooling layer section 1325 shown in FIG. 5 includes a first pooling layer 1326 and a second pooling layer 1327. The first pooling layer 1326 and the second pooling layer 1327 generate an image regarding a feature amount robust to an image position, for example, by calculating a maximum or average value for each predetermined region.

[0064] The first neural network section 1320 shown in FIG. 5 includes two sets of a convolutional layer and a pooling layer, but one set or three or more sets are also effective. Also, a configuration may be adopted in which two or more convolutional layers and pooling layers are successively connected after the first pooling layer and the second pooling layer 1327. Also, a configuration may be adopted in which a normalization linear unit layer (activation function) or the like (not shown) is included after the convolutional layer.

[0065] The image combining section 1328 combines a plurality of images output via the convolutional layer and the pooling layer. The image combining section 1328, for example, cuts out parts indicating signs of drowsiness such as eyelids and eyeballs, and generates a feature map.

[0066] The first state comparison section 1340B stores in advance a reference value which is a reference for input values for each of a plurality of layers in the first neural network section 1320, and determines a change in the state to be determined using a difference value between the reference value and the current value which is a value to be newly input to the layer.

[0067] The reference value is the previous value of the input value for each layer, stored as the result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain the difference value between the current value, which is the input value to be newly input to the layer this time. In the description of the processing to be described later, the case where the previous value is used as the reference value will be described. In this case, the first state comparison unit 1340 only stores the value without performing the state comparison process for the first processing (processing for the first image), but determines the change in the state to be determined in the second and subsequent processes (processing for the second and subsequent images). However, the reference value may be stored and used as the following values.

[0068] As the reference value, a typical processing result that has been learned and modeled in advance can be used. That is, the reference value may be the output value of a typical intermediate layer that has been modeled in advance in the neural network. By using a typical processing result, comparison can be performed without being affected by noise or abnormalities included in the actually observed image.

[0069] Also, as the reference value, some results determined by prior information can be used. Some results determined by prior information are assumed to be, for example, adopting pixels with a high probability of the presence of the head. Also, for example, it is assumed not to adopt pixels with large changes but low importance, such as the background. That is, the reference value may be the output value of predetermined nodes among the nodes included in the neural network. Thereby, memory savings and reduction of data to be handled can be achieved, and higher speed can be achieved.

[0070] The first state comparison unit 1340 determines the change in the state to be determined based on the result of determining the magnitude of the difference between the reference value and the current value by using the difference value and a threshold value stored in advance. Specifically, when the sum of the squares of the difference values is smaller than a threshold value stored in advance for each unit of the image, the first state comparison unit 1340 determines that the state to be determined has not changed. Specifically, when the sum of the absolute values of the difference values stored for each unit of the image is smaller than a threshold value stored in advance, the first state comparison unit 1340 determines that the state to be determined has not changed.

[0071] The first state comparison unit 1340B shown in FIG. 5 includes an image state comparison unit 1341. In the image state comparison unit 1341, "storage" means holding the values of the two-dimensional images (feature maps) of the respective input sources. In the image state comparison unit 1341, "comparison" means, for example, taking the absolute value of the difference for each element (pixel) between the stored two-dimensional image (at the previous time / previous frame) and the latest two-dimensional image, and further performing a magnitude comparison between the sum or average obtained and a predetermined threshold value.

[0072] The image state comparison unit 1341 shown in FIG. 5 includes a storage unit 1341a and a comparison processing unit 1341b.

[0073] When the first state comparison unit 1340 determines that the state to be determined has not changed, the first processing control unit 1380B commands the first neural network unit 1320 not to execute the processing in the layers after the second layer of the state to be determined, and commands to output the feature map stored in the first output storage unit 1390. The first processing control unit 1380B shown in FIG. 15 includes a feature extraction processing control unit 1381B. The feature extraction processing control unit 1381B executes the functions of the first processing control unit 1380B in the feature extraction unit 1300.

[0074] The first output storage unit 1390 stores the output data (determination value (probability value)) output by the first neural network unit 1320. The first output storage unit 1390 stores the feature map output by the first neural network unit 1320. The first output storage unit 1390 shown in FIG. 5 includes a combined image storage unit 1391. The combined image storage unit 1391 stores the feature map, which is the combined image output by being combined by the first neural network unit 1320.

[0075] An example of the internal configuration of the abnormal state determination unit 1500B will be described. FIG. 6 is a diagram showing an example of the internal configuration of the abnormal state determination unit 1500B in the abnormal alarm device 100B and the abnormal determination device 1000B. The abnormal state determination unit 1500B outputs a determination value indicating the state of the determination target using the feature map, which is a two-dimensional image. The abnormal state determination unit 1500B shown in FIG. 6 includes a neural network unit (second neural network unit 1520), a state comparison unit (second state comparison unit 1540), and an output storage unit (second output storage unit) 1590.

[0076] The neural network unit (second neural network unit 1520) has a plurality of layers in the neural network, acquires the feature map output by the feature extraction unit 1300, and outputs, as output data, a determination value indicating the state of the determination target using the feature map. The neural network unit (second neural network unit 1520) shown in FIG. 6 includes a state classification unit 1525 and a probability output layer 1529.

[0077] The state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector and further summarizing the output into an output indicating a dozing state (for example, 4 outputs of eyelid: presence / absence of dozing, eyeball: presence / absence of dozing). The state classification unit 1525 shown in FIG. 6 includes a first fully connected layer 1527 and a second fully connected layer 1528. The state classification unit 1525 generates a one-dimensional vector having the desired number of outputs as the number of elements by passing through the first fully-connected layer 1527 and the second fully-connected layer 1528.

[0078] The probability output layer 1529 applies, for example, a softmax function and makes the sum of the output values equal to 1.0, thereby giving a probabilistic meaning to the output result.

[0079] The second neural network unit 1520 shown in FIG. 6 includes two fully-connected layers, but even when there are no fully-connected layers, it may be configured to include three or more fully-connected layers.

[0080] The second state comparison unit 1540B stores in advance a reference value that is a reference for the input value to at least one layer, i.e., the third layer, among the layers in the second neural network unit 1520, and uses the difference value between the reference value and the current value, which is the value about to be newly input to the third layer, to determine the change in the state of the object to be determined.

[0081] The reference value is the previous value of the input value for each layer as the previous processing result by the plurality of layers of the second neural network unit 1520, and is used to obtain the difference value between the current value, which is the input value about to be newly input to the first layer this time. In the description of the processing to be described later, the case where the previous value is used as the reference value will be described. In this case, the second state comparison unit 1540B only stores the value without performing the state comparison process for the first-time processing (processing for the first image), but determines the change in the state of the object to be determined in the processing after the second time (processing for the second image). However, the reference value may be stored and used as the following values.

[0082] As the reference value, a typical processing result that has been learned and modeled in advance can be used. That is, the reference value may be the output value of a typical intermediate layer that has been modeled in advance in the neural network. By using typical processing results, comparison can be performed without being affected by noise or abnormalities included in the actually observed image.

[0083] In addition, as the reference value, some results determined based on prior information can be used. As for some results determined based on prior information, for example, it is assumed that pixels with a high probability of the presence of the head are adopted. Also, for example, it is assumed that pixels with large changes but low importance such as the background are not adopted. That is, the reference value may be set as the output value of predetermined nodes among the nodes included in the neural network. Thereby, memory savings and reduction of data to be processed can be achieved, and higher speed can be achieved.

[0084] The second state comparison unit 1540B determines the change in the state to be determined based on the result of determining the magnitude of the difference between the reference value and the current value by using the difference value and a threshold value stored in advance. Specifically, for each unit of the image, when the sum of the squares of the difference values is smaller than the threshold value stored in advance, the second state comparison unit 1540B determines that the state to be determined has not changed. Also specifically, for each unit of the image, when the sum of the absolute values of the difference values is smaller than the threshold value stored in advance, the second state comparison unit 1540B determines that the state to be determined has not changed.

[0085] The second state comparison unit 1540B shown in FIG. 6 includes the first fully connected layer state comparison unit 1541 and is configured. In the first fully connected layer state comparison unit 1541, storing means holding the values of the one-dimensional vectors of the respective input sources. Also, in the first fully connected layer state comparison unit 1541, comparison means, for example, taking the absolute value of the difference for each element between the stored one-dimensional vector (at the previous time / previous frame) and the latest one-dimensional vector, and further taking the sum or average, and comparing the result with a predetermined threshold value.

[0086] The first fully-connected layer state comparison unit 1541 shown in FIG. 6 includes a storage unit 1541a and a comparison processing unit 1541b. The storage unit 1541a stores, for each output by the first fully-connected layer 1527, a reference value that is the output value of the first fully-connected layer 1527 and is the input value of the second fully-connected layer 1528. The comparison processing unit 1541b compares the reference value with the current value.

[0087] When the second state comparison unit 1540 determines that the state to be determined has not changed, the second processing control unit 1580B instructs the second neural network unit 1520 not to execute the processing in the layers after the layer to be determined, and also instructs to output the determination value stored in the second output storage unit as output data. The second processing control unit 1580B shown in FIG. 6 includes an abnormal state determination processing control unit 1581B. The abnormal state determination processing control unit 1581B functions to be able to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500B.

[0088] The second output storage unit 1590 stores the determination value output by the second neural network unit 1520. The output storage unit (second output storage unit) 1590 shown in FIG. 6 includes a probability storage unit 1591.

[0089] The probability storage unit stores the determination value output by the second neural network unit 1520. The determination value is a value indicating the state of the person to be determined, and for example, is a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.

[0090] Return to the description of FIG. 4. The alarm output unit 2000 acquires output data that is a determination value indicating the drowsy state or the dozing state of the person to be determined, and outputs an alarm to the person to be determined according to the determination value.

[0091] The abnormality determination device 1000B shown in FIG. 4 is shown as a configuration that does not include the alarm output unit 2000, but it may be configured to include the alarm output unit 2000. When configured in this way, the abnormality determination device 1000B is equivalent to the abnormality alarm device 100B shown in FIG. 4. In the following description, unless it is necessary to distinguish between the abnormality alarm device 100B and the abnormality determination device 1000B, the abnormality determination device 1000B will be described as being configured to include the alarm output unit 2000.

[0092] In addition to the above configuration, the abnormality determination device 1000B may include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire abnormality determination device 1000B and each component. The control unit (not shown) activates the abnormality determination device 1000A according to a command from the outside, for example. Also, the control unit (not shown) controls the state of the abnormality determination device 1000B (operating states such as startup, shutdown, sleep). The storage unit (not shown) stores each data used in the abnormality determination device 1000B. The storage unit (not shown) stores, for example, the output (output data) by each component in the abnormality determination device 1000B, and outputs the data required for each component to the component that requested it. The communication unit (not shown) communicates with an external device. For example, communication is performed between the abnormality determination device 1000B and an imaging device such as an in-vehicle camera. Also, for example, when the abnormality determination device 1000B does not include a display unit or an audio output unit, communication is performed between the abnormality determination device 1000B and an external device such as a display device or an audio output device.

[0093] An example of the processing of the abnormality alarm device 100B and the abnormality determination device 1000B according to the second embodiment will be described. FIG. 7 is a flowchart showing an example of the processing of the abnormality alarm device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure. When an image is input from, for example, a camera, the abnormality determination device 1000B starts the process shown in FIG. 7.

[0094] The abnormality determination device 1000B executes image acquisition processing (step ST2100). In the image acquisition processing, the image acquisition unit 1100 of the abnormality determination device 1000B acquires and outputs an image.

[0095] The abnormality determination device 1000B executes storage and state comparison processing (step ST2200). In the state storage and state comparison processing, the first state comparison unit 1340B of the abnormality determination device 1000B stores the previous value (reference value), which is the input value for the layer, for each layer of the first neural network unit 1320 at least the first time after the start of processing. The first state comparison unit 1340 only stores the value (previous value = reference value) without performing the state comparison processing for the first processing (processing for the first image), but determines the change in the state to be determined in the second and subsequent processes (processing for the second image).

[0096] The abnormality determination device 1000B executes feature extraction processing control processing (step ST2300). The feature extraction processing control unit 1381 of the abnormality determination device 1000B executes a normal processing command or an omission processing command for the first neural network unit 1320 according to the determination result by the first state comparison unit 1340. In addition, when the feature extraction processing control unit 1381 issues an omission processing command, it issues an output command to the combined image storage unit 1391.

[0097] The abnormality determination device 1000B executes combined image output processing (step ST2400). The first neural network unit 1320 executes normal processing and outputs a combined image, or the combined image storage unit 1391 outputs the previous combined image. Thereby, the feature extraction unit 1300 outputs the combined image.

[0098] The abnormality determination device 1000B executes the fully-connected layer state storage and fully-connected layer state comparison process (step ST2500). In the second state comparison unit 1540 of the abnormality determination device 1000B, for each of the plurality of layers of the second neural network unit 1520, a reference value that is a reference for the input value for that layer is stored in advance, and the difference value between the reference value and the current value, which is the value to be newly input to that layer, is used to determine the change in the state to be determined. The second state comparison unit 1540 only stores values without performing state comparison processing for the first-time processing (processing for the first image), but determines the change in the state to be determined in the second and subsequent processing (processing for the second image).

[0099] The abnormality determination device 1000B executes the abnormal state determination process control process (step ST2600). The abnormal state determination process control unit 1581B issues a normal command or an omission command to the second neural network unit 1520. When the abnormal state determination process control unit 1581B issues an omission command, it issues an output command to the probability storage unit 1591.

[0100] The abnormality determination device 1000B executes the result output process (step ST2700). The second neural network unit 1520 executes normal processing and outputs a determination value, or the probability storage unit 1591 outputs the previous determination value (probability value). As a result, the abnormal state determination unit 1500 outputs the determination value as output data.

[0101] The abnormality determination device 1000B executes the alarm output process (step ST2800). In the alarm output process, the alarm output unit 2000 of the abnormality determination device 1000B acquires the output data and outputs an alarm signal to an alarm device (not shown) or the like based on the output data. The alarm output unit 2000 determines whether to output an alarm based on the determination value included in the output data, and if it determines to output an alarm, it outputs an alarm signal to an alarm device (not shown) or the like.

[0102] When the abnormality determination device 1000B executes the process of step ST3800, it ends the series of processes shown in FIG. 7 and repeats from the process of step ST3100. Note that the abnormality determination device 1000B is turned off in conjunction with, for example, when the camera is turned off.

[0103] Here, a detailed example of the processes of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment will be described. FIG. 8 is a flowchart showing an example of a more detailed process of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure. The flowchart of FIG. 8 shows an example of a process corresponding to the process from step ST2200 to step ST2700 in the flowchart of FIG. 7. When the feature extraction unit 1300B in the abnormality determination device 1000B starts the process of step ST2200, first, it executes a storage process (step ST2201). In the storage process, the state comparison unit 1340 of the feature extraction unit 1300B stores the previous value, which is the input value of the first layer, which is at least one layer among all the layers of the neural network unit 1320 (the first neural network unit 1320). In addition, the state comparison unit 1340 stores the input value of the first layer each time the process is performed in the first layer.

[0104] Next, the feature extraction unit 1300B in the abnormality determination device 1000B executes a previous storage determination process (step ST2202). In the previous storage determination process, the state comparison unit 1340 of the feature extraction unit 1300B determines whether the previous input value (previous value = reference value) is stored.

[0105] The feature extraction unit 1300B in the abnormality determination device 1000B executes a comparison process (step ST2203). In the comparison process, the state comparison unit 1340 of the feature extraction unit 1300B calculates the difference (difference value) between the current value, which is the value about to be newly input to the first layer, and the previous value (reference value).

[0106] The feature extraction unit 1300B in the abnormality determination device 1000B executes a difference determination process (step ST2204). In the difference determination process, the state comparison unit 1340 of the feature extraction unit 1300B determines the change in the state of the determination target using the difference value. Specifically, for example, when the sum of the squares of the difference values is smaller than a threshold value stored in advance for each unit of the image, the state comparison unit 1340 determines that the state of the determination target has not changed. Also specifically, for example, when the sum of the absolute values of the difference values is smaller than a threshold value stored in advance for each unit of the image, the state comparison unit 1340 determines that the state of the determination target has not changed.

[0107] When it is determined by the state comparison unit 1340 of the feature extraction unit 1300B that it was not stored previously (step ST2202 “NO”), or when it is determined by the state comparison unit 1340 that the difference is equal to or greater than the threshold value and it is determined that the state of the determination target has changed (step ST2204 “NO”), the feature extraction unit 1300B in the abnormality determination device 1000B executes a normal processing command (step ST2301). In the normal processing command, the processing control unit 1380 of the feature extraction unit 1300B outputs a command to execute the processing in the first layer of the neural network unit 1320 (the first neural network unit 1320).

[0108] Next, the feature extraction unit 1300B in the abnormality determination device 1000B executes an output process (step ST2401). In the output process, the first neural network unit 1320 of the feature extraction unit 1300B executes the process according to the normal processing command and outputs output data indicating the determination result.

[0109] Next, the feature extraction unit 1300B in the abnormality determination device 1000B executes an output storage process (step ST2402). In the output storage process, the output storage unit 1390 (the first output storage unit 1390) of the feature extraction unit 1300B stores the output data output by the neural network unit 1320.

[0110] When the state comparison unit 1340 determines that the difference is less than the threshold value and determines that the state to be determined has not changed (step ST2204 “YES”), the feature extraction unit 1300B in the abnormality determination device 1000B executes an omission processing command (step ST2302). In the omission processing command, the processing control unit 1380 of the feature extraction unit 1300B commands the neural network unit 1320 not to execute the processing in the layer after the first layer, which is at least one of the plurality of layers, and commands the output storage unit 1390 to output the output data stored therein.

[0111] The feature extraction unit 1300B in the abnormality determination device 1000B executes stored data output processing (step ST2403). In the stored data output processing, when the output storage unit 1390 (the combined image storage unit 1391 of the output storage unit 1390) of the feature extraction unit 1300B receives a command from the processing control unit 1380, it outputs the combined image, which is the stored output data, to the alarm output unit 2000.

[0112] The abnormal state determination unit 1500B in the abnormality determination device 1000B executes fully connected layer output storage processing (step ST2501). In the fully connected layer output storage processing, the state comparison unit 1540 of the abnormal state determination unit 1500B stores the value output from the fully connected layer (the first fully connected layer 1527 or the second fully connected layer 1528) in the state classification unit 1525 as the state of the fully connected layer.

[0113] The abnormal state determination unit 1500B in the abnormality determination device 1000B executes previous storage determination processing (step ST2502). In the previous storage determination processing, the state comparison unit 1540 of the abnormal state determination unit 1500B determines whether the previous input value (previous value = reference value) is stored.

[0114] When the abnormality state determination unit 1500B in the abnormality determination device 1000B determines that it was stored previously (step ST2502 “YES”), it executes a comparison process (step ST2503). In the comparison process, the state comparison unit 1540 of the abnormality state determination unit 1500B calculates the difference (difference value) between the current value, which is the value about to be newly input to the first layer (the second fully connected layer 1528 or the probability output layer 1529), and the previous value (reference value).

[0115] The abnormality state determination unit 1500B in the abnormality determination device 1000B executes a difference determination process (step ST2504). In the difference determination process, the state comparison unit 1540 of the abnormality state determination unit 1500B determines the magnitude of the difference between the reference value and the current value using the difference value between the current value and the previous value (reference value) and a threshold value stored in advance.

[0116] When it is determined by the state comparison unit 1540 of the abnormality state determination unit 1500B that the difference is equal to or greater than the threshold value (step ST2504 “NO”), the abnormality state determination unit 1500B in the abnormality determination device 1000B executes a normal process command (step ST2601). In the normal process command, the process control unit 1580 of the abnormality state determination unit 1500B executes a normal process command on the neural network unit 1520 (the second neural network unit 1520).

[0117] The abnormality state determination unit 1500B in the abnormality determination device 1000B executes an output process (step ST2701). In the output process, the neural network unit 1520 (the second neural network unit 1520) of the abnormality state determination unit 1500B outputs the determination value as output data.

[0118] The abnormality state determination unit 1500B in the abnormality determination device 1000B executes an output storage process (step ST2702). In the output storage process, the output storage unit 1590 (the second output storage unit 1590) in the abnormality state determination unit 1500B stores the determination value output by the neural network unit 1520 (the second neural network unit 1520).

[0119] The abnormality state determination unit 1500B in the abnormality determination device 1000B executes an omission process command (step ST2602). In the omission process command, the process control unit 1580 of the abnormality state determination unit 1500B commands the second neural network unit 1520 not to execute the process in the layers after the layer to be determined, and also commands to output the determination value stored in the second output storage unit.

[0120] The abnormality state determination unit 1500B in the abnormality determination device 1000B executes stored data output processing (step ST2703). In the stored data output processing, the second output storage unit 1590 of the abnormality state determination unit 1500B stores the determination value output by the neural network unit 1520 (the second neural network unit 1520).

[0121] When the second neural network unit 1520 or the second output storage unit 1590 outputs a determination value, the series of processes shown in FIG. 21 is terminated.

[0122] In the figure, an example of the configuration is shown in which the first state comparison unit 1340B has the image state comparison unit 1341 and the second state comparison unit 1540B has the first fully connected layer state comparison unit 1541. However, the image state comparison unit 1341, a convolutional layer state comparison unit for determining the state change of the convolutional layer (refer to the image state comparison unit 1342 in the embodiment described later), a pooling layer state comparison unit for determining the state change of the pooling layer (refer to the pooling layer state comparison unit 1343 in the embodiment described later), the first fully connected layer state comparison unit (refer to the first fully connected layer state comparison unit 1541 in the embodiment described later), and a second fully connected layer state comparison unit for determining the state change of the second fully connected layer 1528 (refer to the second fully connected layer state comparison unit 1542 in the embodiment described later) may be configured by any one or a combination of a plurality of them. For example, when any one is adopted, it can be realized by the processes shown in FIGS. 9, 10, 11, 12, and 13 below.

[0123] An example of the processing of the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) is the image state comparison unit 1341 will be described. FIG. 9 is a flowchart showing an example of the processing of the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is the image state comparison unit 1341.

[0124] When the feature extraction unit 1300B starts processing, first, the first state comparison unit 1340B of the feature extraction unit 1300B acquires an image and executes an image storage process (step ST2211). In the image storage process, the image state comparison unit 1341 of the first state comparison unit 1340B stores the image, which is the input data for the first neural network unit 1320. Also, each time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image (the image before being processed by the first neural network unit 1320) in the storage unit 1341a.

[0125] The image state comparison unit 1341 executes a process of determining whether there was a previous storage (step ST2212). The image state comparison unit 1341 refers to the storage unit 1341a and determines whether the previously input image is stored.

[0126] When the image state comparison unit 1341 determines that there was a previous storage (step ST2212 “YES”), it executes a comparison process between the current and previous images (step ST2213). The comparison processing unit 1341b of the image state comparison unit 1341 compares the currently input image with the previously input image.

[0127] The comparison processing unit 1341b executes a process of determining whether the difference is less than the threshold value (step ST2214). The comparison processing unit 1341b determines the magnitude of the difference between the reference value (the image input last time) and the current value (the image to be input this time) using the difference value between the image input this time and the image input last time and a threshold value stored in advance.

[0128] When it is determined by the image state comparison unit 1341 that there was no previous storage (step ST2212 “NO”), or when it is determined by the image state comparison unit 1341 that the difference between the previous value (reference value) and the current value is equal to or greater than the threshold value (step ST2214 “NO”), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2311). The first neural network unit 1320 executes normal processing according to the normal processing command and outputs a combined image (step ST2411). The combined image storage unit 1391, which is the first output storage unit 1390, stores the combined image output by the first neural network unit 1320 (step ST2412).

[0129] When it is determined by the image state comparison unit 1341 that the difference between the previous value (reference value) and the current value is less than the threshold value (step ST2214 “YES”), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues an omission processing command (step ST2312) to the first neural network unit 1320 and issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2413). The first neural network unit 1320 does not execute the processing of the subsequent layers in the first neural network unit 1320 according to the omission processing command. The combined image storage unit 1391 outputs the combined image according to the output command.

[0130] Next, an example of the processing of the feature extraction unit 1300B when the state comparison unit (the first state comparison unit 1340B) is the convolutional layer state comparison unit (refer to the convolutional layer state comparison unit 1342 in the embodiment described later) will be described. FIG. 10 is a flowchart showing an example of the processing of the feature extraction unit 1300B in Embodiment 2 of the present disclosure, where the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B is a convolutional layer state comparison unit.

[0131] When the feature extraction unit 1300B starts processing, first, the first state comparison unit 1340B of the feature extraction unit 1300B executes a convolutional layer state storage process (step ST2221). In the convolutional layer state storage process, the convolutional layer state comparison unit of the first state comparison unit 1340B stores the output value of the convolutional layer unit 1322, which is the input value (the value before being processed by the pooling layer unit 1325) for the pooling layer unit 1325. Also, each time the convolutional layer unit 1322 outputs an output value, the convolutional layer state comparison unit stores the output value (the value before being processed by the pooling layer unit 1325) in the storage unit.

[0132] The convolutional layer state comparison unit executes a process of determining whether there was a previous storage (step ST2222). The convolutional layer state comparison unit refers to the storage unit and determines whether the previously input value (reference value: the previous value in this description) is stored.

[0133] When the convolutional layer state comparison unit determines that there was a previous storage (step ST2222 “YES”), it executes a comparison process between the current time and the previous time (step ST2223). The comparison processing unit of the convolutional layer state comparison unit compares the value that is about to be input to the pooling layer this time (current value) with the value input previously (previous value = reference value).

[0134] The comparison processing unit executes a process of determining whether the difference is less than the threshold value (step ST2224). The comparison processing unit determines the magnitude of the difference between the reference value (the value input previously) and the current value (the value about to be input this time) by using the difference value between the value about to be input this time (the current value) and the value input previously (the previous value), and a threshold value stored in advance.

[0135] When it is determined by the convolutional layer state comparison unit that there was no previous storage (step ST2222 “NO”), or when it is determined by the convolutional layer state comparison unit that the difference between the previous value (reference value) and the current value is greater than or equal to the threshold value (step ST2224 “NO”), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321). The first neural network unit 1320 executes normal processing according to the normal processing command and outputs a combined image (step ST2421). The combined image storage unit 1391, which is the first output storage unit 1390, stores the combined image output by the first neural network unit 1320 (step ST2422).

[0136] When it is determined by the convolutional layer state comparison unit that the difference between the previous value (reference value) and the current value is less than the threshold value (step ST2224 “YES”), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues an omission processing command (step ST2322) to the first neural network unit 1320 and issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423). The first neural network unit 1320 does not execute the processing of the subsequent layers in the first neural network unit 1320 according to the omission processing command. The combined image storage unit 1391 outputs the combined image according to the output command.

[0137] Next, an example of the processing of the feature extraction unit 1300B when the state comparison unit (the first state comparison unit 1340B) is the pooling layer state comparison unit (refer to the pooling layer state comparison unit 1343 in the embodiment described later) will be described. FIG. 11 is a flowchart showing an example of the processing of the feature extraction unit 1300B in Embodiment 2 of the present disclosure when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B is a pooling layer state comparison unit.

[0138] When the feature extraction unit 1300B starts processing, first, the first state comparison unit 1340B of the feature extraction unit 1300B executes pooling layer state storage processing (step ST2221). In the pooling layer state storage processing, the pooling layer state comparison unit of the first state comparison unit 1340B stores the output value of the pooling layer unit 1325, which is the input value (the value before being processed by the image combining unit 1328) for the image combining unit 1328. Also, each time the pooling layer unit 1325 outputs an output value, the pooling layer state comparison unit stores the output value (the value before being processed by the image combining unit 1328) in the storage unit.

[0139] The pooling layer state comparison unit executes a process of determining whether there was a previous storage (step ST2222). The pooling layer state comparison unit refers to the storage unit and determines whether the previously input value (reference value: the previous value in this description) is stored.

[0140] When the pooling layer state comparison unit determines that there was a previous storage (step ST2222 “YES”), it executes a comparison process between the current time and the previous time (step ST2223). The comparison processing unit of the pooling layer state comparison unit compares the value that is about to be input to the image combining unit 1328 this time (current value) with the value input previously (previous value).

[0141] The comparison processing unit executes a process of determining whether the difference is less than the threshold value (step ST2224). The comparison processing unit determines the magnitude of the difference between the reference value (the value input previously) and the current value (the value about to be input this time) by using the difference value between the value about to be input this time (the current value) and the value input previously (the previous value), and a threshold value stored in advance.

[0142] When it is determined by the pooling layer state comparison unit that there was no previous storage (step ST2222 “NO”), or when it is determined by the pooling layer state comparison unit that the difference between the previous value (reference value) and the current value is greater than or equal to the threshold value (step ST2224 “NO”), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321). The first neural network unit 1320 executes normal processing according to the normal processing command and outputs a combined image (step ST2421). The combined image storage unit 1391, which is the first output storage unit 1390, stores the combined image output by the first neural network unit 1320 (step ST2422).

[0143] When it is determined by the pooling layer state comparison unit that the difference between the previous value (reference value) and the current value is less than the threshold value (step ST2224 “YES”), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues an omission processing command (step ST2322) to the first neural network unit 1320 and issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423). The first neural network unit 1320 does not execute the processing of the subsequent layers in the first neural network unit 1320 according to the omission processing command. The combined image storage unit 1391 outputs the combined image according to the output command.

[0144] Next, an example of the processing of the abnormality determination unit 1500B when the state comparison unit (the second state comparison unit 1540B) is the first layer state comparison unit 1541 will be described. FIG. 12 is a flowchart showing an example of the processing of the abnormality determination unit 1500B in Embodiment 2 of the present disclosure when the state comparison unit (second state comparison unit 1540B) in the abnormality determination unit 1500B is the first layer state comparison unit 1541.

[0145] When the abnormality determination unit 1500B executes the process (step ST2511) on the data output by the feature extraction unit 1300B, the second state comparison unit 1540B first acquires the output of the first fully connected layer (step ST2512). The first fully connected layer state storage process is executed (step ST2513). The first layer state comparison unit 1541 in the second state comparison unit 1540B stores the output value from the first fully connected layer 1527, which is the input value to the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528).

[0146] The first layer state comparison unit 1541 executes a process of determining whether there was a previous storage (step ST2514). The first layer state comparison unit 1541 refers to the storage unit 1541a and determines whether a reference value (previous value) is stored.

[0147] When the first layer state comparison unit 1541 determines that there was a previous storage (step ST2514 “YES”), it executes a comparison process between the current time and the previous time (step ST2515). The first fully connected layer state comparison unit 1541 compares the current value with the reference value (previous value).

[0148] The first layer state comparison unit 1541 executes a process of determining whether the difference is smaller than the threshold value (step ST2516). The comparison processing unit 1541b determines the magnitude of the difference between the reference value and the current value using the difference value between the current value and the reference value (previous value) and the threshold value stored in advance.

[0149] When the comparison processing unit 1541b determines that there was no previous storage (step ST2514 “NO”), or when the first layer state comparison unit 1541 determines that the difference is equal to or greater than the threshold value (step ST2516 “NO”), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2611). The second neural network unit 1520 executes normal processing in accordance with the normal processing command and outputs a determination value (step ST2711). The probability storage unit 1591, which is the second output storage unit 1590, stores the determination value output by the second neural network unit 1520 (step ST2712).

[0150] When the first layer state comparison unit 1541 determines that the difference is less than the threshold value (step ST2516 “YES”), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, executes an omission processing command (step ST2612). In the omission processing command, the abnormal state determination processing control unit 1581B commands the second neural network unit 1520 not to execute the processing in the layers after the layer to be determined, and also commands the second output storage unit 1590 to output the determination value stored therein. The second neural network unit 1520 does not execute the processing of the subsequent layers in the second neural network unit 1520 in accordance with the omission processing command. The probability storage unit 1591, which is the second output storage unit 1590, outputs the determination value in accordance with the output command (step ST2713).

[0151] Next, an example of the processing of the abnormal state determination unit 1500B when the state comparison unit (the second state comparison unit 1540B) is the second fully-connected layer state comparison unit (refer to the second fully-connected layer state comparison unit 1542 in the following embodiment) will be described. FIG. 13 is a flowchart showing an example of the processing of the abnormality determination unit 1500B in Embodiment 2 of the present disclosure when the state comparison unit (second state comparison unit 1540B) in the abnormality determination unit 1500B is the second layer state comparison unit.

[0152] When the abnormality determination unit 1500B executes the process for the data output by the feature extraction unit 1300B (step ST2521), next, the second fully connected layer 1528 executes the process for the output by the first fully connected layer (step ST2522). Next, the second state comparison unit 1540B acquires the output of the second fully connected layer (step ST2523). Next, the second state comparison unit 1540B executes the second fully connected layer state storage process (step ST2524). The second layer state comparison unit in the second state comparison unit 1540B stores the output value from the second fully connected layer 1528, which is the input value of the probability output layer 1529 (the value before being processed by the probability output layer 1529).

[0153] The second layer state comparison unit executes the process of determining whether there was a previous storage (step ST2525). The second layer state comparison unit refers to the storage unit and determines whether the reference value (previous value) is stored.

[0154] When the second layer state comparison unit determines that there was a previous storage (step ST2525 “YES”), it executes the comparison process between the current time and the previous time (step ST2526). The second layer state comparison unit compares the current value with the reference value (previous value).

[0155] The second layer state comparison unit executes the process of determining whether the difference is smaller than the threshold value (step ST2527). The comparison processing unit determines the magnitude of the difference between the reference value and the current value using the difference value between the current value and the reference value (previous value) and the threshold value stored in advance.

[0156] When the comparison processing unit determines that there was no previous storage (step ST2525 “NO”), or when the second layer state comparison unit determines that the difference is equal to or greater than the threshold value (step ST2527 “NO”), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2621). The second neural network unit 1520 executes normal processing in accordance with the normal processing command and outputs a determination value (step ST2721). The probability storage unit 1591, which is the second output storage unit 1590, stores the determination value output by the second neural network unit 1520 (step ST2722).

[0157] When the first fully connected layer state comparison unit 1541 determines that the difference is less than the threshold value (step ST2527 “YES”), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, executes an omission processing command (step ST2622). In the omission processing command, the abnormal state determination processing control unit 1581B commands the second neural network unit 1520 not to execute the processing in the layers after the layer to be determined, and also commands the second output storage unit 1590 to output the determination value stored therein. The second neural network unit 1520 does not execute the processing of the subsequent layers in the second neural network unit 1520 in accordance with the omission processing command. The probability storage unit 1591, which is the second output storage unit 1590, outputs the determination value in accordance with the output command (step ST2723).

[0158] The abnormality determination device of the present disclosure is further configured as follows. “The neural network unit includes a first neural network unit and a second neural network unit, The output storage unit includes a first output storage unit and a second output storage unit, The state comparison unit includes a first state comparison unit and a second state comparison unit, The processing control unit includes a first processing control unit and a second processing control unit, A first neural network unit having a plurality of layers that are part of the layers constituting the neural network, acquiring the image output by the image acquisition unit, and outputting a feature map representing the characteristic state of the determination target included in the image, A first output storage unit that stores the feature map output by the first neural network unit, A reference value that is a reference for the input value to at least one layer, which is a second layer, among the layers in the first neural network unit is stored in advance, and the difference value between the reference value and the current value, which is the value to be newly input to the second layer, is used to determine the change in the state of the determination target. The first state comparison unit, And, When the first state comparison unit determines that the state of the determination target has not changed, the first neural network unit is instructed not to execute the processing in the layers after the second layer of the determination target, and the first output storage unit is instructed to output the feature map stored therein. The first processing control unit, A feature extraction unit configured to include, A second neural network unit having a plurality of layers that are part of the layers constituting the neural network, acquiring the feature map output by the feature extraction unit, and outputting a determination value indicating the state of the determination target using the feature map as the output data, A second output storage unit that stores the determination value output by the second neural network unit, A reference value that is a reference for the input value to at least one layer, which is a third layer, among the layers in the second neural network unit is stored in advance, and the difference value between the reference value and the current value, which is the value to be newly input to the third layer, is used to determine the change in the state of the determination target. The second state comparison unit, And, When it is determined by the second state comparison unit that the state of the determination target has not changed, the second neural network unit is instructed not to execute the processing in the layers after the third layer of the determination target, and the second output storage unit is instructed to output the determination value stored therein as the output data. The second processing control unit An abnormal state determination unit configured to include An abnormality determination device comprising. Thereby, the present disclosure can provide an abnormality determination device that can output a more accurate determination result more quickly using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, the present disclosure has the same effect as the above effect by applying the above configuration to the above abnormality determination method.

[0159] The abnormality determination device of the present disclosure is further configured as follows. "The first state comparison unit and the second state comparison unit Based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a threshold value stored in advance, determine the change in the state of the determination target. An abnormality determination device characterized by the above. Thereby, the present disclosure can provide an abnormality determination device that can output a more accurate determination result more quickly using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, the present disclosure has the same effect as the above effect by applying the above configuration to the above abnormality determination method.

[0160] The abnormality determination device of the present disclosure is further configured as follows. "The first state comparison unit and the second state comparison unit When the sum of the squares of the difference values is smaller than a threshold value stored in advance for each unit of the image, it is determined that the state of the determination target has not changed. An abnormality determination device characterized by the above. Accordingly, the present disclosure can provide an abnormality determination device that can output a more accurate determination result more quickly using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure has the same effect as the above effect.

[0161] The abnormality determination device of the present disclosure is further configured as follows. The first state comparison unit and the second state comparison unit When the sum of the absolute values of the difference values is smaller than a threshold value stored in advance for each unit of the image, it is determined that the state of the determination target has not changed. An abnormality determination device characterized by the above. Accordingly, the present disclosure can provide an abnormality determination device that can output a more accurate determination result more quickly using a neural network in the abnormality determination technology, and has the effect of being able to do so. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure has the same effect as the above effect.

[0162] Embodiment 3. Embodiment 3 describes a form in which the basic mechanism of the present disclosure is applied to all layers in a neural network. In Embodiment 3, the configurations and processes already described will be omitted as appropriate.

[0163] An example of the configuration of the abnormality warning device 100C and the abnormality determination device 1000C according to Embodiment 3 will be described. FIG. 14 is a diagram showing an example of the configuration of the abnormality warning device 100C and the abnormality determination device 1000C according to Embodiment 3 of the present disclosure. The abnormality warning device 100C includes an abnormality determination device 1000C and an alarm output unit 2000.

[0164] The abnormality determination device 1000C acquires an image and uses the image to output the state of the determination target imaged in the image. The abnormality determination device 1000C shown in FIG. 14 includes an image acquisition unit 1100C, a feature extraction unit 1300C, and an abnormal state determination unit 1500C. Here, in the abnormality determination device 1000C, the neural network unit already described is configured to include a first neural network unit 1320 and a second neural network unit 1520 as will be described later. The first neural network unit 1320 is included in the feature extraction unit 1300C, and the second neural network unit 1520 is included in the abnormal state determination unit 1500C. Also, in the abnormality determination device 1000C, the output storage unit already described is configured to include a first output storage unit 1390 and a second output storage unit. The first output storage unit 1390 is included in the feature extraction unit 1300C, and the second output storage unit is included in the abnormal state determination unit 1500C. Also, in the abnormality determination device 1000C, the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540. The first state comparison unit 1340 is included in the feature extraction unit 1300C, and the second state comparison unit 1540 is included in the abnormal state determination unit 1500C. Also, in the abnormality determination device 1000C, the processing control unit already described is configured to include a first processing control unit 1380 and a second processing control unit 1580. The first processing control unit 1380 is included in the feature extraction unit 1300C, and the second processing control unit 1580 is included in the abnormal state determination unit 1500C.

[0165] Since the image acquisition unit 1100C is the same as the image acquisition units 1100A and 1100B already described, a detailed description of the image acquisition unit 1100C here is omitted.

[0166] The feature extraction unit 1300C outputs a feature map representing the characteristic state of the determination target included in the image using the image. For example, the feature extraction unit 1300C extracts parts indicating signs of dozing, such as eyelids and eyeballs, included in the image, and generates a feature map representing the state of the determination target person characteristic of an abnormal state such as dozing.

[0167] An example of the internal configuration of the feature extraction unit 1300C will be described. FIG. 15 is a diagram showing an example of the internal configuration of the feature extraction unit 1300C in the abnormal alarm device 100C and the abnormal determination device 1000C. The feature extraction unit 1300C shown in FIG. 15 includes a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340C), a processing control unit (first processing control unit 1380C), and an output storage unit (first output storage unit 1390).

[0168] The first neural network unit 1320 has a plurality of layers that are part of the layers constituting the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map representing the characteristic state of the determination target included in the image. The first neural network unit 1320 shown in FIG. 15 includes an image branching unit 1321, a convolutional layer unit 1322, a pooling layer unit 1325, and an image combining unit 1328.

[0169] The image branching unit 1321 branches and inputs the image to a plurality of nodes in the layer of the neural network. The image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 15, it is branched into two.

[0170] The convolutional layer unit 1322 performs filtering for extracting characteristic parts of the determination target in the image. The convolutional layer unit 1322 shown in FIG. 15 includes a first convolutional layer 1323 and a second convolutional layer 1324. The first convolutional layer 1323 and the second convolutional layer 1324 perform filtering for extracting face (body) parts for detecting a drowsy state by means of convolutional processing (cross-correlation processing) using, for example, pre-prepared convolutional filters sized 3×3 or 5×5.

[0171] The pooling layer section 1325 shown in FIG. 15 includes a first pooling layer 1326 and a second pooling layer 1327. The first pooling layer 1326 and the second pooling layer 1327 generate an image regarding a robust feature amount with respect to an image position, for example, by calculating a maximum or average value for each predetermined region.

[0172] The first neural network section 1320 shown in FIG. 15 includes two sets of a convolutional layer and a pooling layer, but one set or three or more sets are also effective. Further, a configuration in which two or more convolutional layers and pooling layers are successively connected after the first pooling layer and the second pooling layer 1327 may be adopted. Further, a configuration may be adopted such that a normalization linear unit layer (activation function) or the like (not shown) is included after the convolutional layer.

[0173] The image combining section 1328 combines a plurality of images output via the convolutional layer and the pooling layer. The image combining section 1328, for example, cuts out parts indicating signs of drowsiness such as eyelids and eyeballs to generate a feature map.

[0174] The first state comparison section 1340C stores in advance a reference value which is a reference for input values for each of a plurality of layers in the first neural network section 1320, and uses a difference value between the reference value and the current value which is a value to be newly input to the layer to determine a change in the state to be determined.

[0175] The reference value is the previous value of the input value for each layer, which is stored as the result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain the difference value from the current value, which is the input value to be newly input to the layer this time. In the description of the processing to be described later, the case where the previous value is used as the reference value will be described. In this case, the first state comparison unit 1340 only stores values without performing state comparison processing for the first processing (processing for the first image), but determines the change in the state to be determined in the second and subsequent processes (processing for the second and subsequent images). However, the reference value may be stored and used as the following values.

[0176] As the reference value, a typical processing result that has been learned and modeled in advance can be used. That is, the reference value may be the output value of a typical intermediate layer that has been modeled in advance in the neural network. By using a typical processing result, comparison can be performed without being affected by noise or anomalies included in the actually observed image.

[0177] Also, as the reference value, some results determined by prior information can be used. Some results determined by prior information are assumed to include, for example, adopting pixels with a high probability of the presence of a head. Also, for example, it is assumed that pixels with large changes but low importance, such as the background, are not adopted. That is, the reference value may be the output value of predetermined nodes among the nodes included in the neural network. Thereby, memory saving and reduction of data to be handled can be achieved, and higher speed can be achieved.

[0178] The first state comparison unit 1340 determines the change in the state to be determined based on the result of determining the magnitude of the difference between the reference value and the current value by using the difference value and a threshold value stored in advance. Specifically, the first state comparison unit 1340 determines that the state to be determined has not changed when the sum of the squares of the difference values is smaller than a threshold value stored in advance for each unit of the image. Specifically, the first state comparison unit 1340 determines that the state to be determined has not changed when the sum of the absolute values of the difference values stored for each unit of the image is smaller than a threshold value stored in advance.

[0179] The first state comparison unit 1340C shown in FIG. 15 includes an image state comparison unit 1341, a convolutional layer state comparison unit 1342, and a pooling layer state comparison unit 1343. In the image state comparison unit 1341, the convolutional layer state comparison unit 1342, and the pooling layer state comparison unit 1343, "storage" means holding the values of the two-dimensional images (feature maps) of their respective input sources. In the image state comparison unit 1341, the convolutional layer state comparison unit 1342, and the pooling layer state comparison unit 1343, "comparison" means, for example, taking the absolute value of the difference for each element (pixel) between the stored two-dimensional image (from the previous time / previous frame) and the latest two-dimensional image, and further performing a comparison of the sum or average with a predetermined threshold value.

[0180] The image state comparison unit 1341 shown in FIG. 15 includes a storage unit 1341a and a comparison processing unit 1341b.

[0181] The convolutional layer state comparison unit 1342 shown in FIG. 15 includes a storage unit 1342a and a comparison processing unit 1342b.

[0182] The pooling layer state comparison unit 1343 shown in FIG. 15 includes a storage unit 1343a and a comparison processing unit 1343b.

[0183] When the first processing control unit 1380C determines that the state to be determined has not changed by the first state comparison unit 1340, it commands the first neural network unit 1320 not to execute the processing in the layers after the layer to be determined, and also commands to output the feature map stored in the first output storage unit 1390. The first processing control unit 1380C shown in FIG. 15 includes a feature extraction processing control unit 1381C. The feature extraction processing control unit 1381C executes the functions of the first processing control unit 1380C in the feature extraction unit 1300.

[0184] The first output storage unit 1390 stores the output data (determination value (probability value)) output by the first neural network unit 1320. The first output storage unit 1390 stores the feature map output by the first neural network unit 1320. The first output storage unit 1390 shown in FIG. 15 includes a combined image storage unit 1391. The combined image storage unit 1391 stores the feature map, which is the combined image combined and output by the first neural network unit 1320.

[0185] Return to the description of FIG. 14. The abnormal state determination unit 1500C outputs a determination value indicating the state to be determined using the feature map that is a two-dimensional image.

[0186] An example of the internal configuration of the abnormal state determination unit 1500C will be described. FIG. 16 is a diagram showing an example of the internal configuration of the abnormal state determination unit 1500C in the abnormal alarm device 100C and the abnormal determination device 1000C. The abnormal state determination unit 1500C shown in FIG. 16 includes a neural network unit (the second neural network unit 1520), a state comparison unit (the second state comparison unit 1540), and an output storage unit (the second output storage unit) 1590.

[0187] The neural network unit (second neural network unit 1520) has a plurality of layers in the neural network, obtains the feature map output by the feature extraction unit 1300, and uses the feature map to output a determination value indicating the state to be determined as output data. The neural network unit (second neural network unit 1520) shown in FIG. 16 includes a state classification unit 1525 and a probability output layer 1529.

[0188] The state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector and further summarizing the output into an output indicating a drowsy state (for example, 4 outputs of eyelids: presence / absence of drowsiness, eyeballs: presence / absence of drowsiness). The state classification unit 1525 shown in FIG. 16 includes a first fully connected layer 1527 and a second fully connected layer 1528. The state classification unit 1525 generates a one-dimensional vector having the desired number of outputs as the number of elements through the first fully connected layer 1527 and the second fully connected layer 1528.

[0189] The probability output layer 1529 applies, for example, a softmax function and makes the sum of the output values equal to 1.0 to give a probabilistic meaning to the output result.

[0190] The second neural network unit 1520 shown in FIG. 16 has two fully connected layers, but even when there are no fully connected layers, it may be configured to have three or more fully connected layers.

[0191] The second state comparison unit 1540 stores in advance a reference value that is a reference for the input value for each of the plurality of layers of the second neural network unit 1520, and uses the difference value between the reference value and the current value that is the value to be newly input to the layer to determine the change in the state to be determined.

[0192] The reference value is the previous value, which is the input value for each layer, stored as the result of the previous processing by the multiple layers of the second neural network unit 1520. This time, it is used to obtain the difference value between the current value, which is the input value about to be newly input to the first layer, and the reference value. In the description of the subsequent processing, the case where the previous value is used as the reference value will be described. In this case, the second state comparison unit 1540 only stores the value without performing the state comparison process for the first processing (processing for the first image), but determines the change in the state to be determined in the subsequent processes (processing for the second and subsequent images). However, the reference value may be stored and used as the following values.

[0193] As the reference value, a typically processed result that has been pre-learned and modeled can be used. That is, the reference value may be the output value of a typically modeled intermediate layer in the neural network. By using the typically processed result, the comparison can be made without being affected by noise or abnormalities included in the actually observed image.

[0194] Also, as the reference value, some results determined by prior information can be used. Some results determined by prior information are assumed to be, for example, adopting pixels with a high probability of the presence of a head. Also, for example, it is assumed not to adopt pixels such as the background where the change is large but the importance is low. That is, the reference value may be the output value of predetermined nodes among the nodes included in the neural network. Thereby, memory savings and reduction of the data to be handled can be achieved, and higher speed can be achieved.

[0195] The second state comparison unit 1540 determines the change in the state to be determined based on the result of determining the magnitude of the difference between the reference value and the current value by using the difference value and a threshold value stored in advance. Specifically, the second state comparison unit 1540 determines that the state to be determined has not changed when the sum of the squares of the difference values is smaller than a threshold value stored in advance for each unit of the image. Also specifically, the second state comparison unit 1540 determines that the state to be determined has not changed when the sum of the absolute values of the difference values stored for each unit of the image is smaller than a threshold value stored in advance.

[0196] The second state comparison unit 1540C shown in FIG. 16 includes a first layer state comparison unit 1541 and a second layer state comparison unit 1542. The first layer state comparison unit 1541 is also referred to as the first fully-connected layer state comparison unit 1541. The second layer state comparison unit 1542 is also referred to as the second fully-connected layer state comparison unit 1542. In the first fully-connected layer state comparison unit 1541 and the second fully-connected layer state comparison unit 1542, "storage" means holding the values of the one-dimensional vectors of their respective input sources. Also, in the first fully-connected layer state comparison unit 1541 and the second fully-connected layer state comparison unit 1542, "comparison" means, for example, taking the absolute value of the difference for each element between the stored one-dimensional vector (at the previous time / previous frame) and the latest one-dimensional vector, and further performing a magnitude comparison between the sum or average of these and a predetermined threshold value.

[0197] The first fully-connected layer state comparison unit 1541 shown in FIG. 16 includes a storage unit 1541a and a comparison processing unit 1541b. The storage unit 1541a stores, for each output by the first fully-connected layer 1527, a reference value that is the output value of the first fully-connected layer 1527 and is the input value of the second fully-connected layer. The comparison processing unit 1541b compares the reference value with the current value.

[0198] The second fully-connected layer state comparison unit 1542 shown in FIG. 16 includes a storage unit 1542a and a comparison processing unit 1542b. The storage unit 1542a stores, for each output of the second fully connected layer, a reference value that is the output value of the second fully connected layer and the input value of the probability output layer 1529. The comparison processing unit 1542b compares the reference value with the current value.

[0199] When the second state comparison unit 1540 determines that the state of the determination target has not changed, the second processing control unit 1580C instructs the second neural network unit 1520 not to execute the processing in the layers after the layer of the determination target, and also instructs to output the determination value stored in the second output storage unit as output data. The second processing control unit 1580C shown in FIG. 16 includes an abnormal state determination processing control unit 1581C. The abnormal state determination processing control unit 1581C functions so as to be able to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500C.

[0200] The second output storage unit 1590 stores the determination value output by the second neural network unit 1520. The output storage unit (second output storage unit) 1590 shown in FIG. 16 includes a probability storage unit 1591.

[0201] The probability storage unit 1591 stores the determination value output by the second neural network unit 1520. The determination value is a value indicating the state of the determination target person, and is, for example, a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.

[0202] Return to the description of FIG. 14. The alarm output unit 2000 acquires output data that is a determination value indicating the drowsy state or dozing state of the determination target person, and outputs an alarm to the determination target person according to the determination value.

[0203] The abnormality determination device 1000C shown in FIG. 14 is shown as a configuration that does not include the alarm output unit 2000, but it may be configured to include the alarm output unit 2000. When configured in this way, the abnormality determination device 1000C is equivalent to the abnormality alarm device 100C shown in FIG. 14. In the following description, unless it is necessary to distinguish between the abnormality alarm device 100C and the abnormality determination device 1000C, the abnormality determination device 1000C will be described as being configured to include the alarm output unit 2000.

[0204] In addition to the above configuration, the abnormality determination device 1000C may include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire abnormality determination device 1000C and each component. The control unit (not shown) activates the abnormality determination device 1000C according to a command from the outside, for example. Further, the control unit (not shown) controls the state of the abnormality determination device 1000C (operating states such as startup, shutdown, sleep). The storage unit (not shown) stores each data used in the abnormality determination device 1000C. The storage unit (not shown) stores, for example, the output (output data) by each component in the abnormality determination device 1000C, and outputs the data requested for each component to the component that requested it. The communication unit (not shown) communicates with an external device. For example, communication is performed between the abnormality determination device 1000C and an imaging device such as an in-vehicle camera. Further, for example, when the abnormality determination device 1000C does not include a display unit or an audio output unit, communication is performed between the abnormality determination device 1000C and an external device such as a display device or an audio output device.

[0205] An example of the processing of the abnormality alarm device 100C and the abnormality determination device 1000C according to Embodiment 3 will be described. FIG. 17 is a flowchart showing an example of the processing of the abnormality alarm device 100C and the abnormality determination device 1000C according to Embodiment 3 of the present disclosure. When an image is input from a camera, for example, the abnormality determination device 1000C starts the processing shown in FIG. 17.

[0206] The abnormality determination device 1000C executes an image acquisition process (step ST3100). In the image acquisition process, the image acquisition unit 1100 of the abnormality determination device 1000C acquires and outputs an image.

[0207] Next, the abnormality determination device 1000C executes a storage and state comparison process (step ST3200). In the state storage and state comparison process, the first state comparison unit 1340C of the abnormality determination device 1000C stores the previous value, which is the input value for each layer and is the processing result of all layers of the first neural network unit 1320 at least for the first time after the start of processing. The first state comparison unit 1340 only stores the value without performing the state comparison process for the first process (the process for the first image), but determines the change in the state to be determined in the processes after the second time (the processes for the second and subsequent images).

[0208] Next, the abnormality determination device 1000C executes feature extraction process control (step ST3300). The feature extraction process control unit 1381 of the abnormality determination device 1000C executes a normal process command or an omission process command for the first neural network unit 1320 according to the determination result by the first state comparison unit 1340. In addition, when the feature extraction process control unit 1381 issues an omission process command, it issues an output command to the combined image storage unit 1391.

[0209] Next, the abnormality determination device 1000C executes a combined image output process (step ST3400). The first neural network unit 1320 executes a normal process and outputs a combined image, or the combined image storage unit 1391 outputs the previous combined image. Thereby, the feature extraction unit 1300 outputs a combined image.

[0210] The abnormality determination device 1000C executes a fully connected layer state storage and fully connected layer state comparison process (step ST3500). In the second state comparison unit 1540 in the abnormality determination device 1000C, for each of the plurality of layers of the second neural network unit 1520, a reference value that is a reference for the input value for the layer is stored in advance, and the difference value between the reference value and the current value that is the value to be newly input to the layer is used to determine the change in the state of the determination target. The second state comparison unit 1540 only stores values without performing state comparison processing for the first-time processing (processing for the first image), but determines the change in the state of the determination target in the processing after the second time (processing for the second image).

[0211] The abnormality determination device 1000C executes abnormality state determination processing control processing (step ST3600). The abnormality state determination processing control unit 1581C issues a normal command or an omission command to the second neural network unit 1520. When the abnormality state determination processing control unit 1581C issues an omission command, it issues an output command to the probability storage unit 1591.

[0212] The abnormality determination device 1000C executes result output processing (step ST3700). The second neural network unit 1520 executes normal processing and outputs a determination value, or the probability storage unit outputs the previous determination value (probability value). Thereby, the abnormality determination unit outputs the determination value as output data.

[0213] The abnormality determination device 1000C executes alarm output processing (step ST3800). In the alarm output processing, the alarm output unit 2000 of the abnormality determination device 1000C acquires the output data and outputs an alarm signal to an alarm device (not shown) or the like based on the output data. The alarm output unit 2000 determines whether to output an alarm based on the determination value included in the output data, and if it is determined to output an alarm, outputs an alarm signal to an alarm device (not shown) or the like.

[0214] When the abnormality determination device 1000C executes the processing of step ST3800, it ends the series of processing shown in FIG. 17 and repeats from the processing of step ST3100. Note that when the camera is turned off, for example, the abnormality determination device 1000C is turned off in conjunction with it.

[0215] A detailed example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment will be described. FIG. 18 is a flowchart showing a first detailed example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.

[0216] When the feature extraction unit 1300C starts processing, first, the first state comparison unit 1340C of the feature extraction unit 1300C executes image storage processing (step ST3210). In the image storage processing, the image state comparison unit 1341 of the first state comparison unit 1340C stores an image that is input data for the first neural network unit 1320. Also, each time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image in the storage unit 1341a.

[0217] The image state comparison unit 1341 executes a process of determining whether there was a previous storage (step ST3211). The image state comparison unit 1341 refers to the storage unit 1341a to determine whether the previously input image is stored.

[0218] When the image state comparison unit 1341 determines that there was a previous storage (step ST3211 “YES”), it executes a comparison process between the current and previous images (step ST3212). The comparison processing unit 1341b of the image state comparison unit 1341 compares the currently input image with the previously input image.

[0219] The comparison processing unit 1341b executes a process of determining whether the difference is less than the threshold value (step ST3213). The comparison processing unit 1341b determines the magnitude of the difference between the reference value and the current value using the difference value between the currently input image and the previously input image and a threshold value stored in advance.

[0220] When the image state comparison unit 1341 determines that the difference between the previous value and the current value is greater than the threshold value (step ST3213 “NO”), the convolutional layer state comparison unit 1342 executes convolutional layer state storage processing (step ST3214). In the convolutional layer state storage processing, the convolutional layer state comparison unit 1342 stores the input data for the convolutional layer in the storage unit 1342a.

[0221] The convolutional layer state comparison unit 1342 executes a process of determining whether there was a previous storage (step ST3215). The convolutional layer state comparison unit 1342 refers to the storage unit 1342a and determines whether the previous value, which is the previously input value, is stored.

[0222] When the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines that there was a previous storage (step ST3215 “YES”), it executes comparison processing between the current and previous times (step ST3216).

[0223] The feature extraction unit 1300C executes a process of determining whether the difference is less than the threshold value (step ST3217). The comparison processing unit 1342b determines the magnitude of the difference between the previous value and the current value using the difference value between the currently input image and the previously input image and a threshold value stored in advance.

[0224] When the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines that the difference between the previous value and the current value is greater than the threshold value (step ST3217 “NO”), the pooling layer state comparison unit 1343 executes pooling layer state storage processing (step ST3218). In the pooling layer state storage processing, the pooling layer state comparison unit 1343 stores the input data for the pooling layer in the storage unit 1343a.

[0225] The comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process of determining whether there was a previous storage (step ST3219).

[0226] When the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that there was a previous storage (step ST3219 “YES”), it executes a comparison process between the current value and the reference value (previous value) (step ST3220).

[0227] The comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process of determining whether the difference between the reference value (previous value) and the current value is less than the threshold value (step ST3221).

[0228] When it is determined by the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolutional layer state comparison unit 1342, and the comparison processing unit 1343b of the pooling layer state comparison unit 1343 that the difference between the reference value (previous value) and the current value is greater than the threshold value (step ST3213 “NO”, step ST3217 “NO”, step ST3221 “NO”), the feature extraction unit 1300C executes normal processing command processing (step ST3311). In the normal processing command processing, the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320.

[0229] The feature extraction unit 1300C executes output processing (step ST3411). The first neural network unit 1320 of the feature extraction unit 1300C outputs a combined image.

[0230] The feature extraction unit 1300C executes output storage processing (step ST3412). The combined image storage unit 1391 of the output storage unit in the feature extraction unit 1300C stores the combined image output from the first neural network unit 1320.

[0231] When it is determined by any one of the comparison processing units 1341b of the image state comparison unit 1341, the comparison processing units 1342b of the convolutional layer state comparison unit 1342, and the comparison processing units 1343b of the pooling layer state comparison unit 1343 that the difference between the previous value and the current value is smaller than the threshold value (step ST3213 “YES”, step ST3217 “YES”, step ST3221 “YES”), the feature extraction unit 1300C executes an omission processing command process (step ST3312). In the omission processing command process, the feature extraction process control unit 1381 commands the first neural network unit 1320 not to execute the process in the layers after the layer to be determined, and commands to output the feature map stored in the first output storage unit 1390.

[0232] The feature extraction unit 1300C executes stored data output processing (step ST3413). The feature extraction unit 1300C ends the process shown in FIG. 18.

[0233] FIG. 19 is a flowchart showing a first detailed example of the process of the abnormality state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to Embodiment 3 of the present disclosure. When the abnormality state determination unit 1500C acquires the feature map output from the feature extraction unit 1300C, the process starts. The abnormality state determination unit 1500C first executes a process (step ST3510) on the data output by the feature extraction unit 1300.

[0234] The abnormality state determination unit 1500C acquires the fully connected layer output (step ST3511). The second state comparison unit 1540 in the abnormality state determination unit 1500C acquires the output value output from the first fully connected layer 1527.

[0235] The first fully-connected layer state comparison unit 1541 executes the first fully-connected layer state storage process (step ST3512). The first fully-connected layer state comparison unit 1541 stores, as a reference value, the output value from the first fully-connected layer 1527, which is the input value to the second fully-connected layer 1528.

[0236] The first fully-connected layer state comparison unit 1541 executes a process of determining whether there was a previous storage (step ST3513). The first fully-connected layer state comparison unit 1541 refers to the storage unit 1541a and determines whether the reference value (previous value) is stored.

[0237] When the first fully-connected layer state comparison unit 1541 determines that there was a previous storage (step ST3513 “YES”), it executes a comparison process between the current time and the previous time (step ST3514). The first fully-connected layer state comparison unit 1541 compares the current value with the reference value (previous value).

[0238] The first fully-connected layer state comparison unit 1541 executes a process of determining whether the difference is smaller than the threshold value (step ST3515). The comparison processing unit 1541b determines the magnitude of the difference between the reference value and the current value by using the difference value between the current value and the reference value (previous value) and the threshold value stored in advance.

[0239] When it is determined by the first fully-connected layer state comparison unit 1541 that the difference is greater than the threshold value, normal processing is executed in the second fully-connected layer 1528, and the second fully-connected layer state comparison unit 1542 executes the second fully-connected layer state storage process (step ST3516). The second fully-connected layer state comparison unit 1542 stores, as a reference value, the output value from the second fully-connected layer 1528, which is the input value to the probability output layer 1529.

[0240] The second fully-connected layer state comparison unit 1542 executes a process of determining whether there was a previous storage (step ST3517). The second fully-connected layer state comparison unit 1542 refers to the storage unit 1542a and determines whether the reference value (previous value) is stored.

[0241] When the second fully-connected layer state comparison unit 1542 determines that it was stored last time (step ST3517 “YES”), it executes comparison processing between the current time and the previous time (step ST3518). The second fully-connected layer state comparison unit 1542 compares the current value with the reference value (previous value).

[0242] The second fully-connected layer state comparison unit 1542 executes a difference <threshold? determination process (step ST3519). The comparison processing unit 1542b determines the magnitude of the difference between the reference value and the current value by using the difference value between the current value and the reference value (previous value) and the threshold value stored in advance.

[0243] When it is determined by the first fully-connected layer state comparison unit 1541 that the difference is greater than the threshold value (step ST3515 “NO”), and when it is determined by the second fully-connected layer state comparison unit 1542 that the difference is greater than the threshold value (step ST3519 “NO”), the abnormal state determination unit 1500C executes normal processing command processing (step ST3611).

[0244] The abnormal state determination unit 1500C executes output processing (step ST3711). The neural network unit 1520 (second neural network unit 1520) in the abnormal state determination unit 1500C outputs a determination value.

[0245] The abnormal state determination unit 1500C executes output storage processing (step ST3712). The output storage unit 1590 (second output storage unit 1590) in the abnormal state determination unit 1500C stores the determination value output by the neural network unit 1520 (second neural network unit 1520).

[0246] When it is determined by the first fully-connected layer state comparison unit 1541 that the difference is smaller than the threshold value, and when it is determined by the second fully-connected layer state comparison unit 1542 that the difference is smaller than the threshold value, the abnormal state determination unit 1500C executes an omission processing command (step ST3519). In the omission processing command, the abnormal state determination processing control unit 1581 commands the second neural network unit 1520 not to execute the processing in the layers after the layer to be determined, and also commands to output the determination value stored in the second output storage unit 1590.

[0247] The abnormal state determination unit 1500C executes stored data output processing (step ST3713). The second output storage unit 1590 outputs a determination value.

[0248] When the second neural network unit 1520 or the second output storage unit 1590 outputs a determination value, the series of processes shown in FIG. 19 ends.

[0249] Next, another detailed example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to Embodiment 3 will be described. FIG. 20 is a flowchart showing a second detailed example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to Embodiment 3 of the present disclosure.

[0250] When the feature extraction unit 1300C starts processing, first, the first state comparison unit 1340C of the feature extraction unit 1300C executes image storage processing (step ST3210). In the image storage processing, the image state comparison unit 1341 of the first state comparison unit 1340C stores an image that is input data for the first neural network unit 1320. Also, each time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image in the storage unit 1341a.

[0251] The image state comparison unit 1341 executes a process of determining whether it was stored last time (step ST3211). The image state comparison unit 1341 refers to the storage unit 1341a and determines whether the previously input image is stored.

[0252] When the image state comparison unit 1341 determines that it was stored last time (step ST3211 “YES”), it executes a comparison process between this time and last time (step ST3212). The comparison processing unit 1341b of the image state comparison unit 1341 compares the currently input image with the previously input image.

[0253] The comparison processing unit 1341b executes a process of determining whether the difference is less than the threshold value (step ST3213). The comparison processing unit 1341b determines the magnitude of the difference between the reference value and the current value using the difference value between the currently input image and the previously input image and the threshold value stored in advance.

[0254] When it is determined by the image state comparison unit 1341 that it was not stored last time (step ST3211 “NO”), or when it is determined by the image state comparison unit 1341 that the difference between the previous value and the current value is greater than or equal to the threshold value (step ST3213 “NO”), a normal processing command (step ST3231) is issued by the feature extraction processing control unit 1381C, and the process by the convolutional layer unit 1322 is executed. Next, the convolutional layer state comparison unit 1342 executes a convolutional layer state storage process (step ST3214). In the convolutional layer state storage process, the convolutional layer state comparison unit 1342 stores the input data for the convolutional layer in the storage unit 1342a.

[0255] The convolutional layer state comparison unit 1342 executes a process of determining whether it was stored last time (step ST3215). The convolutional layer state comparison unit 1342 refers to the storage unit 1342a and determines whether the previous value (reference value), which is the previously input value, is stored.

[0256] When the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines that it was stored last time (step ST3215 “YES”), it executes the comparison process between this time and last time (step ST3216).

[0257] The feature extraction unit 1300C executes a process of determining whether the difference is less than the threshold value (step ST3217). Specifically, the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines the magnitude of the difference between the previous value and the current value by using the difference value between the value that is about to be input this time (current value) and the value that was input last time (previous value) to the pooling layer unit 1325 and the threshold value stored in advance.

[0258] When it is determined by the convolutional layer state comparison unit 1342 that it was not stored last time (step ST3211 “NO”), or when it is determined by the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 that the difference between the previous value and the current value is greater than or equal to the threshold value (step ST3217 “NO”), a normal processing command (step ST3232) is issued by the feature extraction processing control unit 1381C, and the process by the pooling layer unit 1325 is executed. Next, in the pooling layer state storage process, the pooling layer state comparison unit 1343 stores the input data (input value to be processed) for the pooling layer in the storage unit 1343a.

[0259] Next, the comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process of determining whether it was stored last time (step ST3219).

[0260] When the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that it was stored last time (step ST3219 “YES”), it executes the comparison process between the current value and the reference value (previous value) (step ST3220).

[0261] The comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process of determining whether the difference between the reference value (previous value) and the current value is less than the threshold value (step ST3221).

[0262] When it is determined by the comparison processing unit 1343b of the pooling layer state comparison unit 1343 that the difference between the reference value (previous value) and the current value is equal to or greater than the threshold value (step ST3221 “NO”), the feature extraction unit 1300C executes normal processing command processing (step ST3311). In the normal processing command processing, the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320. The image combining unit 1328 in the first neural network unit 1320 executes image combining processing.

[0263] The feature extraction unit 1300C executes output storage processing (step ST3412). The combined image storage unit 1391 of the output storage unit in the feature extraction unit 1300C stores the combined image output from the first neural network unit 1320.

[0264] When it is determined by any one of the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolutional layer state comparison unit 1342, and the comparison processing unit 1343b of the pooling layer state comparison unit 1343 that the difference between the previous value and the current value is less than the threshold value (the difference value is less than the threshold value) (step ST3213 “YES”, step ST3217 “YES”, step ST3221 “YES”), the feature extraction unit 1300C executes omission processing command processing (step ST3312). In the omission processing command processing, the feature extraction processing control unit 1381 commands the first neural network unit 1320 not to execute the processing in the layers after the layer to be determined, and commands to output the feature map stored in the first output storage unit 1390.

[0265] The feature extraction unit 1300C executes stored data output processing (step ST3413). The feature extraction unit 1300C ends the processing shown in FIG. 20.

[0266] FIG. 21 is a flowchart showing a second detailed example of the processing of the abnormality determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to Embodiment 3 of the present disclosure. When the abnormality determination unit 1500C acquires the feature map output from the feature extraction unit 1300C, it starts processing. The abnormality determination unit 1500C first executes processing on the data output by the feature extraction unit 1300 (step ST3510).

[0267] The abnormality determination unit 1500C acquires the fully connected layer output (step ST3511). The second state comparison unit 1540 in the abnormality determination unit 1500C acquires the output value output from the first fully connected layer 1527.

[0268] The first layer state comparison unit 1541 executes the first layer state storage process (step ST3512). The first fully connected layer state comparison unit 1541 stores, as a reference value, the output value from the first fully connected layer 1527, which is the input value to the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528).

[0269] The first layer state comparison unit 1541 executes a process of determining whether there was a previous storage (step ST3513). The first layer state comparison unit 1541 refers to the storage unit 1541a and determines whether the reference value (previous value) is stored.

[0270] When the first layer state comparison unit 1541 determines that there was a previous storage (step ST3513 “YES”), it executes a comparison process between the current time and the previous time (step ST3514). The first fully connected layer state comparison unit 1541 compares the current value with the reference value (previous value).

[0271] The first layer state comparison unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST3515). The comparison processing unit 1541b in the first layer state comparison unit 1541 determines the magnitude of the difference between the reference value and the current value by using the difference value between the current value and the reference value (previous value) and a threshold value stored in advance.

[0272] When it is determined by the first layer state comparison unit 1541 that it was not stored previously (step ST3211 “NO”), or when it is determined by the comparison processing unit 1541b in the first layer state comparison unit 1541 that the difference value between the current value and the reference value (previous value) is greater than or equal to the threshold value (step ST3515 “NO”), a normal processing command (step ST3520) is issued by the abnormal state determination processing control unit 1581C, and the processing by the second fully-connected layer 1528 is executed.

[0273] Next, the second fully-connected layer state comparison unit 1542 executes a second fully-connected layer state storage process (step ST3516). The second fully-connected layer state comparison unit 1542 stores, as a reference value, the output value from the second fully-connected layer 1528, which is the input value to the probability output layer 1529 (the value before being processed by the probability output layer 1529).

[0274] Next, the second fully-connected layer state comparison unit 1542 executes a process of determining whether it was stored previously (step ST3517). The second fully-connected layer state comparison unit 1542 refers to the storage unit 1542a and determines whether the reference value (previous value) is stored.

[0275] When the second fully-connected layer state comparison unit 1542 determines that it was stored previously (step ST3517 “YES”), it executes a comparison process between the current time and the previous time (step ST3518). The second fully-connected layer state comparison unit 1542 compares the current value with the reference value (previous value).

[0276] The second fully-connected layer state comparison unit 1542 executes a difference <threshold> determination process (step ST3519). The comparison processing unit 1542b in the second fully-connected layer state comparison unit 1542 determines the magnitude of the difference between the reference value and the current value using the difference value between the current value and the reference value (previous value) and a threshold value stored in advance.

[0277] If it is determined by the second fully-connected layer state comparison unit 1542 that it was not stored previously (step ST3517 “NO”), or if it is determined by the comparison processing unit 1542b in the second fully-connected layer state comparison unit 1542 that the difference is greater than or equal to the threshold value (step ST3519 “NO”), the abnormal state determination unit 1500C executes a normal processing command process (step ST3611).

[0278] The abnormal state determination unit 1500C executes an output process (step ST3711). The second neural network unit 1520 in the abnormal state determination unit 1500C outputs a determination value.

[0279] The abnormal state determination unit 1500C executes an output storage process (step ST3712). The second output storage unit 1590 in the abnormal state determination unit 1500C stores the determination value output by the second neural network unit 1520 in the probability storage unit 1591.

[0280] If it is determined by the first layer state comparison unit 1541 that the difference is less than the threshold value (less than the threshold value), and if it is determined by the comparison processing unit 1542b in the second layer state comparison unit 1542 that the difference is less than the threshold value, the abnormal state determination unit 1500C executes an omission process command (step ST3519). In the omission process command, the abnormal state determination process control unit 1581C commands the second neural network unit 1520 not to execute the process in the layers after the layer to be determined, and commands the second output storage unit 1590 to output the determination value stored therein.

[0281] The abnormal state determination unit 1500C executes stored data output processing (step ST3713). The second output storage unit 1590 receives a command from the abnormal state determination processing control unit 1581C and outputs the determination value (probability value) stored in the probability storage unit 1591.

[0282] When the second neural network unit 1520 or the second output storage unit 1590 outputs a determination value (probability value), the series of processes shown in FIG. 21 ends.

[0283] As described above, according to the third embodiment, in each state storage / comparison processing unit, when it is determined that the change is small, subsequent processing can be avoided, so the amount of calculation is reduced accordingly, and high-speed processing can be achieved. At this time, since the output result is the previous highly accurate determination result, the output of highly accurate results can be continued.

[0284] The abnormal determination device of the present disclosure is further configured as follows. "The neural network unit includes a first neural network unit and a second neural network unit, The output storage unit includes a first output storage unit and a second output storage unit, The state comparison unit includes a first state comparison unit and a second state comparison unit, The processing control unit includes a first processing control unit and a second processing control unit, The first neural network unit has a plurality of layers that are part of the layers constituting the neural network, acquires the image output by the image acquisition unit, and outputs a feature map representing the characteristic state of the determination target included in the image. The first output storage unit stores the feature map output by the first neural network unit. For each of the plurality of layers in the first neural network unit, a reference value that is a reference for the input value to the layer is stored in advance, and the change in the state to be determined is determined using the difference value between the reference value and the current value that is the value to be newly input to the layer. The first state comparison unit and When the first state comparison unit determines that the state to be determined has not changed, the first neural network unit is instructed not to execute the processing in the layers after the layer to be determined, and the first output storage unit is instructed to output the feature map stored therein. The first processing control unit A feature extraction unit configured to include The neural network has a plurality of layers, acquires the feature map output by the feature extraction unit, and outputs, as the output data, a determination value indicating the state to be determined using the feature map. The second neural network unit The second output storage unit that stores the determination value output by the second neural network unit For each of the plurality of layers in the second neural network unit, a reference value that is a reference for the input value to the layer is stored in advance, and the change in the state to be determined is determined using the difference value between the reference value and the current value that is the value to be newly input to the layer. The second state comparison unit and When the second state comparison unit determines that the state to be determined has not changed, the second neural network unit is instructed not to execute the processing in the layers after the layer to be determined, and the second output storage unit is instructed to output the determination value stored therein as the output data. The second processing control unit An abnormal state determination unit configured to include An abnormality determination device comprising. As a result, the present disclosure can provide an abnormality determination device that can make it possible to output a determination result with higher accuracy more quickly using a neural network in the abnormality determination technology, and has the effect of achieving this. Furthermore, by applying the above configuration to the above abnormality determination method, the present disclosure has the same effect as the above effect.

[0285] Here, a hardware configuration for realizing the functions of the abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) of the present disclosure described above will be described. FIG. 22 is a diagram showing a first example of a hardware configuration for realizing the functions of the present disclosure. FIG. 23 is a diagram showing a second example of a hardware configuration for realizing the functions of the present disclosure. The abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) of the present disclosure are each realized by hardware as shown in FIG. 22 or FIG. 23.

[0286] The abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) are each composed of, for example, a processor 10001, a memory 10002, and a communication circuit 10004 as shown in FIG. 22. The processor 10001 and the memory 10002 are, for example, those installed in a computer. The memory 10002 stores a program for causing the computer to function as an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, an output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, a first neural network unit 1320, an image branching unit 1321, a convolutional layer unit 1322, a first convolutional layer 1323, a second convolutional layer 1324, a pooling layer unit 1325, a first pooling layer 1326, a second pooling layer 1327, an image combining unit 1328, first state comparison units 1340, 1340B, 1340C, an image state comparison unit 1341, a comparison processing unit 1341b, a convolutional layer state comparison unit 1342, a comparison processing unit 1342b, a pooling layer state comparison unit 1343, a comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, a second neural network unit 1520, a state classification unit 1525, a first fully connected layer 1527, a second fully connected layer 1528, a probability output layer 1529, a second state comparison unit 1540, a first fully connected layer state comparison unit 1541, a comparison processing unit 1541b, a second fully connected layer state comparison unit 1542, a comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, an alarm output unit 2000, and a control unit (not shown).By the processor 10001 reading and executing the program stored in the memory 10002, the functions of the image acquisition unit 1010, neural network unit 1020, state comparison unit 1030, processing control unit 1040, output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, first neural network unit 1320, image branching unit 1321, convolutional layer unit 1322, first convolutional layer 1323, second convolutional layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combining unit 1328, first state comparison units 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolutional layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and the functions of the control unit (not shown) are realized. Also, a storage unit (not shown) is realized by the memory 10002 or another memory (not shown). Further, in the abnormal alarm devices 100(100A, 100B, 100C) and abnormal determination devices 1000(1000A, 1000B, 1000C), the storage units 1341a, 1342a, 1343a, first output storage unit 1390, combined image storage unit 1391, storage units 1541a, 1542a, second output storage unit 1590, and probability storage unit 1591 are realized by the memory 10002 or another memory (not shown). Also, a communication unit (not shown) is realized by the communication circuit 10004. The processor 10001 is, for example, one using a CPU (Central Processing Unit), GPU (Graphics Processing Unit), microprocessor, microcontroller, or DSP (Digital Signal Processor), etc. The memory 10002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory, or may be a magnetic disk such as a hard disk or flexible disk, or may be an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or may be a magneto-optical disk. The processor 10001 and the memory 10002 or the communication circuit 10004 are connected in a state where they can transmit data to each other. Further, the processor 10001, the memory 10002, and the communication circuit 10004 are connected in a state where they can transmit data to and from other hardware via the input / output interface 10003.

[0287] Alternatively, the functions of the image acquisition unit 1010, neural network unit 1020, state comparison unit 1030, processing control unit 1040, output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, first neural network unit 1320, image branching unit 1321, convolutional layer unit 1322, first convolutional layer 1323, second convolutional layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combining unit 1328, first state comparison units 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolutional layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully-connected layer 1527, second fully-connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully-connected layer state comparison unit 1541, comparison processing unit 1541b, second fully-connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and the functions of the control unit (not shown) may be realized by a dedicated processing circuit 20001 as shown in FIG. 23. Also, a communication unit (not shown) is realized by the communication circuit 10004. The processing circuit 20001 uses, for example, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration), etc. Also, a storage unit (not shown) is realized by the memory 20002 or another memory (not shown). The memory 20002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory, or may be a magnetic disk such as a hard disk or flexible disk, or may be an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or may be a magneto-optical disk. Also, a communication unit (not shown) is realized by the communication circuit 20004. The processing circuit 20001 is connected to the memory 20002 or the communication circuit 20004 in a state where data can be transmitted therebetween. Also, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state where data can be transmitted to and from other hardware via the input / output interface 20003. Note that the functions of the image acquisition unit 1010, neural network unit 1020, state comparison unit 1030, processing control unit 1040, output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, first neural network unit 1320, image branching unit 1321, convolutional layer unit 1322, first convolutional layer 1323, second convolutional layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combination unit 1328, first state comparison units 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolutional layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and the functions of the control unit (not shown) may be realized by separate processing circuits or may be realized collectively by a processing circuit.

[0288] Alternatively, some functions of the image acquisition unit 1010, neural network unit 1020, state comparison unit 1030, processing control unit 1040, output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, first neural network unit 1320, image branching unit 1321, convolutional layer unit 1322, first convolutional layer 1323, second convolutional layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combining unit 1328, first state comparison units 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolutional layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and a control unit (not shown) may be realized by the processor 10001 and the memory 10002, and the remaining functions may be realized by the processing circuit 20001.

[0289] Note that in the present disclosure, any combination of the embodiments, any modification of any component of each embodiment, or any omission of any component of each embodiment is possible.

Industrial Applicability

[0290] The present disclosure can enable, in an abnormality determination technique, outputting a highly accurate determination result quickly using a neural network without performing operations in all layers of the neural network. For example, abnormality determination such as fast and highly accurate drowsiness determination becomes possible, and thus it is suitable for application to, for example, an in-vehicle driver monitoring system.

Explanation of Signs

[0291] 100, 100A, 100B, 100C abnormal alarm devices, 1000, 1000A, 1000B, 1000C abnormality determination devices, 1010 image acquisition unit, 1020 neural network unit, 1030 state comparison unit, 1040 processing control unit, 1050 output storage unit, 1100, 1100B, 1100C image acquisition units, 1300, 1300B, 1300C feature extraction units, 1320 neural network unit (first neural network unit), 1321 image branching unit, 1322 convolutional layer unit, 1323 first convolutional layer, 1324 second convolutional layer, 1325 pooling layer unit 1326 first pooling layer, 1327 second pooling layer, 1328 image combination unit, 1340, 1340B, 1340C state comparison units (first state comparison units), 1341 image state comparison unit, 1341a storage unit, 1341b comparison processing unit, 1342 convolutional layer state comparison unit, 1342a storage unit, 1342b comparison processing unit, 1343 pooling layer state comparison unit, 1343a storage unit, 1343b comparison processing unit, 1380, 1380B, 1380C processing control units (first processing control units), 1381, 1381B, 1381C feature extraction processing control units, 1390 output storage unit (first output storage unit), 1391 combined image storage unit, 1500, 1500B, 1500C abnormal state determination units, 1520 neural network unit (second neural network unit), 1525 state classification unit, 1527 first fully connected layer, 1528 second fully connected layer, 1529 probability output layer, 1540 state comparison unit (second state comparison unit), 1541 first layer state comparison unit (first fully connected layer state comparison unit), 1541a storage unit, 1541b comparison processing unit, 1542 second layer state comparison unit (second fully connected layer state comparison unit), 1542a storage unit, 1542b comparison processing unit, 1580, 1580B, 1580C processing control units (second processing control units), 1581, 1581B, 1581C abnormal state determination processing control units, 1590 output storage unit (second output storage unit), 1591 probability storage unit, 2000 alarm output unit, 10001 processor, 10002 memory, 10003 input / output interface, 10004 communication circuit, 20001 processing circuit, 20002 memory, 20003 input / output interface, 20004 communication circuit.

Claims

1. An image acquisition unit that acquires and outputs an image; A neural network unit having a plurality of layers in the neural network, acquiring the image output by the image acquisition unit, and outputting a determination result indicating the state of the determination target included in the image; An output storage unit that stores the output data output by the neural network unit; A reference value that is a reference for the input value to at least one layer among the plurality of layers of the neural network unit is stored in advance, and the difference value between the reference value and the current value, which is the value to be newly input to the first layer, is used to determine the change in the state of the determination target. A state comparison unit; When the state comparison unit determines that the state of the determination target has not changed, it commands the neural network unit not to execute the processing in the layers after the first layer, and commands to output the output data stored in the output storage unit. A processing control unit; An abnormality determination device comprising the above.

2. The neural network unit includes a first neural network unit and a second neural network unit; The output storage unit includes a first output storage unit and a second output storage unit; The state comparison unit includes a first state comparison unit and a second state comparison unit; The processing control unit includes a first processing control unit and a second processing control unit; The first neural network unit having a plurality of layers that are part of the layers constituting the neural network, acquiring the image output by the image acquisition unit, and outputting a feature map representing the characteristic state of the determination target included in the image; The first output storage unit that stores the feature map output by the first neural network unit; A reference value that is a reference for the input value to at least one layer among the layers in the first neural network unit, which is the second layer, is stored in advance, and the difference value between the reference value and the current value, which is the value to be newly input to the second layer, is used to determine the change in the state of the determination target. The first state comparison unit; And, When the first state comparison unit determines that the state of the determination target has not changed, it instructs the first neural network unit not to execute the processing in the layers after the second layer of the determination target, and at the same time instructs the first output storage unit to output the feature map stored therein. The first processing control unit, A feature extraction unit configured to include A second neural network unit having a plurality of layers that are part of the layers constituting the neural network, acquiring the feature map output by the feature extraction unit, and using the feature map to output a determination value indicating the state of the determination target as the output data, The second output storage unit that stores the determination value output by the second neural network unit, A reference value that is a reference for the input value to at least one layer, the third layer, among the layers in the second neural network unit is stored in advance, and the difference value between the reference value and the current value, which is the value to be newly input to the third layer, is used to determine the change in the state of the determination target. The second state comparison unit, And When the second state comparison unit determines that the state of the determination target has not changed, it instructs the second neural network unit not to execute the processing in the layers after the third layer of the determination target, and at the same time instructs the second output storage unit to output the determination value stored therein as the output data. The second processing control unit, An abnormal state determination unit configured to include The abnormal determination device according to claim 1, comprising

3. The neural network unit is configured to include a first neural network unit and a second neural network unit, The output storage unit is configured to include a first output storage unit and a second output storage unit, The state comparison unit is configured to include a first state comparison unit and a second state comparison unit, The processing control unit is configured to include a first processing control unit and a second processing control unit, A first neural network unit having a plurality of layers that are part of the layers constituting the neural network, acquiring the image output by the image acquisition unit, and outputting a feature map representing the characteristic state of the determination target included in the image, The first output storage unit that stores the feature map output by the first neural network unit, For each of the plurality of layers in the first neural network unit, a reference value that is a reference for the input value to the layer is stored in advance, and a difference value between the reference value and the current value, which is the value to be newly input to the layer, is used to determine a change in the state of the determination target, the first state comparison unit. And, When it is determined by the first state comparison unit that there is no change in the state of the determination target, the first neural network unit is commanded not to execute the processing in the layers after the layer of the determination target, and the first output storage unit is commanded to output the feature map stored therein, the first processing control unit. A feature extraction unit configured to include: The neural network has a plurality of layers, obtains the feature map output by the feature extraction unit, and outputs a determination value indicating the state of the determination target using the feature map as the output data, the second neural network unit. The second output storage unit that stores the determination value output by the second neural network unit. For each of the plurality of layers of the second neural network unit, a reference value that is a reference for the input value to the layer is stored in advance, and a difference value between the reference value and the current value, which is the value to be newly input to the layer, is used to determine a change in the state of the determination target, the second state comparison unit. And, When it is determined by the second state comparison unit that there is no change in the state of the determination target, the second neural network unit is commanded not to execute the processing in the layers after the layer of the determination target, and the second output storage unit is commanded to output the determination value stored therein as the output data, the second processing control unit. An abnormal state determination unit configured to include: The abnormal determination device according to claim 1, comprising:

4. The reference value is one that stores the previous value for each layer as the previous processing result in the neural network, and the abnormal determination device according to any one of claims 1 to 3, characterized in that.

5. The reference value is an output value of a pre-modeled typical intermediate layer in the neural network. The abnormal determination device according to any one of claims 1 to 3, characterized in that.

6. The reference value is the output value of a predetermined node among the nodes included in the neural network. The abnormality determination device according to any one of claims 1 to 3, characterized in that.

7. The state comparison unit Determines a change in the state of the determination target based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a threshold value stored in advance. The abnormality determination device according to claim 1, characterized in that.

8. The state comparison unit In units of the image, when the sum of the squares of the difference values is smaller than a threshold value stored in advance, it is determined that the state of the determination target has not changed. The abnormality determination device according to claim 7, characterized in that.

9. At least one of the first state comparison unit and the second state comparison unit Determines a change in the state of the determination target based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a threshold value stored in advance. The abnormality determination device according to claim 2 or claim 3, characterized in that.

10. At least one of the first state comparison unit and the second state comparison unit In units of the image, when the sum of the squares of the difference values is smaller than a threshold value stored in advance, it is determined that the state of the determination target has not changed. The abnormality determination device according to claim 9, characterized in that.

11. The state comparison unit In units of the image, when the sum of the absolute values of the difference values is smaller than a threshold value stored in advance, it is determined that the state of the determination target has not changed. The abnormality determination device according to claim 7, characterized in that.

12. At least one of the first state comparison unit and the second state comparison unit In units of the image, when the sum of the absolute values of the difference values is smaller than a threshold value stored in advance, it is determined that the state of the determination target has not changed. The abnormality determination device according to claim 9, characterized in that.

13. Further includes an alarm output unit that acquires the output data, which is a determination value indicating the drowsy state or dozing state of the person to be determined, and outputs an alarm to the person to be determined according to the determination value. The abnormality determination device according to any one of claims 1, 2, 3, 7, 8, and 11, characterized in that.

14. The image acquisition unit outputs the acquired image to the neural network unit. The neural network unit having a plurality of layers in the neural network acquires the image output by the image acquisition unit and outputs a determination result indicating the state of the determination target included in the image. The output storage unit stores the output data output by the neural network unit. The state comparison unit stores in advance a reference value that is a reference for the input value for a first layer that is at least one of the plurality of layers of the neural network unit, and uses the difference value between the reference value and the current value that is the value to be newly input to the first layer to determine the change in the state of the determination target. When the state comparison unit determines that there is no change in the state of the determination target, the processing control unit commands the neural network unit not to execute the processing in the layers after the first layer, and commands to output the output data stored in the output storage unit. Abnormality determination method.

Images