Abnormality determination device and abnormality determination method
The abnormality determination device employs a neural network with a state comparison unit and processing control to efficiently assess driver states, addressing the challenges of complexity and speed in neural network-based abnormality determination, thereby enhancing accuracy and safety.
Patent Information
- Application Number
- PCT/JP2023/018362
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-21
AI Technical Summary
Existing abnormality determination technologies using neural networks face challenges in achieving high accuracy and quick output due to the complexity and computational intensity required for precise analysis, particularly in determining the state of a driver in a vehicle, leading to potential errors in safety-critical applications.
The proposed solution involves an abnormality determination device with an image acquisition unit, a neural network unit with multiple layers, a state comparison unit that uses reference values to determine changes in the input state, and a processing control unit that instructs the neural network to skip unnecessary processing layers when the state has not changed, allowing for efficient and accurate output.
This configuration enables the rapid and accurate determination of abnormal states, reducing computational load and improving safety by minimizing errors in driver state assessment.
Smart Images

Figure JP2023018362_21112024_PF_FP_ABST
Abstract
Description
Abnormality determination device and abnormality determination method
[0001] The disclosed technology relates to an abnormality determination technology for determining the state of a person to be determined.
[0002] Some abnormality determination techniques determine the state of a subject of determination using measurement results from a sensor device (such as a camera, radar, or infrared sensor). Among these abnormality determination techniques, for example, an abnormality determination technique for determining the state of a subject of determination, such as a driver (occupant) of a vehicle (mobile object), is desirable to perform a highly accurate determination to prevent erroneous determination because it is related to the safety of the subject of determination. In response to this, research into techniques using deep learning has been progressing in recent years in the field of abnormality determination techniques. Deep learning uses a mathematical model called a neural network (NN). The neural network outputs results through layers such as an input layer, a hidden layer (intermediate layer), and an output layer. Patent Document 1 discloses an identification device that uses a convolutional neural network (CNN), a type of neural network, to identify when a driver of a mobile object has become abnormal. Specifically, the identification device of Patent Document 1 uses a convolutional neural network to acquire skeletal information of the driver from the image in a process of identifying the driver's state (e.g., whether or not the driver is in an abnormal state) based on the acquired image. The identification device of Patent Document 1 uses the neural network to improve the accuracy of the skeletal information of the driver.
[0003] Japanese Patent Application Laid-Open No. 2020-052869
[0004] On the other hand, the abnormality detection technology for moving objects as described above often requires not only high accuracy but also rapid detection. To achieve high accuracy using the above-described neural network in the abnormality detection technology, for example, a method can be adopted in which the neural network extracts and uses a large number of detection indicators indicating the state of the subject from an image (e.g., a large number of indicators, such as the open / closed state of the eyelids, the number of blinks, the position of the face, the state of electroencephalograms, the state of heat dissipation, and driving performance (e.g., reaction time)). If the above method is used to output a highly accurate abnormality detection result using a neural network, the number of layers in the neural network increases or the configuration becomes complex, resulting in increased computational complexity and computation time. Therefore, the abnormality detection technology has a problem in that it is difficult to quickly output a highly accurate detection result using a neural network. The identification device of Patent Document 1 merely outputs skeletal information from an image via a neural network, but does not solve the above problem.
[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to enable an abnormality determination technique to quickly output highly accurate determination results using a neural network.
[0006] The abnormality determination device of the present disclosure includes an image acquisition unit that acquires and outputs an image; a neural network unit that has a plurality of layers in a neural network, acquires the image output by the image acquisition unit, and outputs a determination result that indicates the state of an object to be determined contained in the image; an output storage unit that stores output data output by the neural network unit; a state comparison unit that pre-stores a reference value that is a reference for an input value to a first layer that is at least one of the plurality of layers of the neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value that is a value that is about to be newly input to the first layer; and a processing control unit that, when the state comparison unit determines that the state of the object to be determined has not changed, instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs it to output the output data stored in the output storage unit.
[0007] According to the present disclosure, it is possible to achieve an effect of enabling an abnormality determination technique to quickly output a highly accurate determination result using a neural network.
[0008] FIG. 1 is a diagram illustrating an example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to embodiment 1 of the present disclosure. FIG. 2 is a flowchart illustrating an example of processing by the abnormality warning device 100A and the abnormality determination device 1000A according to embodiment 1 of the present disclosure. FIG. 3 is a flowchart illustrating an example of more detailed processing by the abnormality warning device 100A and the abnormality determination device 1000A according to embodiment 1 of the present disclosure. FIG. 4 is a diagram illustrating an example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to embodiment 2 of the present disclosure. FIG. 5 is a diagram illustrating an example of the internal configuration of a feature extraction unit 1300B in the abnormality warning device 100B and the abnormality determination device 1000B. FIG. 6 is a diagram illustrating an example of the internal configuration of an abnormal state determination unit 1500B in the abnormality warning device 100B and the abnormality determination device 1000B. FIG. 7 is a flowchart illustrating an example of processing by the abnormality warning device 100B and the abnormality determination device 1000B according to embodiment 2 of the present disclosure. FIG. 8 is a flowchart showing an example of more detailed processing by the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure. FIG. 9 is a flowchart showing an example of processing by the feature extraction unit 1300B when the state comparison unit 1030 (first state comparison unit 1030B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is an image state comparison unit 1341. FIG. 10 is a flowchart showing an example of processing by the feature extraction unit 1300B when the state comparison unit 1030 (first state comparison unit 1030B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is a convolutional layer state comparison unit. FIG. 11 is a flowchart showing an example of processing by the feature extraction unit 1300B when the state comparison unit 1030 (first state comparison unit 1030B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is a pooling layer state comparison unit. Fig. 12 is a flowchart showing an example of processing by abnormal state determination unit 1500B in the second embodiment of the present disclosure when the state comparison unit (second state comparison unit 1540B) in abnormal state determination unit 1500B is the first fully connected layer state comparison unit 1541. Fig. 13 is a flowchart showing an example of processing by abnormal state determination unit 1500B in the second embodiment of the present disclosure when the state comparison unit (second state comparison unit 1540B) in abnormal state determination unit 1500B is the second fully connected layer state comparison unit.FIG. 14 is a diagram illustrating an example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 15 is a diagram illustrating an example of the internal configuration of a feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C. FIG. 16 is a diagram illustrating an example of the internal configuration of an abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C. FIG. 17 is a flowchart illustrating an example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 18 is a flowchart illustrating a detailed first example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 19 is a flowchart illustrating a detailed first example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. Fig. 20 is a flowchart showing a second detailed example of the processing of feature extraction unit 1300C in abnormality warning device 100C and abnormality determination device 1000C according to embodiment 3 of the present disclosure. Fig. 21 is a flowchart showing a second detailed example of the processing of abnormal state determination unit 1500C in abnormality warning device 100C and abnormality determination device 1000C according to embodiment 3 of the present disclosure. Fig. 22 is a diagram showing a first example of a hardware configuration for realizing the functions of the present disclosure. Fig. 23 is a diagram showing a second example of a hardware configuration for realizing the functions of the present disclosure.
[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings. In the following description, ordinal numbers such as "first," "second," and "third" may be used. These terms are used for convenience to facilitate understanding of the contents of the embodiments, and are not limited to the order that may result from these ordinal numbers.
[0010] First Embodiment In a first embodiment, one mode of the basic configuration of the present disclosure will be described.
[0011] An example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to embodiment 1 will be described. FIG. 1 is a diagram showing an example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to embodiment 1 of the present disclosure. The abnormality warning device 100A acquires an image, uses the image to determine the state of an object to be determined captured in the image, and outputs an alarm if the state of the object to be determined indicates an abnormality. The abnormality warning device 100 shown in FIG. 1 is configured to include an abnormality determination device 1000A and an alarm output unit 2000.
[0012] The abnormality determination device 1000A acquires an image and uses the image to output the state of the object to be determined captured in the image. The state of the object to be determined is output data including values in the form of, for example, a determination value or a probability value. The abnormality determination device 1000A includes an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, and an output storage unit 1050.
[0013] The image acquisition unit 1010 acquires and outputs images. The image acquisition unit 1010 acquires images by inputting images (moving images or still images) captured by, for example, a camera. The camera is, for example, an imaging device installed in the interior of a vehicle, which captures an image of a determination target present in the interior of the vehicle (the determination target is, for example, a living body such as a passenger including the driver) and outputs the captured image. If the image is a moving image, the camera divides the moving image into still images (frames) at regular time intervals, and the images are input to the image acquisition unit 1010.
[0014] The neural network unit 1020 is configured by a neural network. The neural network unit 1020 has multiple layers in a neural network, acquires an image output by the image acquisition unit 1010, and outputs output data that is a determination result indicating the state of a determination target included in the image. The multiple layers in the neural network are an input layer, a hidden layer (intermediate layer), and an output layer, and more specifically, layers such as a convolutional layer and a pooling layer. Here, a neural network with many layers (deep) is particularly called a DNN (Deep Neural Network), and derivatives thereof include, for example, a CNN (Convolutional Neural Network) and an RNN (Recurrent Neural Network). CNNs are often used in the fields of object recognition and image recognition, while RNNs are often used in time series processing, speech recognition, natural language processing, and the like. The neural network in the present disclosure is applicable regardless of the form of the neural network, and in this description, the form of CNN will be described as an example.
[0015] The state comparison unit 1030 determines a change in the state of the object to be determined. The state comparison unit 1030 stores in advance a reference value that is a reference for input values to a first layer, which is at least one of the multiple layers of the neural network unit 1020, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value, which is a value that is about to be newly input to the first layer. The first layer may be a predetermined one of all layers in the neural network unit 1020, a predetermined portion of all layers, or all layers.
[0016] The reference value is a previous value, which is an input value of at least one layer, the first layer, stored as a result of the previous processing by the multiple layers of the neural network unit 1020, and is used to obtain a difference value from the current value, which is an input value that is about to be newly input to the first layer. In the description of the processing that will be described later, the previous value will be used as the reference value. However, the following values may also be stored and used as the reference value.
[0017] The reference value can be a typical processing result that has been learned and modeled in advance. That is, the reference value may be an output value of a typical intermediate layer that has been modeled in advance in a neural network. By using a typical processing result, comparison can be performed without being affected by noise or anomalies contained in the actually observed image.
[0018] Furthermore, the reference value may be a partial result determined by prior information. The partial result determined by prior information may, for example, be the adoption of pixels with a high probability of a head being present. Alternatively, it may be the rejection of pixels that are known to have a large change but low importance, such as background pixels. In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for memory savings and a reduction in the amount of data handled, resulting in faster processing.
[0019] The state comparison unit 1030 determines whether the state of the object to be determined has changed based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value. Specifically, the state comparison unit 1030 determines that the state of the object to be determined has not changed if, for example, the sum of the squares of the difference values is smaller than the pre-stored threshold value for each image. Also specifically, the state comparison unit 1030 determines that the state of the object to be determined has not changed if, for example, the sum of the absolute values of the difference values is smaller than the pre-stored threshold value for each image.
[0020] The processing control unit 1040 issues a command to restrict the current processing operation in the neural network unit 1020 in response to a change in the state of the subject of evaluation, and also issues a command to output the output data output by the previous processing of the neural network unit 1020. When the state comparison unit 1030 determines that there is no change in the state of the subject of evaluation (no change includes a case where there is little change), the processing control unit 1040 instructs the neural network unit 1020 not to execute processing in at least one layer of the multiple layers, i.e., the first layer or subsequent layers, and also instructs the neural network unit 1020 to output the output data stored in the output storage unit 1050. The output data is, for example, a judgment value that indicates the state of the subject of evaluation. In the embodiments of the present disclosure, the judgment value is also referred to as a probability value that includes a probabilistic element.
[0021] The output storage unit 1050 stores the output data (determination values (probability values)) output by the neural network unit 1020. When the output storage unit 1050 receives a command from the processing control unit 1040, it outputs the stored output data. For example, the output storage unit 1050 outputs the output data to the alarm output unit 2000. The output storage unit 1050 holds the latest output data output as a result of processing by the neural network unit 1020. For example, every time the output storage unit 1050 obtains output data output by the neural network unit 1020, it erases the previously stored output data and stores the latest output data.
[0022] The alarm output unit 2000 acquires output data and outputs an alarm signal based on the output data to an alarm device (not shown) etc. The alarm output unit 2000 acquires output data that is a determination value indicating the drowsy state or dozing state of the person to be evaluated, and outputs an alarm to the person to be evaluated according to the determination value.
[0023] Although the abnormality determination device 1000A shown in Fig. 1 is shown as having a configuration that does not include the alarm output unit 2000, it may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000A is equivalent to the abnormality warning device 100A shown in Fig. 14. In the following description, the abnormality determination device 1000A will be described as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100A and the abnormality determination device 1000A.
[0024] In addition to the above components, the abnormality determination device 1000A may also include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire abnormality determination device 1000A and each of its components. For example, the control unit (not shown) starts up the abnormality determination device 1000A in response to an external command. The control unit (not shown) also controls the state (operating state, such as startup, shutdown, or sleep) of the abnormality determination device 1000A. The storage unit (not shown) stores various data used in the abnormality determination device 1000A. For example, the storage unit (not shown) stores output (output data) from each component of the abnormality determination device 1000A and outputs data requested by each component to the requesting component. The communication unit (not shown) communicates with external devices. For example, the communication unit (not shown) communicates between the abnormality determination device 1000A and an imaging device such as an in-vehicle camera. Furthermore, for example, if the abnormality determination device 1000A does not include a display unit or an audio output unit, communication is performed between the abnormality determination device 1000A and an external device such as a display unit or an audio output unit.
[0025] An example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to embodiment 1 will be described. Fig. 2 is a flowchart showing an example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to embodiment 1 of the present disclosure. When an image is input from a camera, for example, the abnormality determination device 1000A starts the processing shown in Fig. 2.
[0026] The abnormality determination device 1000A executes an image acquisition process (step ST100). In the image acquisition process, the image acquisition unit 1010 of the abnormality determination device 1000A acquires and outputs an image.
[0027] The abnormality determination device 1000A executes a state storage and state comparison process (step ST200). In the state storage and state comparison process, the state comparison unit 1030 of the abnormality determination device 1000A stores a previous value, which is an input value of at least one of the first layers, among the processing results of all layers of the neural network unit 1020 at least once after the start of processing. The state comparison unit 1030 does not perform a state comparison process for the first processing and only stores the previous value, but determines a change in the state of the object to be determined from the second processing onwards. The state comparison unit 1030 pre-stores a reference value that is a reference for the input value to the first layer, which is at least one of the multiple layers of the neural network unit 1020, and determines a change in the state of the object to be determined using the difference between the reference value and a current value, which is a value that is about to be newly input to the first layer.
[0028] Abnormality determination device 1000A executes a process control process (step ST300). In the process control process, process control unit 1040 of abnormality determination device 1000A issues a command to limit the current processing operation in neural network unit 1020 in response to a change in the state of the object to be determined, and also issues a command to output the output data output by the previous processing of neural network unit 1020. Specifically, when state comparison unit 1030 determines that the state of the object to be determined has not changed (no change includes a case where there has been little change), process control unit 1040 instructs neural network unit 1020 not to execute processing in at least one layer of the multiple layers, i.e., the first layer and subsequent layers, and also instructs neural network unit 1020 to output the output data stored in output storage unit 1050.
[0029] Abnormality determination device 1000A executes a state output process (step ST400). In the state output process, neural network unit 1020 or output storage unit 1050 of abnormality determination device 1000A outputs output data. When processing has been performed in all layers, neural network unit 1020 outputs the output data to alarm output unit 2000. Upon receiving a command from process control unit 1040, output storage unit 1050 outputs the latest output data stored therein to alarm output unit 2000.
[0030] Abnormality determination device 1000A executes an alarm output process (step ST500). In the alarm output process, alarm output unit 2000 of abnormality determination device 1000A acquires output data and outputs an alarm signal to an alarm device (not shown) or the like based on the output data. Alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and if it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) or the like.
[0031] 2 and repeats the process from step ST100. When the camera is turned off, for example, the abnormality determination device 1000A is also turned off.
[0032] A detailed example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to embodiment 1 will be described. Fig. 3 is a flowchart showing a more detailed example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to embodiment 1 of the present disclosure. When the abnormality determination device 1000A starts the processing shown in Fig. 3, first, it executes an image acquisition process (step ST100) similar to step ST100 described above.
[0033] The abnormality determination device 1000A executes a storage process (step ST201). In the storage process, the state comparison unit 1030 of the abnormality determination device 1000A stores a previous value, which is an input value of at least one layer, that is, a first layer, among the processing results of all layers of the neural network unit 1020 at least for the first time since the start of processing. Furthermore, the state comparison unit 1030 stores the input value of the first layer every time processing is performed in the first layer.
[0034] The state comparison unit 1030 of the abnormality determination device 1000A executes a previous storage determination process to determine whether the data has been stored previously (step ST202).
[0035] The state comparison unit 1030 of the abnormality determination device 1000A executes a comparison process between the current value and the previous value (step ST203). The state comparison unit 1030 calculates the difference (differential value) between the current value, which is a value to be newly input to the first layer, and the previous value (reference value).
[0036] The state comparison unit 1030 of the abnormality determination device 1000A determines a change in the state of the object to be determined using the difference value (step ST204). The state comparison unit 1030 determines the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value, and determines the change in the state of the object to be determined based on the result. Specifically, the state comparison unit 1030 determines that the state of the object to be determined has not changed if, for example, the sum of the squares of the difference values is smaller than the pre-stored threshold value for each image. Also specifically, the state comparison unit 1030 determines that the state of the object to be determined has not changed if, for example, the sum of the absolute values of the difference values is smaller than the pre-stored threshold value for each image.
[0037] When the state comparison unit 1030 of the abnormality determination device 1000A determines that the difference is equal to or greater than the threshold value and that the state of the object to be determined has changed (step ST204 "NO"), the process control unit 1040 of the abnormality determination device 1000A executes a normal process command process (step ST301). The process control unit 1040 outputs a command to execute processing in the first layer of the neural network unit 1020.
[0038] The neural network unit 1020 of the abnormality determination device 1000A executes an output process (step ST401). The neural network unit 1020 outputs output data to the alarm output unit 2000.
[0039] The output storage unit 1050 of the abnormality determination device 1000A executes an output storage process (step ST402). The output storage unit 1050 stores the output data output by the neural network unit 1020.
[0040] If the state comparison unit 1030 of the abnormality determination device 1000A determines that the difference is less than the threshold value and that the state of the object to be determined has not changed ("YES" in step ST204), the process control unit 1040 of the abnormality determination device 1000A executes an omission process command process (step ST302). In the omission process command process, the process control unit 1040 commands the neural network unit 1020 not to execute the process in the first layer or subsequent layers, which is at least one layer out of the multiple layers, and commands the output storage unit 1050 to output the output data stored therein.
[0041] The output storage unit 1050 of the abnormality determination device 1000A executes a stored data output process (step ST403). When the output storage unit 1050 receives a command from the process control unit 1040, it outputs the stored output data to the alarm output unit 2000.
[0042] The alarm output unit 2000 executes an alarm output process (step ST500). As in the process of step ST500 already described, the alarm output unit 2000 determines whether to output an alarm based on the determination value included in the output data, and if it determines to output an alarm, outputs an alarm signal to an alarm device or the like (not shown).
[0043] 3, and then repeats the process from step ST100. When the camera is turned off, for example, the abnormality determination device 1000A is also turned off.
[0044] The configuration and processing described above allows the abnormality determination device to omit processing of layers in the neural network depending on the conditions.
[0045] The abnormality determination device of the present disclosure has the following configuration. 」。 English: "An abnormality determination device comprising: an image acquisition unit that acquires and outputs an image; a neural network unit that has a plurality of layers in a neural network, that acquires the image output by the image acquisition unit, and outputs a determination result that indicates the state of an object to be determined included in the image; an output storage unit that stores output data (determination values (probability values)) output by the neural network unit; a state comparison unit that pre-stores a reference value that is a reference for an input value to a first layer that is at least one of the plurality of layers of the neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer; and a processing control unit that, when the state comparison unit determines that the state of the object to be determined has not changed, instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs it to output the output data stored in the output storage unit." Thus, the present disclosure has an effect of being able to provide an abnormality determination device that enables, in an abnormality determination technology, to quickly output a highly accurate determination result using a neural network.
[0046] The abnormality determination method of the present disclosure has the following configuration. "An abnormal state determination method, in which an image acquisition unit outputs the acquired image to a neural network unit; the neural network unit, which has a plurality of layers in a neural network, acquires the image output by the image acquisition unit and outputs a determination result indicating the state of an object to be determined contained in the image; an output storage unit stores output data (determination value (probability value)) output by the neural network unit; a state comparison unit pre-stores a reference value that is a reference for an input value to a first layer, which is at least one of the plurality of layers of the neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value, which is a value that is to be newly input to the first layer; and when the state comparison unit determines that the state of the object to be determined has not changed, a processing control unit instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs it to output the output data stored in the output storage unit." As a result, the present disclosure has an effect of providing an abnormality determination method that enables a highly accurate determination result to be output quickly using a neural network.
[0047] The anomaly determination device of the present disclosure is further configured as follows: "The anomaly determination device, wherein the reference value is a previous value stored for each layer as a result of the previous processing in the neural network." This provides an effect of providing an anomaly determination device that can quickly output a highly accurate determination result using a neural network in an anomaly determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the anomaly determination method.
[0048] The abnormality determination device of the present disclosure is further configured as follows: "The reference value is an output value of a typical intermediate layer that is modeled in advance in the neural network." This provides an effect of providing an abnormality determination device that can quickly output highly accurate determination results using a neural network in an abnormality determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the abnormality determination method.
[0049] The abnormality determination device of the present disclosure is further configured as follows: "The reference value is an output value of a predetermined node among the nodes included in the neural network." This provides an effect of providing an abnormality determination device that can quickly output highly accurate determination results using a neural network in an abnormality determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the abnormality determination method.
[0050] The abnormality determination device of the present disclosure is further configured as follows: "The abnormality determination device is characterized in that the state comparison unit determines the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value to determine a change in the state of the object to be determined based on the result of the determination." This provides an effect of providing an abnormality determination device that can quickly output highly accurate determination results using a neural network in an abnormality determination technique. Furthermore, the present disclosure provides the same effect as the above effect by applying the above configuration to the above abnormality determination method.
[0051] The anomaly determination device of the present disclosure is further configured as follows: "The anomaly determination device, wherein the state comparison unit determines that the state of the object to be determined has not changed if the sum of squares of the difference values for each image is smaller than a pre-stored threshold value." This provides an effect of providing an anomaly determination device that can quickly output highly accurate determination results using a neural network in an anomaly determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the anomaly determination method.
[0052] The anomaly determination device of the present disclosure is further configured as follows: "The anomaly determination device, wherein the state comparison unit determines that the state of the object to be determined has not changed if the sum of the absolute values of the difference values for each image is smaller than a pre-stored threshold value." This provides an effect of providing an anomaly determination device that can quickly output highly accurate determination results using a neural network in an anomaly determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the anomaly determination method.
[0053] The abnormality determination device of the present disclosure is further configured as follows: "The abnormality determination device further includes an alarm output unit that acquires the output data as a determination value indicating a drowsy state or a dozing state of a person to be determined, and outputs an alarm to the person to be determined in accordance with the determination value." This provides an effect of providing an abnormality determination device that can quickly output highly accurate determination results using a neural network in an abnormality determination technique. Furthermore, the present disclosure provides the same effect as the above effect by applying the above configuration to the abnormality determination method.
[0054] Second Embodiment In a second embodiment, a form will be described in which the basic mechanism of the present disclosure is applied to layers in the neural networks of a feature extraction unit and an abnormal state determination unit.
[0055] An example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to embodiment 2 will be described. Fig. 4 is a diagram showing an example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to embodiment 2 of the present disclosure. The abnormality warning device 100B is configured to include an abnormality determination device 1000B and an alarm output unit 2000.
[0056] The abnormality determination device 1000B acquires an image and uses the image to output the state of the object to be determined captured in the image. The abnormality determination device 1000B shown in FIG. 4 includes an image acquisition unit 1100B, a feature extraction unit 1300B, and an abnormal state determination unit 1500B. In the abnormality determination device 1000B, the previously described neural network unit includes a first neural network unit 1320 and a second neural network unit 1520, as will be described later. The first neural network unit 1320 is included in the feature extraction unit 1300B, and the second neural network unit 1520 is included in the abnormal state determination unit 1500B. Furthermore, in the abnormality determination device 1000B, the previously described output storage unit includes a first output storage unit 1390 and a second output storage unit. The first output storage unit 1390 is included in the feature extraction unit 1300B, and the second output storage unit is included in the abnormal state determination unit 1500B. In addition, in the abnormality determination device 1000B, the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540. The first state comparison unit 1340 is included in the feature extraction unit 1300B, and the second state comparison unit 1540 is included in the abnormal state determination unit 1500B. In addition, in the abnormality determination device 1000B, the process control unit already described is configured to include a first process control unit 1380 and a second process control unit 1580. The first process control unit 1380 is included in the feature extraction unit 1300B, and the second process control unit 1580 is included in the abnormal state determination unit 1500B.
[0057] The image acquisition unit 1100B is similar to the image acquisition unit 1100A already described, and therefore a detailed description of the image acquisition unit 1100B will be omitted here.
[0058] The feature extraction unit 1300B uses the image to output a feature map that represents a characteristic state of the subject included in the image. The feature extraction unit 1300B extracts, for example, parts of the image that show signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map that represents a state of the subject that is characteristic of an abnormal state such as drowsiness.
[0059] An example of the internal configuration of the feature extraction unit 1300B will be described. Fig. 5 is a diagram showing an example of the internal configuration of the feature extraction unit 1300B in the abnormality warning device 100B and the abnormality determination device 1000B. The feature extraction unit 1300B shown in Fig. 5 is configured to include a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340B), a processing control unit (first processing control unit 1380B), and an output storage unit (first output storage unit 1390).
[0060] The first neural network unit 1320 has a plurality of layers that are part of the layers that make up the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map that represents the characteristic state of the object to be determined contained in the image. The first neural network unit 1320 shown in FIG. 5 is configured to include an image branching unit 1321, a convolution layer unit 1322, a pooling layer unit 1325, and an image combining unit 1328.
[0061] The image branching unit 1321 branches an image into multiple nodes in a layer of a neural network and inputs the image. The image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 5, the image is branched into two.
[0062] The convolution layer unit 1322 performs filtering to extract characteristic parts of the target object in the image. The convolution layer unit 1322 shown in FIG. 5 includes a first convolution layer 1323 and a second convolution layer 1324. The first convolution layer 1323 and the second convolution layer 1324 perform filtering to extract face (body) parts for detecting drowsy driving, for example, by convolution processing (cross-correlation processing) using a pre-prepared convolution filter of 3×3 or 5×5 size.
[0063] 5 includes a first pooling layer 1326 and a second pooling layer 1327. The first pooling layer 1326 and the second pooling layer 1327 generate an image relating to feature quantities that are robust against image position by, for example, calculating a maximum or average value for each predetermined region.
[0064] The first neural network unit 1320 shown in Figure 5 includes two sets of convolutional layers and pooling layers, but it is also effective to use one set or three or more sets. Furthermore, the first pooling layer and the second pooling layer 1327 may be followed by two or more convolutional layers and two or more pooling layers. Furthermore, a normalized linear unit layer (activation function) or the like (not shown) may be included after the convolutional layers.
[0065] The image combining unit 1328 combines the multiple images output from the convolution layer and the pooling layer, and generates a feature map by extracting areas that indicate signs of drowsiness, such as the eyelids and eyeballs.
[0066] The first state comparison unit 1340B stores in advance, for each of the multiple layers in the first neural network unit 1320, a reference value that is the standard for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and the current value, which is the value that is about to be newly input to that layer.
[0067] The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain a difference value from the current value, which is an input value that is about to be newly input to that layer. In the description of the processing described below, a case will be described in which the previous value is used as the reference value. In this case, the first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) but simply stores the value, but determines changes in the state of the object to be determined for the second and subsequent processing (processing for the second image). However, the reference value may be stored and used as follows:
[0068] The reference value can be a typical processing result that has been learned and modeled in advance. That is, the reference value may be an output value of a typical intermediate layer that has been modeled in advance in a neural network. By using a typical processing result, comparison can be performed without being affected by noise or anomalies contained in the actually observed image.
[0069] Furthermore, the reference value may be a partial result determined by prior information. The partial result determined by prior information may, for example, be the adoption of pixels with a high probability of a head being present. Alternatively, it may be the rejection of pixels that are known to have a large change but low importance, such as background pixels. In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for memory savings and a reduction in the amount of data handled, resulting in faster processing.
[0070] The first state comparison unit 1340 determines whether the state of the object to be determined has changed based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value. Specifically, the first state comparison unit 1340 determines that the state of the object to be determined has not changed if the sum of the squares of the difference values for each image is smaller than the pre-stored threshold value. Also specifically, the first state comparison unit 1340 determines that the state of the object to be determined has not changed if the sum of the absolute values of the difference values for each image is smaller than the pre-stored threshold value.
[0071] The first state comparison unit 1340B shown in Fig. 5 includes an image state comparison unit 1341. In the image state comparison unit 1341, "storing" means holding the values of the two-dimensional images (feature maps) of the respective input sources. In addition, in the image state comparison unit 1341, "comparison" means, for example, calculating the absolute value of the difference for each element (pixel) between the stored two-dimensional image (of the previous time / previous frame) and the latest two-dimensional image, and then comparing the sum or average with a predetermined threshold value.
[0072] The image state comparison unit 1341 shown in FIG. 5 includes a storage unit 1341a and a comparison processing unit 1341b.
[0073] When the first state comparison unit 1340 determines that the state of the object to be determined has not changed, the first process control unit 1380B instructs the first neural network unit 1320 not to execute processing in the second layer and subsequent layers of the object to be determined, and also instructs the first output storage unit 1390 to output the feature map stored in the first output storage unit 1390. The first process control unit 1380B shown in FIG. 15 is configured to include a feature extraction process control unit 1381B. The feature extraction process control unit 1381B executes the function of the first process control unit 1380B in the feature extraction unit 1300.
[0074] The first output storage unit 1390 stores output data (determination values (probability values)) output by the first neural network unit 1320. The first output storage unit 1390 stores the feature map output by the first neural network unit 1320. The first output storage unit 1390 shown in FIG. 5 is configured to include a combined image storage unit 1391. The combined image storage unit 1391 stores the feature map, which is the combined image combined and output by the first neural network unit 1320.
[0075] An example of the internal configuration of the abnormal condition determination unit 1500B will be described. FIG. 6 is a diagram showing an example of the internal configuration of the abnormal condition determination unit 1500B in the abnormality warning device 100B and the abnormality determination device 1000B. The abnormal condition determination unit 1500B outputs a determination value indicating the state of the object to be determined using a feature map, which is a two-dimensional image. The abnormal condition determination unit 1500B shown in FIG. 6 is configured to include a neural network unit (second neural network unit 1520), a state comparison unit (second state comparison unit 1540), and an output storage unit (second output storage unit) 1590.
[0076] The neural network unit (second neural network unit 1520) has multiple layers in a neural network, acquires the feature map output by the feature extraction unit 1300, and uses the feature map to output a judgment value indicating the state of the object to be judged as output data. The neural network unit (second neural network unit 1520) shown in FIG. 6 is configured to include a state classification unit 1525 and a probability output layer 1529.
[0077] The state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector and further summarizing the output into an indication of the drowsy state (for example, four outputs: eyelids: dozing / not dozing, eyeballs: dozing / not dozing). The state classification unit 1525 shown in FIG. 6 is configured to include a first fully connected layer 1527 and a second fully connected layer 1528. The state classification unit 1525 generates a one-dimensional vector whose number of elements is the desired number of outputs through the first fully connected layer 1527 and the second fully connected layer 1528.
[0078] The probability output layer 1529 applies, for example, a softmax function and sets the sum of the output values to 1.0, thereby giving the output results a probabilistic meaning.
[0079] The second neural network unit 1520 shown in FIG. 6 has two fully connected layers, but may have three or more fully connected layers, or may have no fully connected layers.
[0080] The second state comparison unit 1540B pre-stores a reference value that is a standard for the input value to the third layer, which is at least one of the layers in the second neural network unit 1520, and determines a change in the state of the object to be determined using the difference between the reference value and the current value, which is the value that is about to be newly input to the third layer.
[0081] The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the second neural network unit 1520, and is used to obtain a difference value from the current value, which is an input value that is about to be newly input to the first layer. In the description of the processing that follows, we will explain how the previous value is used as the reference value. In this case, the second state comparison unit 1540B does not perform state comparison processing for the first processing (processing for the first image) but simply stores the value, but determines changes in the state of the object to be determined for the second and subsequent processing (processing for the second image). However, the reference value may be stored and used as follows:
[0082] The reference value can be a typical processing result that has been learned and modeled in advance. That is, the reference value may be an output value of a typical intermediate layer that has been modeled in advance in a neural network. By using a typical processing result, comparison can be performed without being affected by noise or anomalies contained in the actually observed image.
[0083] Furthermore, the reference value may be a partial result determined by prior information. The partial result determined by prior information may, for example, be the adoption of pixels with a high probability of a head being present. Alternatively, it may be the rejection of pixels that are known to have a large change but low importance, such as background pixels. In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for memory savings and a reduction in the amount of data handled, resulting in faster processing.
[0084] The second state comparison unit 1540B determines a change in the state of the object to be determined based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value. Specifically, the second state comparison unit 1540B determines that the state of the object to be determined has not changed if the sum of the squares of the difference values in units of images is smaller than the pre-stored threshold value. Also, specifically, the second state comparison unit 1540B determines that the state of the object to be determined has not changed if the sum of the absolute values of the difference values in units of images is smaller than the pre-stored threshold value.
[0085] The second state comparison unit 1540B shown in Figure 6 is configured to include a first fully connected layer state comparison unit 1541. In the first fully connected layer state comparison unit 1541, "storing" means retaining the values of the one-dimensional vectors of each input source. In addition, in the first fully connected layer state comparison unit 1541, "comparison" means, for example, taking the absolute value of the difference for each element between the stored one-dimensional vector (of the previous time / previous frame) and the latest one-dimensional vector, and then comparing the sum or average with a predetermined threshold value.
[0086] 6 includes a storage unit 1541a and a comparison processing unit 1541b. The storage unit 1541a stores a reference value, which is an output value of the first fully connected layer 1527 and an input value of the second fully connected layer 1528, for each output from the first fully connected layer 1527. The comparison processing unit 1541b compares the reference value with the current value.
[0087] When the second state comparison unit 1540 determines that the state of the object to be determined has not changed, the second process control unit 1580B instructs the second neural network unit 1520 not to execute processing in the layer of the object to be determined and subsequent layers, and also instructs the second process control unit 1580B to output the determination value stored in the second output storage unit as output data. The second process control unit 1580B shown in Figure 6 is configured to include an abnormal state determination process control unit 1581B. The abnormal state determination process control unit 1581B functions to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500B.
[0088] The second output storage unit 1590 stores the judgment value output by the second neural network unit 1520. The output storage unit (second output storage unit) 1590 shown in FIG.
[0089] The probability storage unit stores the judgment value output by the second neural network unit 1520. The judgment value is a value indicating the state of the person to be judged, and is, for example, a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.
[0090] Returning to the description of Fig. 4, the warning output unit 2000 acquires output data that is a determination value indicating the drowsy state or dozing state of the person to be determined, and outputs a warning to the person to be determined according to the determination value.
[0091] Although abnormality determination device 1000B shown in Fig. 4 is shown as having a configuration that does not include alarm output unit 2000, it may be configured to include alarm output unit 2000. When configured in this manner, abnormality determination device 1000B is equivalent to abnormality warning device 100B shown in Fig. 4. In the following description, the abnormality determination device 1000B will be described as being configured to include alarm output unit 2000, except in cases where it is necessary to distinguish between abnormality warning device 100B and abnormality determination device 1000B.
[0092] In addition to the above components, the abnormality determination device 1000B may also include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire abnormality determination device 1000B and each of its components. For example, the control unit (not shown) activates the abnormality determination device 1000A in accordance with an external command. The control unit (not shown) also controls the state (operating state, such as startup, shutdown, or sleep) of the abnormality determination device 1000B. The storage unit (not shown) stores data used by the abnormality determination device 1000B. For example, the storage unit (not shown) stores output (output data) from each component of the abnormality determination device 1000B and outputs data requested by each component to the requesting component. The communication unit (not shown) communicates with external devices. For example, the communication unit (not shown) communicates between the abnormality determination device 1000B and an imaging device such as an in-vehicle camera. Furthermore, for example, if the abnormality determination device 1000B does not include a display unit or an audio output unit, communication is performed between the abnormality determination device 1000B and an external device such as a display unit or an audio output unit.
[0093] An example of the processing of the abnormality warning device 100B and the abnormality determination device 1000B according to embodiment 2 will be described. Fig. 7 is a flowchart showing an example of the processing of the abnormality warning device 100B and the abnormality determination device 1000B according to embodiment 2 of the present disclosure. When an image is input from a camera, for example, the abnormality determination device 1000B starts the processing shown in Fig. 7.
[0094] Abnormality determination device 1000B executes image acquisition processing (step ST2100). In the image acquisition processing, image acquisition unit 1100 of abnormality determination device 1000B acquires and outputs an image.
[0095] Abnormality determination device 1000B executes storage and state comparison processing (step ST2200). In the state storage and state comparison processing, first state comparison unit 1340B of abnormality determination device 1000B stores, for each layer of first neural network unit 1320 at least for the first time after processing starts, a previous value (reference value), which is an input value to the layer. First state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value (previous value = reference value), but determines a change in the state of the object to be determined for the second and subsequent processing (processing for the second image).
[0096] The abnormality determination device 1000B executes a feature extraction process control process (step ST2300). The feature extraction process control unit 1381 of the abnormality determination device 1000B issues a normal process command or an omission process command to the first neural network unit 1320, depending on the determination result by the first state comparison unit 1340. When issuing an omission process command, the feature extraction process control unit 1381 issues an output command to the combined image storage unit 1391.
[0097] Abnormality determination device 1000B executes a combined image output process (step ST2400). First neural network unit 1320 executes normal processing to output a combined image, or combined image storage unit 1391 outputs the previous combined image. This causes feature extraction unit 1300 to output the combined image.
[0098] The abnormality determination device 1000B executes a fully connected layer state storage process and a fully connected layer state comparison process (step ST2500). The second state comparison unit 1540 in the abnormality determination device 1000B pre-stores, for each of the multiple layers of the second neural network unit 1520, a reference value that is a reference for the input value to that layer, and determines a change in the state of the object to be determined using the difference between the reference value and the current value, which is the value that is about to be newly input to that layer. The second state comparison unit 1540 does not perform a state comparison process for the first processing (processing for the first image) but simply stores the value. However, it determines a change in the state of the object to be determined for the second and subsequent processing (processing for the second image).
[0099] Abnormality determination device 1000B executes abnormal state determination process control process (step ST2600). Abnormal state determination process control unit 1581B issues a normal command or an omission command to second neural network unit 1520. When issuing an omission command, abnormal state determination process control unit 1581B issues an output command to probability storage unit 1591.
[0100] Abnormality determination device 1000B executes a result output process (step ST2700). Second neural network unit 1520 executes normal processing and outputs a determination value, or probability storage unit 1591 outputs the previous determination value (probability value). As a result, abnormal state determination unit 1500 outputs the determination value as output data.
[0101] Abnormality determination device 1000B executes alarm output processing (step ST2800). In the alarm output processing, alarm output unit 2000 of abnormality determination device 1000B acquires output data and outputs an alarm signal to an alarm device (not shown) or the like based on the output data. Alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and if it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) or the like.
[0102] 7, and then repeats the process from step ST3100. Note that abnormality determination device 1000B also becomes power-off in conjunction with, for example, the camera being powered off.
[0103] Here, a detailed example of the processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment will be described. FIG. 8 is a flowchart illustrating a more detailed example of the processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure. The flowchart in FIG. 8 illustrates an example of processing corresponding to the processing of steps ST2200 to ST2700 in the flowchart in FIG. 7. When the feature extraction unit 1300B in the abnormality determination device 1000B starts the processing of step ST2200, the feature extraction unit 1300B first executes a storage process (step ST2201). In the storage process, the state comparison unit 1340 of the feature extraction unit 1300B stores a previous value, which is an input value of at least one of all layers of the neural network unit 1320 (first neural network unit 1320). Furthermore, the state comparison unit 1340 stores the input value of the first layer each time processing is performed in the first layer.
[0104] Next, feature extraction unit 1300B in abnormality determination device 1000B executes a previous storage determination process (step ST2202). In the previous storage determination process, state comparison unit 1340 of feature extraction unit 1300B determines whether the previous input value (previous value=reference value) is stored.
[0105] Feature extraction unit 1300B in abnormality determination device 1000B executes a comparison process (step ST2203). In the comparison process, state comparison unit 1340 of feature extraction unit 1300B calculates the difference (differential value) between a current value, which is a value to be newly input to the first layer, and a previous value (reference value).
[0106] Feature extraction unit 1300B in abnormality determination device 1000B executes difference determination processing (step ST2204). In the difference determination processing, state comparison unit 1340 of feature extraction unit 1300B determines a change in the state of the object to be determined using the difference value. Specifically, state comparison unit 1340 determines that the state of the object to be determined has not changed if, for example, the sum of the squares of the difference values is smaller than a pre-stored threshold value for each image. Also specifically, state comparison unit 1340 determines that the state of the object to be determined has not changed if, for example, the sum of the absolute values of the difference values is smaller than a pre-stored threshold value for each image.
[0107] If the state comparison unit 1340 of the feature extraction unit 1300B determines that the data was not previously stored ("NO" in step ST2202), or if the state comparison unit 1340 determines that the difference is equal to or greater than the threshold value and that the state of the object to be determined has changed ("NO" in step ST2204), the feature extraction unit 1300B in the abnormality determination device 1000B executes a normal processing command (step ST2301). In response to the normal processing command, the processing control unit 1380 of the feature extraction unit 1300B outputs a command to execute processing in the first layer of the neural network unit 1320 (first neural network unit 1320).
[0108] Next, feature extraction unit 1300B in abnormality determination device 1000B executes output processing (step ST2401). In the output processing, first neural network unit 1320 of feature extraction unit 1300B executes processing in accordance with the normal processing command and outputs output data indicating the determination result.
[0109] Next, feature extraction unit 1300B in abnormality determination device 1000B executes output storage processing (step ST2402). In the output storage processing, output storage unit 1390 (first output storage unit 1390) of feature extraction unit 1300B stores the output data output by neural network unit 1320.
[0110] If the state comparison unit 1340 determines that the difference is less than the threshold value and that the state of the object to be determined has not changed ("YES" in step ST2204), the feature extraction unit 1300B in the abnormality determination device 1000B executes an omission processing command (step ST2302). In the omission processing command, the processing control unit 1380 of the feature extraction unit 1300B instructs the neural network unit 1320 not to execute processing in the first layer or subsequent layers, which is at least one layer among the multiple layers, and also instructs the output storage unit 1390 to output the output data stored therein.
[0111] Feature extraction unit 1300B in abnormality determination device 1000B executes stored data output processing (step ST2403). In the stored data output processing, upon receiving a command from process control unit 1380, output storage unit 1390 (combined image storage unit 1391 of output storage unit 1390) of feature extraction unit 1300B outputs the stored combined image, which is output data, to alarm output unit 2000.
[0112] The abnormal state determination unit 1500B in the abnormality determination device 1000B executes a fully connected layer output storage process (step ST2501). In the fully connected layer output storage process, the state comparison unit 1540 of the abnormal state determination unit 1500B stores the value output from the fully connected layer (first fully connected layer 1527 or second fully connected layer 1528) in the state classification unit 1525 as the state of the fully connected layer.
[0113] Abnormal state determination unit 1500B in abnormality determination device 1000B executes a previous storage determination process (step ST2502). In the previous storage determination process, state comparison unit 1540 of abnormal state determination unit 1500B determines whether the previous input value (previous value=reference value) is stored.
[0114] When the abnormal state determination unit 1500B in the abnormality determination device 1000B determines that the previous value has been stored ("YES" in step ST2502), it executes a comparison process (step ST2503). In the comparison process, the state comparison unit 1540 of the abnormal state determination unit 1500B calculates the difference (differential value) between the current value, which is a value to be newly input to the first layer (the second fully connected layer 1528 or the probability output layer 1529), and the previous value (reference value).
[0115] Abnormal state determination unit 1500B in abnormality determination device 1000B executes a difference determination process (step ST2504). In the difference determination process, state comparison unit 1540 of abnormal state determination unit 1500B determines the magnitude of the difference between the reference value and the current value using a difference between the current value and the previous value (reference value) and a pre-stored threshold value.
[0116] If the state comparison unit 1540 of the abnormal state determination unit 1500B determines that the difference is equal to or greater than the threshold value ("NO" in step ST2504), the abnormal state determination unit 1500B in the abnormality determination device 1000B executes a normal processing command (step ST2601). In response to the normal processing command, the processing control unit 1580 of the abnormal state determination unit 1500B executes a normal processing command to the neural network unit 1520 (second neural network unit 1520).
[0117] Abnormal state determination unit 1500B in abnormality determination device 1000B executes output processing (step ST2701). In the output processing, neural network unit 1520 (second neural network unit 1520) of abnormal state determination unit 1500B outputs a determination value as output data.
[0118] Abnormal state determination unit 1500B in abnormality determination device 1000B executes output storage processing (step ST2702). In the output storage processing, output storage unit 1590 (second output storage unit 1590) in abnormal state determination unit 1500B stores the determination value output by neural network unit 1520 (second neural network unit 1520).
[0119] Abnormal state determination unit 1500B in abnormality determination device 1000B executes an omission processing command (step ST2602). In the omission processing command, processing control unit 1580 of abnormal state determination unit 1500B instructs second neural network unit 1520 not to execute processing in layers subsequent to the layer to be determined, and also instructs second output storage unit 1520 to output the determination value stored in the second output storage unit.
[0120] Abnormal state determination unit 1500B in abnormality determination device 1000B executes a stored data output process (step ST2703). In the stored data output process, second output storage unit 1590 of abnormal state determination unit 1500B stores the determination value output by neural network unit 1520 (second neural network unit 1520).
[0121] When second neural network unit 1520 or second output storage unit 1590 outputs the determination value, the series of processes shown in FIG. 21 ends.
[0122] Note that, in the figure, an example configuration is shown in which first state comparison unit 1340B has image state comparison unit 1341 and second state comparison unit 1540B has first fully connected layer state comparison unit 1541. However, the configuration may be configured using one or a combination of the following: image state comparison unit 1341, a convolution layer state comparison unit that determines a state change of the convolution layer (see image state comparison unit 1342 in the embodiment described later), a pooling layer state comparison unit that determines a state change of the pooling layer (see pooling layer state comparison unit 1343 in the embodiment described later), a first fully connected layer state comparison unit (see first fully connected layer state comparison unit 1541 in the embodiment described later), and a second fully connected layer state comparison unit that determines a state change of the second fully connected layer 1528 (see second fully connected layer state comparison unit 1542 in the embodiment described later). For example, when any one of these is adopted, it can be realized by the processing shown in Figures 9, 10, 11, 12, and 13 below.
[0123] A description will be given of an example of processing by the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) is the image state comparison unit 1341. Fig. 9 is a flowchart illustrating an example of processing by the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is the image state comparison unit 1341.
[0124] When feature extraction unit 1300B starts processing, first, first state comparison unit 1340B of feature extraction unit 1300B acquires an image and executes image storage processing (step ST2211). In the image storage processing, image state comparison unit 1341 of first state comparison unit 1340B stores the image that is input data to first neural network unit 1320. Furthermore, every time first neural network unit 1320 acquires an image, image state comparison unit 1341 stores the image (the image before being processed by first neural network unit 1320) in storage unit 1341a.
[0125] The image state comparison unit 1341 executes a process of determining whether or not the image has been stored previously (step ST2212). The image state comparison unit 1341 refers to the storage unit 1341a and determines whether or not the image input previously has been stored.
[0126] When the image state comparison unit 1341 determines that the image has been stored previously ("YES" in step ST2212), it executes a comparison process between the current image and the previous image (step ST2213). The comparison processing unit 1341b of the image state comparison unit 1341 compares the image input this time with the image input previously.
[0127] The comparison processing unit 1341b executes a process of determining whether the difference is less than a threshold value (step ST2214). The comparison processing unit 1341b determines the magnitude of the difference between the reference value (the previously input image) and the current value (the image to be input this time) using the difference value between the currently input image and the previously input image and a pre-stored threshold value.
[0128] If the image state comparison unit 1341 determines that the image was not previously stored (step ST2212, "NO"), or if the image state comparison unit 1341 determines that the difference between the previous value (reference value) and the current value is equal to or greater than the threshold value (step ST2214, "NO"), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2311). The first neural network unit 1320 executes normal processing in accordance with the normal processing command and outputs a combined image (step ST2411). The combined image storage unit 1391, which is the first output storage unit 1390, stores the combined image output by the first neural network unit 1320 (step ST2412).
[0129] If the image state comparison unit 1341 determines that the difference between the previous value (reference value) and the current value is smaller than the threshold value ("YES" in step ST2214), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2312) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2413). In accordance with the omission processing command, the first neural network unit 1320 does not execute processing of subsequent layers in the first neural network unit 1320. The combined image storage unit 1391 outputs the combined image in accordance with the output command.
[0130] Next, an example of processing by the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) is a convolutional layer state comparison unit (see the convolutional layer state comparison unit 1342 in the embodiment described later) will be described. Fig. 10 is a flowchart showing an example of processing by the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is a convolutional layer state comparison unit.
[0131] When the feature extraction unit 1300B starts processing, first, the first state comparison unit 1340B of the feature extraction unit 1300B executes a convolutional layer state storage process (step ST2221). In the convolutional layer state storage process, the convolutional layer state comparison unit of the first state comparison unit 1340B stores an input value (a value before being processed by the pooling layer unit 1325) that is an output value of the convolutional layer unit 1322 and is input data to the pooling layer unit 1325. In addition, every time the convolutional layer unit 1322 outputs an output value, the convolutional layer state comparison unit stores the output value (a value before being processed by the pooling layer unit 1325) in a storage unit.
[0132] The convolution layer state comparison unit executes a process of determining whether a value has been stored previously (step ST2222). The convolution layer state comparison unit refers to the storage unit and determines whether a value input previously (reference value: in this description, the previous value) is stored.
[0133] When the convolutional layer state comparison unit determines that the value has been stored previously ("YES" in step ST2222), it executes a comparison process between the current value and the previous value (step ST2223). The comparison processing unit of the convolutional layer state comparison unit compares the value to be input to the pooling layer this time (current value) with the value input previously (previous value = reference value).
[0134] The comparison processing unit executes a process to determine whether the difference is less than a threshold value (step ST2224). The comparison processing unit determines whether the difference between the reference value (previously input value) and the current value (value to be input) is large or small using the difference between the value to be input this time (current value) and the value input last time (previous value) and a pre-stored threshold value.
[0135] If the convolutional layer state comparison unit determines that the previous value was not stored (step ST2222 "NO"), or if the convolutional layer state comparison unit determines that the difference between the previous value (reference value) and the current value is equal to or greater than the threshold value (step ST2224 "NO"), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321). The first neural network unit 1320 executes normal processing in accordance with the normal processing command and outputs a combined image (step ST2421). The combined image storage unit 1391, which is the first output storage unit 1390, stores the combined image output by the first neural network unit 1320 (step ST2422).
[0136] If the convolution layer state comparison unit determines that the difference between the previous value (reference value) and the current value is smaller than the threshold value ("YES" in step ST2224), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2322) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423). In accordance with the omission processing command, the first neural network unit 1320 does not execute processing of subsequent layers in the first neural network unit 1320. The combined image storage unit 1391 outputs the combined image in accordance with the output command.
[0137] Next, an example of processing by the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) is a pooling layer state comparison unit (see the pooling layer state comparison unit 1343 in the embodiment described later) will be described. Fig. 11 is a flowchart illustrating an example of processing by the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is a pooling layer state comparison unit.
[0138] When the feature extraction unit 1300B starts processing, first, the first state comparison unit 1340B of the feature extraction unit 1300B executes pooling layer state storage processing (step ST2221). In the pooling layer state storage processing, the pooling layer state comparison unit 1340B stores an input value (a value before being processed by the image combination unit 1328) that is an output value of the pooling layer unit 1325 and is input data to the image combination unit 1328. In addition, every time the pooling layer unit 1325 outputs an output value, the pooling layer state comparison unit stores the output value (a value before being processed by the image combination unit 1328) in a storage unit.
[0139] The pooling layer state comparison unit executes a process of determining whether a value has been stored previously (step ST2222). The pooling layer state comparison unit refers to the storage unit and determines whether a value input previously (a reference value: in this description, a previous value) is stored.
[0140] When the pooling layer state comparison unit determines that the image data has been stored previously ("YES" in step ST2222), it executes a comparison process between the current value and the previous value (step ST2223). The comparison processing unit of the pooling layer state comparison unit compares the value to be input to the image combination unit 1328 this time (current value) with the value input previously (previous value).
[0141] The comparison processing unit executes a process to determine whether the difference is less than a threshold value (step ST2224). The comparison processing unit determines whether the difference between the reference value (previously input value) and the current value (value to be input) is large or small using the difference between the value to be input this time (current value) and the value input last time (previous value) and a pre-stored threshold value.
[0142] If the pooling layer state comparison unit determines that no image data was stored previously (step ST2222 "NO"), or if the pooling layer state comparison unit determines that the difference between the previous value (reference value) and the current value is equal to or greater than the threshold value (step ST2224 "NO"), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321). The first neural network unit 1320 executes normal processing in accordance with the normal processing command and outputs a combined image (step ST2421). The combined image storage unit 1391, which is the first output storage unit 1390, stores the combined image output by the first neural network unit 1320 (step ST2422).
[0143] If the pooling layer state comparison unit determines that the difference between the previous value (reference value) and the current value is smaller than the threshold value ("YES" in step ST2224), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2322) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423). In accordance with the omission processing command, the first neural network unit 1320 does not execute processing of subsequent layers in the first neural network unit 1320. The combined image storage unit 1391 outputs the combined image in accordance with the output command.
[0144] Next, a description will be given of an example of processing by the abnormal state determination unit 1500B when the state comparison unit (second state comparison unit 1540B) is the first layer state comparison unit 1541. Fig. 12 is a flowchart showing an example of processing by the abnormal state determination unit 1500B when the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B according to the second embodiment of the present disclosure is the first layer state comparison unit 1541.
[0145] When the abnormal state determination unit 1500B executes processing on the data output by the feature extraction unit 1300B (step ST2511), the second state comparison unit 1540B first acquires the output of the first fully connected layer (step ST2512). Then, it executes a first fully connected layer state storage process (step ST2513). The first layer state comparison unit 1541 in the second state comparison unit 1540B stores the output value from the first fully connected layer 1527, which is the input value of the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528).
[0146] The first layer state comparison unit 1541 executes a process of determining whether a reference value (previous value) has been stored (step ST2514). The first layer state comparison unit 1541 refers to the storage unit 1541a and determines whether a reference value (previous value) has been stored.
[0147] If the first layer state comparison unit 1541 determines that the value has been stored previously ("YES" in step ST2514), it performs a comparison process between the current value and the previous value (step ST2515). The first fully connected layer state comparison unit 1541 compares the current value with a reference value (previous value).
[0148] The first layer state comparing unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST2516). The comparing unit 1541b determines whether the difference between the current value and the reference value (previous value) is large or small using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
[0149] If the comparison processing unit 1541b determines that no data was stored previously ("NO" in step ST2514), or if the first layer state comparison unit 1541 determines that the difference is equal to or greater than the threshold value ("NO" in step ST2516), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2611). The second neural network unit 1520 executes normal processing in accordance with the normal processing command and outputs a determination value (step ST2711). The probability storage unit 1591, which is the second output storage unit 1590, stores the determination value output by the second neural network unit 1520 (step ST2712).
[0150] If the first layer state comparator 1541 determines that the difference is smaller than the threshold value ("YES" in step ST2516), the abnormal state determination process controller 1581B, which is the second process controller 1580, executes an omission process command (step ST2612). In the omission process command, the abnormal state determination process controller 1581B instructs the second neural network unit 1520 not to execute processes on the layer to be determined and subsequent layers, and also instructs the second output storage unit 1590 to output the determination value stored in the second output storage unit 1590. In accordance with the omission process command, the second neural network unit 1520 does not execute processes on subsequent layers in the second neural network unit 1520. The probability storage unit 1591, which is the second output storage unit 1590, outputs the determination value in accordance with the output command (step ST2713).
[0151] Next, an example of processing by the abnormal state determination unit 1500B when the state comparison unit (second state comparison unit 1540B) is a second fully connected layer state comparison unit (see the second fully connected layer state comparison unit 1542 in the embodiment described later) will be described. Fig. 13 is a flowchart showing an example of processing by the abnormal state determination unit 1500B when the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B according to the second embodiment of the present disclosure is a second layer state comparison unit.
[0152] After the abnormal state determination unit 1500B processes the data output by the feature extraction unit 1300B (step ST2521), the second fully connected layer 1528 then processes the output from the first fully connected layer (step ST2522). The second state comparison unit 1540B then acquires the second fully connected layer output (step ST2523). The second state comparison unit 1540B then performs second fully connected layer state storage processing (step ST2524). The second layer state comparison unit in the second state comparison unit 1540B stores the output value from the second fully connected layer 1528, which is the input value of the probability output layer 1529 (the value before being processed by the probability output layer 1529).
[0153] The second layer state comparison unit executes a process of determining whether a reference value (previous value) has been stored (step ST2525). The second layer state comparison unit refers to the storage unit and determines whether a reference value (previous value) has been stored.
[0154] When the second layer state comparison unit determines that the value has been stored previously ("YES" in step ST2525), the second layer state comparison unit performs a comparison process between the current value and the previous value (step ST2526). The second layer state comparison unit compares the current value with the reference value (previous value).
[0155] The second layer state comparison unit executes a process of determining whether the difference is smaller than a threshold value (step ST2527). The comparison processing unit determines whether the difference between the current value and the reference value (previous value) is large or small using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
[0156] If the comparison processing unit determines that no data was stored previously ("NO" in step ST2525), or if the second layer state comparison unit determines that the difference is equal to or greater than the threshold value ("NO" in step ST2527), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2621). The second neural network unit 1520 executes normal processing in accordance with the normal processing command and outputs a determination value (step ST2721). The probability storage unit 1591, which is the second output storage unit 1590, stores the determination value output by the second neural network unit 1520 (step ST2722).
[0157] If the first fully connected layer state comparison unit 1541 determines that the difference is smaller than the threshold value ("YES" in step ST2527), the abnormal state determination process control unit 1581B, which is the second process control unit 1580, executes an omission process command (step ST2622). In the omission process command, the abnormal state determination process control unit 1581B instructs the second neural network unit 1520 not to execute processes in the layer to be determined and subsequent layers, and also instructs the second output storage unit 1590 to output the determination value stored in the second output storage unit 1590. In accordance with the omission process command, the second neural network unit 1520 does not execute processes in the subsequent layers in the second neural network unit 1520. The probability storage unit 1591, which is the second output storage unit 1590, outputs the determination value in accordance with the output command (step ST2723).
[0158] The abnormality determination device of the present disclosure is further configured as follows. "The neural network unit includes a first neural network unit and a second neural network unit; the output storage unit includes a first output storage unit and a second output storage unit; the state comparison unit includes a first state comparison unit and a second state comparison unit; the processing control unit includes a first processing control unit and a second processing control unit; the first neural network unit has a plurality of layers that are part of the layers that make up the neural network, and acquires an image output by the image acquisition unit and outputs a feature map that represents a characteristic state of the object to be determined contained in the image; the first output storage unit stores the feature map output by the first neural network unit; the first state comparison unit stores in advance a reference value that is a reference for an input value to a second layer that is at least one layer of the first neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value that is a value that is to be newly input to the second layer; and the first processing control unit, which, when the first state comparison unit determines that the state of the object to be determined has not changed, instructs the first neural network unit not to execute processing in the second layer and subsequent layers of the object to be determined, and instructs it to output the feature map stored in the first output storage unit; the second neural network unit, which has a plurality of layers that are part of the layers that make up the neural network, acquires the feature map output by the feature extraction unit, and uses the feature map to output a judgment value that indicates the state of the object to be determined as the output data; the second output storage unit, which stores the judgment value output by the second neural network unit; the second state comparison unit, which stores in advance a reference value that is a reference for an input value to a third layer that is at least one layer of the second neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value that is a value that is to be newly input to the third layer;and the second processing control unit that, when the second state comparison unit determines that the state of the object to be determined has not changed, instructs the second neural network unit not to execute processing in the third layer and subsequent layers of the object to be determined, and instructs the second neural network unit to output the determination value stored in the second output storage unit as the output data. This provides an effect of providing an abnormality determination device that can output highly accurate determination results more quickly using a neural network in an abnormality determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the above abnormality determination method.
[0159] The abnormality determination device of the present disclosure is further configured as follows: "The first state comparison unit and the second state comparison unit determine the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value to determine a change in the state of the object to be determined based on the result of the determination." This provides an effect of providing an abnormality determination device that can output highly accurate determination results more quickly using a neural network in an abnormality determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the above abnormality determination method.
[0160] The anomaly determination device of the present disclosure is further configured as follows: "The anomaly determination device, wherein the first state comparison unit and the second state comparison unit determine that the state of the object to be determined has not changed if the sum of squares of the difference values for each image is smaller than a pre-stored threshold value." This provides an effect of providing an anomaly determination device that can output highly accurate determination results more quickly using a neural network in an anomaly determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the anomaly determination method.
[0161] The anomaly determination device of the present disclosure is further configured as follows: "The anomaly determination device, wherein the first state comparison unit and the second state comparison unit determine that the state of the object to be determined has not changed if the sum of the absolute values of the difference values for each image is smaller than a pre-stored threshold value." This provides an effect of providing an anomaly determination device that can output highly accurate determination results more quickly using a neural network in an anomaly determination technique. Furthermore, the present disclosure provides the same effect as the above by applying the above configuration to the anomaly determination method.
[0162] Embodiment 3. In embodiment 3, a form in which the basic mechanism of the present disclosure is applied to all layers in a neural network will be described. In embodiment 3, the configurations and processes already described will be omitted as appropriate.
[0163] An example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to embodiment 3 will be described. Fig. 14 is a diagram showing an example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to embodiment 3 of the present disclosure. The abnormality warning device 100C is configured to include an abnormality determination device 1000C and an alarm output unit 2000.
[0164] The abnormality determination device 1000C acquires an image and uses the image to output the state of the object to be determined captured in the image. The abnormality determination device 1000C shown in FIG. 14 includes an image acquisition unit 1100C, a feature extraction unit 1300C, and an abnormal state determination unit 1500C. In the abnormality determination device 1000C, the previously described neural network unit includes a first neural network unit 1320 and a second neural network unit 1520, as described below. The first neural network unit 1320 is included in the feature extraction unit 1300C, and the second neural network unit 1520 is included in the abnormal state determination unit 1500C. Furthermore, in the abnormality determination device 1000C, the previously described output storage unit includes a first output storage unit 1390 and a second output storage unit. The first output storage unit 1390 is included in the feature extraction unit 1300C, and the second output storage unit is included in the abnormal state determination unit 1500C. In addition, in the abnormality determination device 1000C, the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540. The first state comparison unit 1340 is included in the feature extraction unit 1300C, and the second state comparison unit 1540 is included in the abnormal state determination unit 1500C. In addition, in the abnormality determination device 1000C, the process control unit already described is configured to include a first process control unit 1380 and a second process control unit 1580. The first process control unit 1380 is included in the feature extraction unit 1300C, and the second process control unit 1580 is included in the abnormal state determination unit 1500C.
[0165] The image acquisition unit 1100C is similar to the image acquisition units 1100A and 1100B already described, and therefore a detailed description of the image acquisition unit 1100C will be omitted here.
[0166] The feature extraction unit 1300C uses the image to output a feature map that represents a characteristic state of the subject included in the image. The feature extraction unit 1300C extracts, for example, parts of the image that show signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map that represents a state of the subject that is characteristic of an abnormal state such as drowsiness.
[0167] An example of the internal configuration of the feature extraction unit 1300C will be described. Fig. 15 is a diagram showing an example of the internal configuration of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C. The feature extraction unit 1300C shown in Fig. 15 is configured to include a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340C), a processing control unit (first processing control unit 1380C), and an output storage unit (first output storage unit 1390).
[0168] The first neural network unit 1320 has a plurality of layers that are part of the layers that make up the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map that represents the characteristic state of the object to be determined contained in the image. The first neural network unit 1320 shown in FIG. 15 is configured to include an image branching unit 1321, a convolution layer unit 1322, a pooling layer unit 1325, and an image combining unit 1328.
[0169] The image branching unit 1321 branches an image into multiple nodes in a layer of a neural network and inputs the image. The image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 15, the image is branched into two.
[0170] The convolution layer unit 1322 performs filtering to extract characteristic parts of the target object in the image. The convolution layer unit 1322 shown in FIG. 15 includes a first convolution layer 1323 and a second convolution layer 1324. The first convolution layer 1323 and the second convolution layer 1324 perform filtering to extract face (body) parts for detecting drowsy driving, for example, by convolution processing (cross-correlation processing) using a pre-prepared convolution filter of 3×3 or 5×5 size.
[0171] 15 includes a first pooling layer 1326 and a second pooling layer 1327. The first pooling layer 1326 and the second pooling layer 1327 generate an image relating to feature quantities that are robust against image position by, for example, calculating the maximum or average value for each predetermined region.
[0172] The first neural network unit 1320 shown in Figure 15 includes two sets of convolutional layers and pooling layers, but it is also effective to use one set or three or more sets. Furthermore, the first pooling layer and the second pooling layer 1327 may be followed by two or more convolutional layers and two or more pooling layers. Furthermore, a normalized linear unit layer (activation function) or the like (not shown) may be included after the convolutional layers.
[0173] The image combining unit 1328 combines the multiple images output from the convolution layer and the pooling layer, and generates a feature map by extracting areas that indicate signs of drowsiness, such as the eyelids and eyeballs.
[0174] The first state comparison unit 1340C stores in advance, for each of the multiple layers in the first neural network unit 1320, a reference value that is the standard for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and the current value, which is the value that is about to be newly input to that layer.
[0175] The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain a difference value from the current value, which is an input value that is about to be newly input to that layer. In the description of the processing described below, a case will be described in which the previous value is used as the reference value. In this case, the first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) but simply stores the value, but determines changes in the state of the object to be determined for the second and subsequent processing (processing for the second image). However, the reference value may be stored and used as follows:
[0176] The reference value can be a typical processing result that has been learned and modeled in advance. That is, the reference value may be an output value of a typical intermediate layer that has been modeled in advance in a neural network. By using a typical processing result, comparison can be performed without being affected by noise or anomalies contained in the actually observed image.
[0177] Furthermore, the reference value may be a partial result determined by prior information. The partial result determined by prior information may, for example, be the adoption of pixels with a high probability of a head being present. Alternatively, it may be the rejection of pixels that are known to have a large change but low importance, such as background pixels. In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for memory savings and a reduction in the amount of data handled, resulting in faster processing.
[0178] The first state comparison unit 1340 determines whether the state of the object to be determined has changed based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value. Specifically, the first state comparison unit 1340 determines that the state of the object to be determined has not changed if the sum of the squares of the difference values for each image is smaller than the pre-stored threshold value. Also specifically, the first state comparison unit 1340 determines that the state of the object to be determined has not changed if the sum of the absolute values of the difference values for each image is smaller than the pre-stored threshold value.
[0179] 15 includes an image state comparison unit 1341, a convolutional layer state comparison unit 1342, and a pooling layer state comparison unit 1343. In the image state comparison unit 1341, the convolutional layer state comparison unit 1342, and the pooling layer state comparison unit 1343, "storing" refers to retaining the values of the two-dimensional images (feature maps) of the respective input sources. In the image state comparison unit 1341, the convolutional layer state comparison unit 1342, and the pooling layer state comparison unit 1343, "comparison" refers to, for example, calculating the absolute value of the difference between a stored two-dimensional image (of a previous time / frame) and the latest two-dimensional image for each element (pixel), and then comparing the sum or average of the difference with a predetermined threshold.
[0180] The image state comparison unit 1341 shown in FIG. 15 includes a storage unit 1341a and a comparison processing unit 1341b.
[0181] The convolution layer state comparison unit 1342 shown in FIG. 15 includes a storage unit 1342a and a comparison processing unit 1342b.
[0182] The pooling layer state comparison unit 1343 shown in FIG. 15 includes a storage unit 1343a and a comparison processing unit 1343b.
[0183] When the first state comparison unit 1340 determines that the state of the object to be determined has not changed, the first process control unit 1380C instructs the first neural network unit 1320 not to execute processing in the layer of the object to be determined and subsequent layers, and also instructs the first output storage unit 1390 to output the feature map stored in the first output storage unit 1390. The first process control unit 1380C shown in FIG. 15 is configured to include a feature extraction process control unit 1381C. The feature extraction process control unit 1381C executes the functions of the first process control unit 1380C in the feature extraction unit 1300.
[0184] The first output storage unit 1390 stores output data (determination values (probability values)) output by the first neural network unit 1320. The first output storage unit 1390 stores the feature map output by the first neural network unit 1320. The first output storage unit 1390 shown in FIG. 15 is configured to include a combined image storage unit 1391. The combined image storage unit 1391 stores the feature map, which is the combined image combined and output by the first neural network unit 1320.
[0185] Returning to the description of Fig. 14, the abnormal state determination unit 1500C uses a feature map, which is a two-dimensional image, to output a determination value indicating the state of the object to be determined.
[0186] An example of the internal configuration of the abnormal state determination unit 1500C will be described. Fig. 16 is a diagram showing an example of the internal configuration of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C. The abnormal state determination unit 1500C shown in Fig. 16 is configured to include a neural network unit (second neural network unit 1520), a state comparison unit (second state comparison unit 1540), and an output storage unit (second output storage unit) 1590.
[0187] The neural network unit (second neural network unit 1520) has multiple layers in a neural network, acquires the feature map output by the feature extraction unit 1300, and uses the feature map to output a judgment value indicating the state of the object to be judged as output data. The neural network unit (second neural network unit 1520) shown in FIG. 16 is configured to include a state classification unit 1525 and a probability output layer 1529.
[0188] The state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector and further summarizing the output into an indication of the drowsy state (for example, four outputs: eyelids: dozing / not dozing, eyeballs: dozing / not dozing). The state classification unit 1525 shown in FIG. 16 is configured to include a first fully connected layer 1527 and a second fully connected layer 1528. The state classification unit 1525 generates a one-dimensional vector whose number of elements is the desired number of outputs through the first fully connected layer 1527 and the second fully connected layer 1528.
[0189] The probability output layer 1529 applies, for example, a softmax function and sets the sum of the output values to 1.0, thereby giving the output results a probabilistic meaning.
[0190] The second neural network unit 1520 shown in FIG. 16 has two fully connected layers, but may have three or more fully connected layers, or may have no fully connected layers.
[0191] The second state comparison unit 1540 stores in advance, for each of the multiple layers of the second neural network unit 1520, a reference value that is the standard for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and the current value, which is the value that is about to be newly input to that layer.
[0192] The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the second neural network unit 1520, and is used to obtain a difference value from the current value, which is an input value that is about to be newly input to the first layer. In the description of the processing described below, the previous value is used as the reference value. In this case, the second state comparison unit 1540 does not perform state comparison processing for the first processing (processing for the first image) but simply stores the value, but determines changes in the state of the object to be determined for the second and subsequent processing (processing for the second image). However, the reference value may be stored and used as follows:
[0193] The reference value can be a typical processing result that has been learned and modeled in advance. That is, the reference value may be an output value of a typical intermediate layer that has been modeled in advance in a neural network. By using a typical processing result, comparison can be performed without being affected by noise or anomalies contained in the actually observed image.
[0194] Furthermore, the reference value may be a partial result determined by prior information. The partial result determined by prior information may, for example, be the adoption of pixels with a high probability of a head being present. Alternatively, it may be the rejection of pixels that are known to have a large change but low importance, such as background pixels. In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for memory savings and a reduction in the amount of data handled, resulting in faster processing.
[0195] The second state comparison unit 1540 determines whether the state of the object to be determined has changed based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value. Specifically, the second state comparison unit 1540 determines that the state of the object to be determined has not changed if the sum of the squares of the difference values in units of images is smaller than the pre-stored threshold value. Also specifically, the second state comparison unit 1540 determines that the state of the object to be determined has not changed if the sum of the absolute values of the difference values in units of images is smaller than the pre-stored threshold value.
[0196] The second state comparison unit 1540C shown in FIG. 16 includes a first layer state comparison unit 1541 and a second layer state comparison unit 1542. The first layer state comparison unit 1541 is also referred to as the first fully connected layer state comparison unit 1541. The second layer state comparison unit 1542 is also referred to as the second fully connected layer state comparison unit 1542. In the first fully connected layer state comparison unit 1541 and the second fully connected layer state comparison unit 1542, "storing" means retaining the values of the one-dimensional vectors of the respective input sources. In the first fully connected layer state comparison unit 1541 and the second fully connected layer state comparison unit 1542, "comparison" means, for example, calculating the absolute value of the difference between the stored one-dimensional vector (of the previous time / previous frame) and the latest one-dimensional vector for each element, and then comparing the sum or average with a predetermined threshold.
[0197] 16 includes a storage unit 1541a and a comparison processing unit 1541b. The storage unit 1541a stores a reference value, which is an output value of the first fully connected layer 1527 and an input value of the second fully connected layer, for each output by the first fully connected layer 1527. The comparison processing unit 1541b compares the reference value with the current value.
[0198] 16 includes a storage unit 1542a and a comparison processing unit 1542b. The storage unit 1542a stores a reference value, which is an output value of the second fully connected layer and an input value of the probability output layer 1529, for each output by the second fully connected layer. The comparison processing unit 1542b compares the reference value with the current value.
[0199] When the second state comparison unit 1540 determines that the state of the object to be determined has not changed, the second process control unit 1580C instructs the second neural network unit 1520 not to execute processing in the layer of the object to be determined and subsequent layers, and also instructs the second process control unit 1580C to output the determination value stored in the second output storage unit as output data. The second process control unit 1580C shown in Fig. 16 is configured to include an abnormal state determination process control unit 1581C. The abnormal state determination process control unit 1581C functions to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500C.
[0200] The second output storage unit 1590 stores the judgment value output by the second neural network unit 1520. The output storage unit (second output storage unit) 1590 shown in FIG.
[0201] The probability storage unit 1591 stores the judgment value output by the second neural network unit 1520. The judgment value is a value indicating the state of the person to be judged, and is, for example, a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.
[0202] Returning to the description of Fig. 14, the warning output unit 2000 acquires output data that is a determination value indicating the drowsy state or dozing state of the person to be determined, and outputs a warning to the person to be determined according to the determination value.
[0203] Although the abnormality determination device 1000C shown in Fig. 14 is shown as having a configuration that does not include the alarm output unit 2000, it may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000C is equivalent to the abnormality warning device 100C shown in Fig. 14. In the following description, the abnormality determination device 1000C will be described as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100C and the abnormality determination device 1000C.
[0204] In addition to the above components, the abnormality determination device 1000C may also include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire abnormality determination device 1000C and each of its components. For example, the control unit (not shown) starts up the abnormality determination device 1000C in response to an external command. The control unit (not shown) also controls the state (operating state, such as startup, shutdown, or sleep) of the abnormality determination device 1000C. The storage unit (not shown) stores various data used by the abnormality determination device 1000C. For example, the storage unit (not shown) stores output (output data) from each component of the abnormality determination device 1000C and outputs data requested by each component to the requesting component. The communication unit (not shown) communicates with external devices. For example, the communication unit (not shown) communicates between the abnormality determination device 1000C and an imaging device such as an in-vehicle camera. Furthermore, for example, if the abnormality determination device 1000C does not include a display unit or an audio output unit, communication is performed between the abnormality determination device 1000C and an external device such as a display unit or an audio output unit.
[0205] An example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to embodiment 3 will be described. Fig. 17 is a flowchart showing an example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to embodiment 3 of the present disclosure. When an image is input from a camera, for example, the abnormality determination device 1000C starts the processing shown in Fig. 17.
[0206] Abnormality determination device 1000C executes image acquisition processing (step ST3100). In the image acquisition processing, image acquisition unit 1100 of abnormality determination device 1000C acquires and outputs an image.
[0207] Next, abnormality determination device 1000C executes storage and state comparison processing (step ST3200). In the state storage and state comparison processing, first state comparison unit 1340C of abnormality determination device 1000C stores previous values, which are input values for each layer, that are the processing results for all layers of first neural network unit 1320 at least the first time after processing starts. First state comparison unit 1340 does not perform state comparison processing for the first time (processing for the first image) and simply stores the values, but determines changes in the state of the object to be determined for the second and subsequent times (processing for the second image).
[0208] Next, abnormality determination device 1000C executes feature extraction process control (step ST3300). Feature extraction process control unit 1381 of abnormality determination device 1000C issues a normal process command or an omission process command to first neural network unit 1320, depending on the determination result by first state comparison unit 1340. Furthermore, when issuing an omission process command, feature extraction process control unit 1381 issues an output command to combined image storage unit 1391.
[0209] Next, abnormality determination device 1000C executes a combined image output process (step ST3400). First neural network unit 1320 executes normal processing to output a combined image, or combined image storage unit 1391 outputs the previous combined image. This causes feature extraction unit 1300 to output the combined image.
[0210] The abnormality determination device 1000C executes a fully connected layer state storage process and a fully connected layer state comparison process (step ST3500). The second state comparison unit 1540 in the abnormality determination device 1000C pre-stores, for each of the multiple layers of the second neural network unit 1520, a reference value that is a reference for the input value to that layer, and determines a change in the state of the object to be determined using the difference between the reference value and the current value, which is the value that is about to be newly input to that layer. The second state comparison unit 1540 does not perform a state comparison process for the first processing (processing for the first image) but simply stores the value. However, it determines a change in the state of the object to be determined for the second and subsequent processing (processing for the second image).
[0211] Abnormality determination device 1000C executes abnormal state determination process control process (step ST3600). Abnormal state determination process control unit 1581C issues a normal command or an omission command to second neural network unit 1520. When issuing an omission command, abnormal state determination process control unit 1581C issues an output command to probability storage unit 1591.
[0212] Abnormality determination device 1000C executes a result output process (step ST3700). Second neural network unit 1520 executes normal processing and outputs a determination value, or the probability storage unit outputs the previous determination value (probability value). As a result, the abnormality determination unit outputs the determination value as output data.
[0213] Abnormality determination device 1000C executes alarm output processing (step ST3800). In the alarm output processing, alarm output unit 2000 of abnormality determination device 1000C acquires output data and outputs an alarm signal to an alarm device (not shown) or the like based on the output data. Alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and if it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) or the like.
[0214] 17 and repeats the process from step ST3100. Note that abnormality determination device 1000C also becomes power-off in conjunction with, for example, the camera being powered off.
[0215] A detailed example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to embodiment 3 will be described below. Fig. 18 is a flowchart showing a detailed first example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to embodiment 3 of the present disclosure.
[0216] When feature extraction unit 1300C starts processing, first, first state comparison unit 1340C of feature extraction unit 1300C executes image storage processing (step ST3210). In the image storage processing, image state comparison unit 1341 of first state comparison unit 1340C stores images that are input data to first neural network unit 1320. Furthermore, image state comparison unit 1341 stores the image in storage unit 1341a every time first neural network unit 1320 acquires an image.
[0217] The image state comparison unit 1341 executes a process of determining whether or not the image has been stored previously (step ST3211). The image state comparison unit 1341 refers to the storage unit 1341a and determines whether or not the image input previously has been stored.
[0218] When the image state comparison unit 1341 determines that the image has been stored previously ("YES" in step ST3211), it executes a comparison process between the current image and the previous image (step ST3212). The comparison processing unit 1341b of the image state comparison unit 1341 compares the image input this time with the image input previously.
[0219] The comparison processing unit 1341b executes a process of determining whether the difference is less than a threshold value (step ST3213). The comparison processing unit 1341b determines whether the difference between the reference value and the current value is large or small using the difference value between the currently input image and the previously input image and a pre-stored threshold value.
[0220] If the image state comparison unit 1341 determines that the difference between the previous value and the current value is greater than the threshold value ("NO" in step ST3213), the convolution layer state comparison unit 1342 executes a convolution layer state storage process (step ST3214). In the convolution layer state storage process, the convolution layer state comparison unit 1342 stores input data for the convolution layer in the storage unit 1342a.
[0221] The convolution layer state comparison unit 1342 executes a process of determining whether a previous value has been stored (step ST3215). The convolution layer state comparison unit 1342 refers to the storage unit 1342a and determines whether a previous value, which is a value input previously, is stored.
[0222] When the comparison processing unit 1342b of the convolutional layer state comparing unit 1342 determines that the previous data has been stored ("YES" in step ST3215), it executes a comparison process between the current data and the previous data (step ST3216).
[0223] The feature extraction unit 1300C executes a process of determining whether the difference is less than a threshold value (step ST3217). The comparison processing unit 1342b determines whether the difference between the previous value and the current value is large or small using the difference value between the currently input image and the previously input image and a pre-stored threshold value.
[0224] If the comparison processing unit 1342b of the convolutional layer state comparing unit 1342 determines that the difference between the previous value and the current value is greater than the threshold value ("NO" in step ST3217), the pooling layer state comparing unit 1343 executes a pooling layer state storing process (step ST3218). In the pooling layer state storing process, the pooling layer state comparing unit 1343 stores the input data for the pooling layer in the storage unit 1343a.
[0225] The comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process of determining whether or not the data has been stored previously (step ST3219).
[0226] When the comparison processing unit 1343b of the pooling layer state comparing unit 1343 determines that the value has been stored previously ("YES" in step ST3219), it executes a comparison process between the current value and the reference value (previous value) (step ST3220).
[0227] The comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process of determining whether the difference between the reference value (previous value) and the current value is less than a threshold value (step ST3221).
[0228] If the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolution layer state comparison unit 1342, and the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determine that the difference between the reference value (previous value) and the current value is greater than the threshold value (step ST3213 "NO", step ST3217 "NO", step ST3221 "NO"), the feature extraction unit 1300C executes normal processing command processing (step ST3311). In the normal processing command processing, the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320.
[0229] The feature extraction unit 1300C executes an output process (step ST3411). The first neural network unit 1320 of the feature extraction unit 1300C outputs the combined image.
[0230] The feature extraction unit 1300C executes an output storage process (step ST3412). The combined image storage unit 1391 of the output storage unit in the feature extraction unit 1300C stores the combined image output from the first neural network unit 1320.
[0231] If the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolutional layer state comparison unit 1342, or the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the difference between the previous value and the current value is smaller than the threshold value (YES in step ST3213, YES in step ST3217, or YES in step ST3221), the feature extraction unit 1300C executes an omission command process (step ST3312). In the omission command process, the feature extraction processing control unit 1381 instructs the first neural network unit 1320 not to execute processing in the layer to be determined and subsequent layers, and also instructs the first output storage unit 1390 to output the feature map.
[0232] The feature extraction unit 1300C executes the stored data output process (step ST3413), and then ends the process shown in FIG.
[0233] 19 is a flowchart showing a first detailed example of the processing of abnormal state determination unit 1500C in abnormality warning device 100C and abnormality determination device 1000C according to embodiment 3 of the present disclosure. Abnormal state determination unit 1500C starts processing upon acquiring the feature map output from feature extraction unit 1300C. Abnormal state determination unit 1500C first executes processing on the data output by feature extraction unit 1300C (step ST3510).
[0234] The abnormal state determination unit 1500C acquires the output of the fully connected layer (step ST3511). The second state comparison unit 1540 in the abnormal state determination unit 1500C acquires the output value output from the first fully connected layer 1527.
[0235] The first fully connected layer state comparison unit 1541 executes a first fully connected layer state storage process (step ST3512). The first fully connected layer state comparison unit 1541 stores the output value from the first fully connected layer 1527, which is the input value to the second fully connected layer 1528, as a reference value.
[0236] The first fully connected layer state comparison unit 1541 executes a process of determining whether a reference value (previous value) has been stored (step ST3513). The first fully connected layer state comparison unit 1541 determines whether a reference value (previous value) has been stored by referring to the storage unit 1541a.
[0237] When the first fully connected layer state comparison unit 1541 determines that the previous value has been stored ("YES" in step ST3513), it executes a comparison process between the current value and the previous value (step ST3514). The first fully connected layer state comparison unit 1541 compares the current value with the reference value (previous value).
[0238] The first fully connected layer state comparing unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST3515). The comparing unit 1541b determines whether the difference between the current value and the reference value (previous value) is large or small using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
[0239] If the first fully connected layer state comparing unit 1541 determines that the difference is greater than the threshold, normal processing is performed in the second fully connected layer 1528, and the second fully connected layer state comparing unit 1542 performs second fully connected layer state storage processing (step ST3516). The second fully connected layer state comparing unit 1542 stores the output value from the second fully connected layer 1528, which is the input value of the probability output layer 1529, as a reference value.
[0240] The second fully connected layer state comparison unit 1542 executes a process of determining whether a reference value (previous value) has been stored (step ST3517). The second fully connected layer state comparison unit 1542 determines whether a reference value (previous value) has been stored by referring to the storage unit 1542a.
[0241] If the second fully connected layer state comparison unit 1542 determines that the previous value has been stored ("YES" in step ST3517), it executes a comparison process between the current value and the previous value (step ST3518). The second fully connected layer state comparison unit 1542 compares the current value with the reference value (previous value).
[0242] The second fully connected layer state comparison unit 1542 executes a process of determining whether the difference is smaller than the threshold value (step ST3519). The comparison processing unit 1542b determines whether the difference between the current value and the reference value (previous value) is large or small using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
[0243] If the first fully connected layer state comparison unit 1541 determines that the difference is greater than the threshold value (step ST3515 "NO"), and if the second fully connected layer state comparison unit 1542 determines that the difference is greater than the threshold value (step ST3519 "NO"), the abnormal state determination unit 1500C executes normal processing command processing (step ST3611).
[0244] The abnormal state determination unit 1500C executes an output process (step ST3711). The neural network unit 1520 (second neural network unit 1520) in the abnormal state determination unit 1500C outputs a determination value.
[0245] The abnormal state determination unit 1500C executes an output storage process (step ST3712). The output storage unit 1590 (second output storage unit 1590) in the abnormal state determination unit 1500C stores the determination value output by the neural network unit 1520 (second neural network unit 1520).
[0246] If the first fully connected layer state comparing unit 1541 determines that the difference is smaller than the threshold value, and if the second fully connected layer state comparing unit 1542 determines that the difference is smaller than the threshold value, the abnormal state determining unit 1500C executes an omission command (step ST3519). In the omission command, the abnormal state determining process control unit 1581 commands the second neural network unit 1520 not to execute processing in the layer to be determined and subsequent layers, and also commands the second output storage unit 1590 to output the determination value stored in the second output storage unit 1590.
[0247] The abnormal state determination section 1500C executes a stored data output process (step ST3713). The second output storage section 1590 outputs the determination value.
[0248] When second neural network unit 1520 or second output storage unit 1590 outputs the determination value, the series of processes shown in FIG. 19 ends.
[0249] Next, a detailed example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to embodiment 3 will be described. Fig. 20 is a flowchart showing a second detailed example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to embodiment 3 of the present disclosure.
[0250] When feature extraction unit 1300C starts processing, first, first state comparison unit 1340C of feature extraction unit 1300C executes image storage processing (step ST3210). In the image storage processing, image state comparison unit 1341 of first state comparison unit 1340C stores images that are input data to first neural network unit 1320. Furthermore, image state comparison unit 1341 stores the image in storage unit 1341a every time first neural network unit 1320 acquires an image.
[0251] The image state comparison unit 1341 executes a process of determining whether or not the image has been stored previously (step ST3211). The image state comparison unit 1341 refers to the storage unit 1341a and determines whether or not the image input previously has been stored.
[0252] When the image state comparison unit 1341 determines that the image has been stored previously ("YES" in step ST3211), it executes a comparison process between the current image and the previous image (step ST3212). The comparison processing unit 1341b of the image state comparison unit 1341 compares the image input this time with the image input previously.
[0253] The comparison processing unit 1341b executes a process of determining whether the difference is less than a threshold value (step ST3213). The comparison processing unit 1341b determines whether the difference between the reference value and the current value is large or small using the difference value between the currently input image and the previously input image and a pre-stored threshold value.
[0254] If the image state comparison unit 1341 determines that the value has not been stored previously ("NO" in step ST3211), or if the image state comparison unit 1341 determines that the difference between the previous value and the current value is equal to or greater than the threshold value ("NO" in step ST3213), the feature extraction processing control unit 1381C issues a normal processing command (step ST3231), and processing is performed by the convolution layer unit 1322. Next, the convolution layer state comparison unit 1342 performs convolution layer state storage processing (step ST3214). In the convolution layer state storage processing, the convolution layer state comparison unit 1342 stores the input data for the convolution layer in the storage unit 1342a.
[0255] The convolution layer state comparison unit 1342 executes a process of determining whether a previous value (reference value), which is a value input previously, has been stored (step ST3215). The convolution layer state comparison unit 1342 refers to the storage unit 1342a and determines whether a previous value (reference value), which is a value input previously, has been stored.
[0256] When the comparison processing unit 1342b of the convolutional layer state comparing unit 1342 determines that the previous data has been stored ("YES" in step ST3215), it executes a comparison process between the current data and the previous data (step ST3216).
[0257] The feature extraction unit 1300C executes a process of determining whether the difference is less than a threshold value (step ST3217). Specifically, the comparison processing unit 1342b of the convolution layer state comparison unit 1342 determines whether the difference between the previous value and the current value is large or small, using a difference between a value to be input this time (current value) to the pooling layer unit 1325 and a value input last time (previous value) and a pre-stored threshold value.
[0258] If the convolutional layer state comparing unit 1342 determines that the value has not been stored previously ("NO" in step ST3211), or if the comparison processing unit 1342b of the convolutional layer state comparing unit 1342 determines that the difference between the previous value and the current value is equal to or greater than the threshold value ("NO" in step ST3217), the feature extraction processing control unit 1381C issues a normal processing command (step ST3232), and processing is performed by the pooling layer unit 1325. Next, in the pooling layer state storing processing, the pooling layer state comparing unit 1343 stores the input data for the pooling layer (input value to be processed) in the storage unit 1343a.
[0259] Next, the comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process of determining whether or not the data has been stored previously (step ST3219).
[0260] When the comparison processing unit 1343b of the pooling layer state comparing unit 1343 determines that the value has been stored previously ("YES" in step ST3219), it executes a comparison process between the current value and the reference value (previous value) (step ST3220).
[0261] The comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process of determining whether the difference between the reference value (previous value) and the current value is less than a threshold value (step ST3221).
[0262] If the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the difference between the reference value (previous value) and the current value is equal to or greater than the threshold value ("NO" in step ST3221), the feature extraction unit 1300C executes normal processing command processing (step ST3311). In the normal processing command processing, the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320. The image combination unit 1328 in the first neural network unit 1320 executes image combination processing.
[0263] The feature extraction unit 1300C executes an output storage process (step ST3412). The combined image storage unit 1391 of the output storage unit in the feature extraction unit 1300C stores the combined image output from the first neural network unit 1320.
[0264] If the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolutional layer state comparison unit 1342, or the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the difference between the previous value and the current value is smaller than the threshold value (YES in step ST3213, YES in step ST3217, or YES in step ST3221), the feature extraction unit 1300C executes an omission command process (step ST3312). In the omission command process, the feature extraction processing control unit 1381 instructs the first neural network unit 1320 not to execute processing in the layer to be determined and subsequent layers, and also instructs the first output storage unit 1390 to output the feature map.
[0265] The feature extraction unit 1300C executes the stored data output process (step ST3413), and then ends the process shown in FIG.
[0266] 21 is a flowchart showing a second detailed example of the processing of abnormal state determination unit 1500C in abnormality warning device 100C and abnormality determination device 1000C according to embodiment 3 of the present disclosure. Abnormal state determination unit 1500C starts processing upon acquiring the feature map output from feature extraction unit 1300C. Abnormal state determination unit 1500C first executes processing on the data output by feature extraction unit 1300C (step ST3510).
[0267] The abnormal state determination unit 1500C acquires the output of the fully connected layer (step ST3511). The second state comparison unit 1540 in the abnormal state determination unit 1500C acquires the output value output from the first fully connected layer 1527.
[0268] The first layer state comparison unit 1541 executes a first layer state storage process (step ST3512). The first fully connected layer state comparison unit 1541 stores, as a reference value, the output value from the first fully connected layer 1527 and the input value to the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528).
[0269] The first layer state comparing unit 1541 executes a process of determining whether a reference value (previous value) has been stored (step ST3513). The first layer state comparing unit 1541 refers to the storage unit 1541a and determines whether a reference value (previous value) has been stored.
[0270] If the first layer state comparison unit 1541 determines that the value has been stored previously ("YES" in step ST3513), it performs a comparison process between the current value and the previous value (step ST3514). The first fully connected layer state comparison unit 1541 compares the current value with the reference value (previous value).
[0271] The first layer state comparing unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST3515). The comparison processing unit 1541b in the first layer state comparing unit 1541 determines whether the difference between the reference value and the current value is large or small, using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
[0272] If the first layer state comparison unit 1541 determines that no data was stored previously (step ST3211 "NO"), or if the comparison processing unit 1541b in the first layer state comparison unit 1541 determines that the difference between the current value and the reference value (previous value) is equal to or greater than a threshold value (step ST3515 "NO"), the abnormal state determination processing control unit 1581C issues a normal processing command (step ST3520), and processing by the second fully connected layer 1528 is executed.
[0273] Next, the second fully connected layer state comparison unit 1542 executes a second fully connected layer state storage process (step ST3516). The second fully connected layer state comparison unit 1542 stores the output value from the second fully connected layer 1528 and the input value to the probability output layer 1529 (the value before being processed by the probability output layer 1529) as a reference value.
[0274] Next, the second fully connected layer state comparison unit 1542 executes a process of determining whether a reference value (previous value) has been stored (step ST3517). The second fully connected layer state comparison unit 1542 determines whether a reference value (previous value) has been stored by referring to the storage unit 1542a.
[0275] If the second fully connected layer state comparison unit 1542 determines that the previous value has been stored ("YES" in step ST3517), it executes a comparison process between the current value and the previous value (step ST3518). The second fully connected layer state comparison unit 1542 compares the current value with the reference value (previous value).
[0276] The second fully connected layer state comparison unit 1542 executes a process of determining whether the difference is less than the threshold value (step ST3519). The comparison processing unit 1542b in the second fully connected layer state comparison unit 1542 determines whether the difference between the current value and the reference value (previous value) is large or small using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
[0277] If the second fully connected layer state comparison unit 1542 determines that no data was stored previously (step ST3517 "NO"), or if the comparison processing unit 1542b in the second fully connected layer state comparison unit 1542 determines that the difference is greater than or equal to the threshold value (step ST3519 "NO"), the abnormal state determination unit 1500C executes normal processing command processing (step ST3611).
[0278] The abnormal state determination unit 1500C executes an output process (step ST3711). The second neural network unit 1520 in the abnormal state determination unit 1500C outputs a determination value.
[0279] The abnormal state determination unit 1500C executes an output storage process (step ST3712). The second output storage unit 1590 in the abnormal state determination unit 1500C stores the determination value output by the second neural network unit 1520 in the probability storage unit 1591.
[0280] If the first layer state comparing unit 1541 determines that the difference is smaller than the threshold value (less than the threshold value), and if the comparison processing unit 1542b in the second layer state comparing unit 1542 determines that the difference is smaller than the threshold value, the abnormal state determining unit 1500C executes an omission processing command (step ST3519). In the omission processing command, the abnormal state determining processing control unit 1581C instructs the second neural network unit 1520 not to execute processing on the layer to be determined and subsequent layers, and also instructs the second output storage unit 1590 to output the determination value stored in the second output storage unit 1590.
[0281] The abnormal state determination unit 1500C executes the stored data output process (step ST3713). The second output storage unit 1590 outputs the determination value (probability value) stored in the probability storage unit 1591 in response to a command from the abnormal state determination process control unit 1581C.
[0282] When second neural network unit 1520 or second output storage unit 1590 outputs the judgment value (probability value), the series of processes shown in FIG. 21 ends.
[0283] As described above, according to the third embodiment, when each state storage and comparison unit determines that the change is small, it is possible to avoid further processing, thereby reducing the amount of calculation and achieving higher speed. In this case, the output result is the previous highly accurate determination result, so it is possible to continue outputting highly accurate results.
[0284] The abnormality determination device of the present disclosure is further configured as follows: "The neural network unit includes a first neural network unit and a second neural network unit, The output storage unit includes a first output storage unit and a second output storage unit, The state comparison unit includes a first state comparison unit and a second state comparison unit, The processing control unit includes a first processing control unit and a second processing control unit, The first neural network unit has a plurality of layers that are a part of the layers that make up the neural network, and acquires an image output by the image acquisition unit and outputs a feature map that represents a characteristic state of the object to be determined contained in the image, The first output storage unit stores the feature map output by the first neural network unit, The first state comparison unit stores in advance, for each of a plurality of layers in the first neural network unit, a reference value that is a reference for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and a current value that is a value that is to be newly input to that layer, And the first processing control unit, when the first state comparison unit determines that the state of the object to be determined has not changed, instructs the first neural network unit not to execute processing in the layer of the object to be determined and subsequent layers, and instructs the first processing control unit to output the feature map stored in the first output storage unit; the second neural network unit, which has a plurality of layers in the neural network, acquires the feature map output by the feature extraction unit, and uses the feature map to output a judgment value indicating the state of the object to be determined as the output data; the second output storage unit, which stores the judgment value output by the second neural network unit; the second state comparison unit, which stores in advance a reference value that is a reference for the input value to that layer for each of a plurality of layers of the second neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value that is a value that is to be newly input to that layer;and an abnormal state determination unit configured to include the second processing control unit, which, when the second state comparison unit determines that the state of the object to be determined has not changed, instructs the second neural network unit not to execute processing in the layer of the object to be determined and subsequent layers, and instructs the second neural network unit to output the determination value stored in the second output storage unit as the output data." This provides an effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to output a highly accurate determination result more quickly. Furthermore, the present disclosure provides an effect similar to the effect described above by applying the above configuration to the above abnormality determination method.
[0285] Here, a hardware configuration for realizing the functions of the abnormality warning device 100 (100A, 100B, 100C) and abnormality determination device 1000 (1000A, 1000B, 1000C) of the present disclosure will be described. FIG. 22 is a diagram showing a first example of a hardware configuration for realizing the functions of the present disclosure. FIG. 23 is a diagram showing a second example of a hardware configuration for realizing the functions of the present disclosure. The abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) of the present disclosure are each realized by hardware such as that shown in FIG. 22 or FIG. 23.
[0286] As shown in FIG. 22 , each of the abnormality warning devices 100 (100A, 100B, 100C) and the abnormality determination devices 1000 (1000A, 1000B, 1000C) is configured with, for example, a processor 10001, a memory 10002, and a communication circuit 10004. The processor 10001 and the memory 10002 are mounted on, for example, a computer. The memory 10002 stores the computer, an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, an output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, a first neural network unit 1320, an image branching unit 1321, and a convolution layer unit 1322. , first convolutional layer 1323, second convolutional layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combination unit 1328, first state comparison units 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolutional layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, a comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and a program for functioning as a control unit not shown are stored.The processor 10001 reads out and executes the program stored in the memory 10002, thereby controlling the image acquisition unit 1010, the neural network unit 1020, the state comparison unit 1030, the processing control unit 1040, the output storage unit 1050, the image acquisition units 1100, 1100B, and 1100C, the feature extraction units 1300, 1300B, and 1300C, and the first neural network unit 13 20, image branching unit 1321, convolution layer unit 1322, first convolution layer 1323, second convolution layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combination unit 1328, first state comparison unit 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolution layer state comparison unit 1342, comparison Processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528 , probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and a control unit (not shown) are realized. In addition, a storage unit (not shown) is realized by memory 10002 or another memory (not shown). In addition, the storage units 1341a, 1342a, 1343a, first output storage unit 1390, combined image storage unit 1391, storage units 1541a, 1542a, second output storage unit 1590, and probability storage unit 1591 in the abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) are realized by memory 10002 or another memory not shown. In addition, a communication unit not shown is realized by communication circuit 10004.The processor 10001 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, or a DSP (Digital Signal Processor). The memory 10002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory) or flash memory, or a magnetic disk such as a hard disk or flexible disk, or an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or a magneto-optical disk. The processor 10001 and the memory 10002 or the communication circuit 10004 are connected in a state capable of transmitting data to each other. The processor 10001, memory 10002, and communication circuit 10004 are connected via an input / output interface 10003 so as to be capable of transmitting data to and from other hardware.
[0287] Alternatively, the image acquisition unit 1010, the neural network unit 1020, the state comparison unit 1030, the processing control unit 1040, the output storage unit 1050, the image acquisition units 1100, 1100B, 1100C, the feature extraction units 1300, 1300B, 1300C, the first neural network unit 1320, the image branching unit 1321, the convolution layer unit 1322, the first convolution layer 1323, the second Convolutional layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combining unit 1328, first state comparison units 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolutional layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first The functions of the processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and control units not shown may be realized by a dedicated processing circuit 20001, as shown in FIG. 23 . Furthermore, a communication unit (not shown) is realized by the communication circuit 10004. The processing circuit 20001 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration). Furthermore, a storage unit (not shown) is realized by the memory 20002 or another memory (not shown).The memory 20002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory, or a magnetic disk such as a hard disk or flexible disk, or an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or a magneto-optical disk. The communication circuit 20004 implements a communication unit (not shown). The processing circuit 20001 and the memory 20002 or the communication circuit 20004 are connected in a state where they can transmit data to each other. In addition, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state where they can transmit data to other hardware via the input / output interface 20003.The image acquisition unit 1010, neural network unit 1020, state comparison unit 1030, processing control unit 1040, output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, first neural network unit 1320, image branching unit 1321, convolution layer unit 1322, first convolution layer 1323, second A convolutional layer 1324, a pooling layer unit 1325, a first pooling layer 1326, a second pooling layer 1327, an image combining unit 1328, first state comparison units 1340, 1340B, 1340C, an image state comparison unit 1341, a comparison processing unit 1341b, a convolutional layer state comparison unit 1342, a comparison processing unit 1342b, a pooling layer state comparison unit 1343, a comparison processing unit 1343b, The functions of the first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and a control unit not shown may be realized by separate processing circuits, or may be realized together by a processing circuit.
[0288] Alternatively, an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, an output storage unit 1050, image acquisition units 1100, 1100B, 1100C, feature extraction units 1300, 1300B, 1300C, a first neural network unit 1320, an image branching unit 1321, a convolution layer unit 1322, a first convolution layer 1323, a second convolution layer 1324, a 4. Pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combination unit 1328, first state comparison units 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolution layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380 B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and some of the functions of a control unit (not shown) may be realized by processor 10001 and memory 10002, and the remaining functions may be realized by processing circuit 20001.
[0289] It should be noted that the present disclosure allows for free combination of the embodiments, modification of any of the components of the embodiments, or omission of any of the components of the embodiments.
[0290] The present disclosure enables an abnormality determination technology to quickly output highly accurate determination results using a neural network without performing calculations in all layers of the neural network, thereby enabling abnormality determination such as fast and highly accurate determination of drowsiness, for example, and is therefore suitable for application to, for example, an in-vehicle driver monitoring system.
[0291] 100, 100A, 100B, 100C: abnormality warning device, 1000, 1000A, 1000B, 1000C: abnormality determination device, 1010: image acquisition unit, 1020: neural network unit, 1030: state comparison unit, 1040: processing control unit, 1050: output storage unit, 1100, 1100B, 1100C: image acquisition unit, 1300, 1300B, 1300C: feature extraction unit, 1320: neural network unit (first neural network unit), 1321: image branching unit, 1322: convolution layer unit, 1323: first convolution layer, 1324: second convolution layer, 1325: pooling layer unit, 1326: first pooling layer, 1327: second pooling layer, 1328 Image combination unit, 1340, 1340B, 1340C state comparison unit (first state comparison unit), 1341 image state comparison unit, 1341a storage unit, 1341b comparison processing unit, 1342 convolution layer state comparison unit, 1342a storage unit, 1342b comparison processing unit, 1343 pooling layer state comparison unit, 1343a storage unit, 1343b comparison processing unit, 1380, 1380B, 1380C processing control unit (first processing control unit), 1381, 1381B, 1381C feature extraction processing control unit, 1390 output storage unit (first output storage unit), 1391 combined image storage unit, 1500, 1500B, 1500C abnormal state determination unit, 1520 neural network unit (second neural network unit), 1525 state classification unit, 1527 First fully connected layer, 1528 Second fully connected layer, 1529 Probability output layer, 1540 State comparison unit (second state comparison unit), 1541 First layer state comparison unit (first fully connected layer state comparison unit), 1541a Storage unit, 1541b Comparison processing unit, 1542 Second layer state comparison unit (second fully connected layer state comparison unit), 1542a Storage unit, 1542b Comparison processing unit, 1580, 1580B, 1580C Processing control unit (second processing control unit), 1581, 1581B, 1581C Abnormal state determination processing control unit, 1590 Output storage unit (second output storage unit), 1591 Probability storage unit, 2000 Alarm output unit, 10001 Processor, 10002 Memory, 10003 Input / output interface, 10004 Communication circuit, 20001 Processing circuit, 20002 memory, 20003 input / output interface, 20004 communication circuit.
Claims
1. An anomaly detection device comprising: an image acquisition unit that acquires and outputs an image; a neural network unit having multiple layers in a neural network, that acquires the image output by the image acquisition unit and outputs a detection result showing the state of an object to be detected contained in the image; an output storage unit that stores the output data output by the neural network unit; a state comparison unit that pre-stores a reference value that is a standard for an input value to a first layer, which is at least one of the multiple layers of the neural network unit, and determines a change in the state of the object to be detected using a difference value between the reference value and a current value, which is a value to be newly input to the first layer; and a processing control unit that, when the state comparison unit determines that the state of the object to be detected has not changed, instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs it to output the output data stored in the output storage unit.
2. The neural network unit includes a first neural network unit and a second neural network unit, the output storage unit includes a first output storage unit and a second output storage unit, the state comparison unit includes a first state comparison unit and a second state comparison unit, and the processing control unit includes a first processing control unit and a second processing control unit, the first neural network unit having a plurality of layers which are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map which represents a characteristic state of the object to be determined contained in the image, the first output storage unit storing the feature map output by the first neural network unit, the first state comparison unit storing in advance a reference value which is a reference for an input value to a second layer which is at least one layer of the first neural network unit, and determining a change in the state of the object to be determined by using a difference value between the reference value and a current value which is a value to be newly input to the second layer, and the first processing control unit, when it is determined by the first state comparison unit that the state of the object to be determined has not changed, instructs the first neural network unit not to execute processing in the second layer and subsequent layers of the object to be determined, and instructs it to output the feature map stored in the first output storage unit; the second neural network unit has a plurality of layers which are a part of the layers constituting the neural network, acquires the feature map output by the feature extraction unit, and uses the feature map to output a judgment value indicating the state of the object to be determined as the output data; the second output storage unit, which stores the judgment value output by the second neural network unit; the second state comparison unit, which stores in advance a reference value which is a reference for an input value to a third layer which is at least one layer of the layers in the second neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value which is a value to be newly input to the third layer;and an abnormal state determination unit configured to include the second processing control unit, when the second state comparison unit determines that the state of the object to be determined has not changed, instructing the second neural network unit not to execute processing in the third layer and subsequent layers of the object to be determined, and instructing the second neural network unit to output the determination value stored in the second output storage unit as the output data.
3. The neural network unit includes a first neural network unit and a second neural network unit, the output storage unit includes a first output storage unit and a second output storage unit, the state comparison unit includes a first state comparison unit and a second state comparison unit, and the processing control unit includes a first processing control unit and a second processing control unit, the first neural network unit having a plurality of layers which are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map which represents a characteristic state of the object to be determined contained in the image, the first output storage unit storing the feature map output by the first neural network unit, the first state comparison unit storing in advance a reference value which is a reference for an input value to the layer for each of the plurality of layers in the first neural network unit, and determining a change in the state of the object to be determined by using a difference value between the reference value and a current value which is a value to be newly input to the layer, and the first processing control unit, when it is determined by the first state comparison unit that the state of the object to be determined has not changed, instructs the first neural network unit not to execute processing in the layer of the object to be determined and subsequent layers, and instructs the first processing control unit to output the feature map stored in the first output storage unit; the second neural network unit, having a plurality of layers in the neural network, acquiring the feature map output by the feature extraction unit, and outputting a judgment value indicating the state of the object to be determined as the output data using the feature map; the second output storage unit, storing the judgment value output by the second neural network unit; the second state comparison unit, which stores in advance a reference value that is a reference for an input value to the layer for each of a plurality of layers of the second neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value that is a value to be newly input to the layer;and an abnormal state determination unit configured to include the second processing control unit, when the second state comparison unit determines that the state of the object to be determined has not changed, instructing the second neural network unit not to execute processing in the layer of the object to be determined and subsequent layers, and instructing the second neural network unit to output the determination value stored in the second output storage unit as the output data.
4. An abnormality determination device as claimed in any one of claims 1 to 3, characterized in that the reference value is a previous value stored for each layer as a result of the previous processing in the neural network.
5. The abnormality determination device according to any one of claims 1 to 3, characterized in that the reference value is a typical output value of a previously modeled intermediate layer in the neural network.
6. The abnormality determination device according to any one of claims 1 to 3, characterized in that the reference value is an output value of a predetermined node among the nodes included in the neural network.
7. The abnormality determination device according to claim 1, characterized in that the state comparison unit determines a change in the state of the object to be determined based on the result of determining the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value.
8. The abnormality determination device according to claim 7, characterized in that the state comparison unit determines that the state of the object to be determined has not changed if the sum of the squares of the difference values for each image is smaller than a pre-stored threshold value.
9. An abnormality determination device as described in claim 2 or claim 3, characterized in that at least one of the first state comparison unit and the second state comparison unit determines a change in the state of the object to be determined based on a result of determining the magnitude of the difference between the reference value and the current value using the difference value and a pre-stored threshold value.
10. The abnormality determination device as described in claim 9, characterized in that at least one of the first state comparison unit and the second state comparison unit determines that the state of the object to be determined has not changed if the sum of the squares of the difference values for each image unit is smaller than a pre-stored threshold value.
11. The abnormality determination device according to claim 7, characterized in that the state comparison unit determines that the state of the object to be determined has not changed when the sum of the absolute values of the difference values for each image is smaller than a pre-stored threshold value.
12. The abnormality determination device as described in claim 9, characterized in that at least one of the first state comparison unit and the second state comparison unit determines that the state of the object to be determined has not changed if the sum of the absolute values of the difference values for each image unit is smaller than a pre-stored threshold value.
13. An abnormality judgment device as described in any one of claims 1 to 12, further comprising an alarm output unit that acquires the output data, which is a judgment value indicating the drowsy state or dozing state of the person being judged, and outputs an alarm to the person being judged according to the judgment value.
14. A method for determining an anomaly, comprising: an image acquisition unit outputs an acquired image to a neural network unit; the neural network unit, having a plurality of layers in a neural network, acquires the image output by the image acquisition unit and outputs a determination result indicating the state of an object to be determined contained in the image; an output storage unit stores the output data output by the neural network unit; a state comparison unit pre-stores a reference value that is a standard for an input value to a first layer, which is at least one of the plurality of layers of the neural network unit, and determines a change in the state of the object to be determined using a difference value between the reference value and a current value, which is a value to be newly input to the first layer; and when the state comparison unit determines that the state of the object to be determined has not changed, a processing control unit instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs it to output the output data stored in the output storage unit.
Citation Information
Patent Citations
Information processor and monitoring device
JP1994139224A
Sorting Instances of Input Data for Processing through a Neural Network
US20200174748A1
Re-USING PROCESSING ELEMENTS OF AN ARTIFICIAL INTELLIGENCE PROCESSOR
US20210072984A1
Signal processing device, imaging device, and signal processing method
WO2022070947A1
