State determination device, state determination method, and recording medium

Abstract

The state determination device (100) comprises: an extraction unit (10) for extracting a facial region representing a region of a face from each of a plurality of frames sequentially obtained by photographing a user's face, extracting facial feature points representing a part of the face from the facial region, calculating a facial feature quantity extraction region based on the facial feature points, extracting a facial feature quantity as a feature quantity from the facial feature quantity extraction region, the facial feature quantity extraction region being a region in the facial region that changes when the user is in an unconscious state; a state determination unit (160) for determining whether the user is in an unconscious state based on the facial feature quantities of each of the plurality of frames and determination information generated in advance; and an output unit (170) for outputting a determination result.

Description

Technical Field

[0001] The present invention relates to a state determination device, a state determination method, and a recording medium having a state determination program recorded thereon. Background Art

[0002] A person may be in an unconscious state. For example, an unconscious state is a drowsy state or a state of being drowsy due to drinking.

[0003] In addition, in order to prevent accidents, technology for determining the unconscious state of a person in a car or a factory is being developed. For example, a feature quantity that changes when in an unconscious state is extracted from heart rate, brain waves, blinking, etc. Then, the feature quantity is compared with a threshold. The unconscious state is determined based on the comparison result. In this way, the unconscious state can be determined based on biological signals such as heart rate. However, in this determination method, a person wears a sensor. Therefore, in this determination method, the person feels troublesome. In addition, when a person does not wear a sensor, the determination method cannot be used. In addition, since this determination method uses a sensor, it is costly.

[0004] Here, a technology for detecting a drowsy state is proposed (see Patent Document 1). For example, a drowsy driving detection device of Patent Document 1 detects a driver's drowsy state by using the blinking frequency. In addition, a technology for determining a driver's sleepiness state is proposed (see Patent Document 2). For example, a sleepiness determination device of Patent Document 2 determines a sleepiness state by using the blinking frequency.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 11-339200

[0008] Patent Document 2: Japanese Patent Application Publication No. 2008-212298

[0009] Non-patent literature

[0010] Non-patent literature 1: Paul Viola, Michael J Jones "Robust Real-Time Face Detection International Journal of Computer Vision 57 (2)", 2004

[0011] Non-patent document 2: Laurenz Wiskott, Jean-Marc Fellous, Norbert Kruger, Christoph von der Malsburg "Face Recognition by Elastic Bunch Graph Matching", 1996

[0012] Non-patent document 3: Navneet Dalal and Bill Triggs "Histograms of Oriented Gradients for Human Detection" Summary of the invention

[0013] Problems to be solved by the invention

[0014] The techniques of Patent Documents 1 and 2 use blinking frequency. However, individual differences in blinking frequency are large. For example, in the case of a person who blinks very frequently when awake, it is difficult to use the techniques of Patent Documents 1 and 2 to determine whether the person is in an unconscious state.

[0015] Therefore, how to determine the unconscious state with high accuracy becomes a problem.

[0016] An object of the present invention is to determine an unconscious state with high accuracy.

[0017] Means for solving problems

[0018] A state determination device according to one embodiment of the present invention is provided. The state determination device comprises: an extraction unit that extracts a facial region representing a region of the face from each of a plurality of frames sequentially obtained by photographing the face of a user, extracts facial feature points representing a part of the face from the facial region, calculates a facial feature quantity extraction region based on the facial feature points, extracts a facial feature quantity as a feature quantity from the facial feature quantity extraction region, and the facial feature quantity extraction region is a region in the facial region that changes when the user is in an unconscious state; a state determination unit that determines whether the user is in the unconscious state based on the facial feature quantities of each of the plurality of frames and pre-generated determination information; and an output unit that outputs the determination result.

[0019] Effects of the Invention

[0020] According to the present invention, an unconscious state can be determined with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a diagram showing a state determination device according to the first embodiment.

[0022] Figure 2 This is a diagram showing the hardware configuration of the state determination device according to the first embodiment.

[0023] Figure 3 This is a diagram showing an example of a facial feature point table according to the first embodiment.

[0024] Figure 4 (A) and (B) are diagrams showing calculation examples of facial feature extraction regions.

[0025] Figure 5 This is a diagram showing an example of a facial feature scale according to the first embodiment.

[0026] Figure 6 This is a diagram showing an example of a state determination model table according to the first embodiment.

[0027] Figure 7 This is a diagram for explaining the method of calculating the number of times wrinkles occur between eyebrows according to the first embodiment.

[0028] Figure 8 This is a diagram showing a specific example of the unconsciousness level according to the first embodiment.

[0029] Fig. 9 This is a diagram showing an example of a determination result table according to the first embodiment.

[0030] Fig.10 1 is a flowchart showing the calculation process of the facial feature extraction region according to the first embodiment.

[0031] Fig.11 1 is a flowchart showing the facial feature value extraction process according to the first embodiment.

[0032] Fig.12 This is a flowchart showing the counting process in the first embodiment.

[0033] Fig.13 This is a flowchart showing the unconscious state determination process according to the first embodiment.

[0034] Fig.14 This is a functional block diagram showing the structure of the state determination device according to the second embodiment.

[0035] Fig.15 This is a diagram showing an example of an average facial feature point model table according to the second embodiment.

[0036] Fig.16 (A) to (C) are diagrams showing an example of a face condition table.

[0037] Fig.17 This is a diagram showing an example of an extraction region determination model table according to the second embodiment.

[0038] Fig.18 This is a flowchart showing the facial condition determination process of the second embodiment.

[0039] Fig.19 This is a flowchart showing the process of determining the facial feature extraction region according to the second embodiment. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments will be described with reference to the drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present invention.

[0041] Implementation Method 1

[0042] Figure 1 1 is a diagram showing a state determination device according to Embodiment 1. State determination device 100 is a device that executes a state determination method.

[0043] The state determination device 100 determines an unconscious state. For example, an unconscious state is a drowsy state, a state of being drowsy due to drinking, etc. In addition, an unconscious state includes a state of being hazy. For example, a state of being hazy is a state in which the user is drowsy, a state in which the user is drunk due to drinking, etc. Here, the user sometimes becomes drowsy again after temporarily becoming awake from a drowsy state. In such a case where the user becomes drowsy again after a short period of time from a drowsy state, it can also be considered that the user is in a drowsy state. Therefore, an unconscious state includes a case in which the user becomes drowsy again after temporarily becoming awake from a drowsy state.

[0044] Next, the hardware included in the state determination device 100 will be described.

[0045] Figure 2 This is a diagram showing a hardware configuration of the state determination device according to Embodiment 1. The state determination device 100 includes a processor 101 , a volatile storage device 102 , a nonvolatile storage device 103 , a camera 104 , and a display 105 .

[0046] The processor 101 controls the state determination device 100 as a whole. For example, the processor 101 is a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array). The processor 101 may be a multiprocessor. The state determination device 100 may be implemented by a processing circuit, or may be implemented by software, firmware, or a combination thereof. In addition, the processing circuit may be a single circuit or a composite circuit.

[0047] The volatile storage device 102 is a main storage device of the state determination device 100. For example, the volatile storage device 102 is a RAM (Random Access Memory). The nonvolatile storage device 103 is an auxiliary storage device of the state determination device 100. For example, the nonvolatile storage device 103 is a SSD (Solid State Drive).

[0048] The camera 104 is a device for photographing a face. In addition, the camera 104 is also called an imaging device. The display 105 is a device for displaying information. In addition, the display 105 is also called a display device.

[0049] In addition, the state determination device 100 can also be regarded as an information processing device when it does not have the camera 104 and the display 105.

[0050] return Figure 1 , the state determination device 100 is described.

[0051] The state determination device 100 includes an acquisition unit 110, an extraction unit 10, a state determination unit 160, and an output unit 170. The extraction unit 10 includes a face region extraction unit 120, a face feature point extraction unit 130, a face feature amount extraction region calculation unit 140, and a face feature amount extraction unit 150.

[0052] Furthermore, the state determination device 100 has a facial feature point storage unit 180 , a facial feature amount storage unit 181 , a state determination model storage unit 182 , and a determination result storage unit 183 .

[0053] Part or all of the extraction unit 10 , the acquisition unit 110 , the facial region extraction unit 120 , the facial feature point extraction unit 130 , the facial feature amount extraction region calculation unit 140 , the facial feature amount extraction unit 150 , the state determination unit 160 and the output unit 170 may be implemented by the processor 101 .

[0054] In addition, part or all of the extraction unit 10, the acquisition unit 110, the facial region extraction unit 120, the facial feature point extraction unit 130, the facial feature amount extraction region calculation unit 140, the facial feature amount extraction unit 150, the state determination unit 160, and the output unit 170 may be implemented as a module of a program executed by the processor 101. For example, the program executed by the processor 101 is also referred to as a state determination program. For example, the state determination program is stored in a recording medium such as the volatile storage device 102 or the non-volatile storage device 103.

[0055] The facial feature point storage unit 180 , the facial feature value storage unit 181 , the state determination model storage unit 182 , and the determination result storage unit 183 may be realized as storage areas reserved in the volatile storage device 102 or the nonvolatile storage device 103 .

[0056] The acquisition unit 110 acquires a plurality of frames sequentially acquired by photographing the user's face from the camera 104. The plurality of frames may be expressed as a dynamic image. A frame is an image. In addition, the plurality of frames may also be expressed as a plurality of frames obtained by photographing the user's face at different times.

[0057] The extraction unit 10 extracts a facial region from each of the plurality of frames, extracts facial feature points from the facial region, calculates a facial feature quantity extraction region based on the facial feature points, and extracts facial feature quantities from the facial feature quantity extraction region. The facial region refers to a region of the face. The facial feature points refer to a part of the face. The facial feature quantity extraction region is a region in the facial region that changes when the user is in an unconscious state. The facial feature quantity is a feature quantity.

[0058] Furthermore, the facial feature extraction region may be expressed as a region in which a change occurs in the facial region when the user is unconscious and in which no individual difference occurs in the facial movement of the user when the user is unconscious.

[0059] The processing performed by the extraction section 10 will be described in detail using the facial region extraction section 120 , the facial feature point extraction section 130 , the facial feature amount extraction region calculation section 140 , and the facial feature amount extraction section 150 .

[0060] The facial region extraction unit 120 extracts a facial region from a dynamic image. The facial region extraction unit 120 can be implemented using a classifier using Haar-like features based on Adaboost learning. For example, Non-Patent Document 1 describes a method for extracting a facial region.

[0061] The facial feature point extraction unit 130 extracts facial feature points such as the contour, eyebrows, eyes, nose, and mouth based on the facial region. For example, Non-Patent Document 2 describes a method for extracting facial feature points.

[0062] The facial feature point extraction unit 130 stores the facial feature points in the facial feature point storage unit 180. Here, the facial feature point storage unit 180 will be described. The facial feature point storage unit 180 stores a facial feature point table.

[0063] Figure 31 is a diagram showing an example of a facial feature point table according to Embodiment 1. The facial feature point table 180a is stored in the facial feature point storage unit 180. The facial feature point table 180a has items of feature points and facial orientation. In addition, the facial feature point table 180a has items of coordinates and angles.

[0064] For example, the coordinates of the inner end of the left eye extracted by the facial feature point extraction unit 130 are registered in the facial feature point table 180a. The inner end of the left eye is also called the left inner corner of the eye.

[0065] The facial feature point extraction unit 130 calculates the facial orientation based on the facial feature points. The facial orientation is represented by Yaw, Pitch, and Roll. The facial feature point extraction unit 130 registers the facial orientation in the facial feature point table 180a.

[0066] return Figure 1 , the facial feature extraction region calculation unit 140 will be described.

[0067] The facial feature quantity extraction region calculation unit 140 calculates a facial feature quantity extraction region used for determining an unconscious state.

[0068] Here, as an action performed when the user is in an unconscious state or as a precursor action of the user becoming unconscious, the user performs an action to resist drowsiness. The action to resist drowsiness refers to the action of the user intentionally closing his eyes. The action to resist drowsiness can also be manifested as a strong blink of the user intentionally closing his eyes. When the user intentionally closes his eyes, wrinkles are generated between the eyebrows. Therefore, the facial feature extraction area calculation unit 140 calculates the glabellar area in the facial area as the facial feature extraction area.

[0069] Furthermore, when the user is in an unconscious state due to drinking, etc., the user licks his lips to relieve thirst. Therefore, the facial feature quantity extraction region calculation unit 140 calculates the mouth region in the face region as the facial feature quantity extraction region.

[0070] Furthermore, the user yawns as an action performed when the user is in an unconscious state or as a premonitory action that the user becomes unconscious. When the user yawns, the mouth opens. In addition, when the user yawns, wrinkles are generated on the cheeks. Therefore, the facial feature extraction area calculation unit 140 calculates the mouth area and the cheek area in the face area as the facial feature extraction area.

[0071] Here, calculation of the glabellar region and calculation of the mouth region will be described as specific examples.

[0072] Figure 4 (A) and (B) are diagrams showing calculation examples of facial feature extraction regions. Figure 4(A) is an example of calculating the glabellar region. The facial feature quantity extraction region calculation unit 140 determines the left and right inner corners of the eye from the facial feature points extracted by the facial feature point extraction unit 130. The facial feature quantity extraction region calculation unit 140 calculates the middle point 200 between the left inner corner of the eye and the right inner corner of the eye. The facial feature quantity extraction region calculation unit 140 calculates a rectangular region 201 (i.e., a [pixel] × a [pixel] region) centered on the middle point 200. The rectangular region 201 is the glabellar region. In this way, the facial feature quantity extraction region calculation unit 140 calculates the glabellar region.

[0073] Furthermore, the facial feature quantity extraction region calculation unit 140 increases each side of the rectangular region 201 by k times based on the rectangular region as the facial region extracted by the facial region extraction unit 120. Therefore, the facial feature quantity extraction region calculation unit 140 can calculate a rectangular region of (a×k) [pixels]×(a×k) [pixels].

[0074] Here, sometimes the glabellar area cannot be accurately calculated according to the facial orientation. Therefore, the facial feature extraction area calculation unit 140 can calculate the glabellar area based on the facial orientation and the middle point calculated by the facial feature point extraction unit 130. Let me explain in detail. For example, suppose the face is facing left. When the face is facing left, the center of the area where wrinkles occur between the eyebrows is located to the left of the middle point between the left inner corner of the eye and the right inner corner of the eye. Therefore, the facial feature extraction area calculation unit 140 calculates the coordinates of the middle point moved parallel to the left by l [pixel] as the center coordinates of the glabellar area. The facial feature extraction area calculation unit 140 calculates a rectangular area centered on the center coordinates.

[0075] Figure 4 (B) is an example of calculating the mouth area. The facial feature amount extraction area calculation unit 140 determines the left and right corners of the mouth from the facial feature points extracted by the facial feature point extraction unit 130. The facial feature amount extraction area calculation unit 140 calculates the middle point 210 between the left corner of the mouth and the right corner of the mouth. The facial feature amount extraction area calculation unit 140 calculates a rectangular area 211 (i.e., an area of ​​b [pixels] × b [pixels]) centered on the middle point 210. The rectangular area 211 is the mouth area. In this way, the facial feature amount extraction area calculation unit 140 calculates the mouth area.

[0076] Furthermore, the facial feature quantity extraction region calculation unit 140 increases each side of the rectangular region 211 by k times based on the rectangular region extracted as the facial region by the facial region extraction unit 120. Therefore, the facial feature quantity extraction region calculation unit 140 can calculate a rectangular region of (b×k) [pixels]×(b×k) [pixels].

[0077] Furthermore, the facial feature extraction region calculation unit 140 may calculate the mouth region based on the facial orientation and facial feature points calculated by the facial feature point extraction unit 130. The calculation method is as described above.

[0078] The facial feature quantity extraction region calculation unit 140 can also calculate the cheek region.

[0079] return Figure 1 , the facial feature extraction unit 150 will be described.

[0080] The facial feature extraction unit 150 extracts facial features based on the glabellar region, the mouth region, and the cheek region. The facial features are HOG (Histograms of Oriented Gradients) features. For example, HOG features are described in Non-Patent Document 3.

[0081] Furthermore, the facial feature quantity may be a feature quantity other than the HOG feature quantity, for example, a SIFT (Scaled Invariance Feature Transform) feature quantity, a SURF (Speeded-up Robust Features), a Haar-like feature quantity, or the like.

[0082] The facial feature amount extraction unit 150 stores the facial feature amount in the facial feature amount storage unit 181. Here, the facial feature amount storage unit 181 will be described. The facial feature amount storage unit 181 stores a facial feature amount table.

[0083] Figure 5 1 is a diagram showing an example of a facial feature amount table according to Embodiment 1. The facial feature amount table 181a is stored in the facial feature amount storage unit 181. The facial feature amount table 181a has items of feature amounts and values. The facial feature amount extraction unit 150 registers the facial feature amount in the facial feature amount table 181a. That is, the facial feature amount extraction unit 150 registers information indicating the facial feature amount in the item of the facial feature amount in the facial feature amount table 181a. Furthermore, the facial feature amount extraction unit 150 registers the value corresponding to the facial feature amount in the item of the value in the facial feature amount table 181a.

[0084] In addition, the facial feature amount table 181a registers HOG feature amounts corresponding to n (n is an integer greater than or equal to 2) frames acquired in a predetermined time by the acquisition unit 110. The predetermined time is, for example, 5 minutes.

[0085] return Figure 1 , the state determination unit 160 will be described.

[0086] The state determination unit 160 determines whether the user is in an unconscious state based on the facial feature values ​​of each of the plurality of frames and the pre-generated determination information. In addition, the state determination unit 160 may also be expressed as: determining whether the user is in an unconscious state based on the facial feature values ​​corresponding to the plurality of frames and the pre-stored determination information.

[0087] The state determination unit 160 will be described in detail.

[0088] The state determination unit 160 determines the unconscious state based on the HOG feature amount. Specifically, the state determination unit 160 determines the unconscious state based on the number of times the user performs the above three behaviors within a specified time. In addition, the three behaviors are: the user's behavior of wrinkling between the eyebrows, the user's behavior of licking lips to relieve thirst, and the user's behavior of yawning. In addition, for example, the specified time is 5 minutes.

[0089] Here, information for determining an unconscious state is stored in the state determination model storage unit 182. In addition, before the state determination device 100 performs determination of an unconscious state, the information is stored in the state determination model storage unit 182 in advance.

[0090] This information is called a state determination model table. The state determination model table is described below.

[0091] Figure 6 1 is a diagram showing an example of a state determination model table according to Embodiment 1. The state determination model table 182a is stored in the state determination model storage unit 182. The state determination model table 182a has items such as the level of unconsciousness, the number of times wrinkles are generated between the eyebrows within 5 minutes, the number of times lips are licked within 5 minutes, and the number of times yawning is performed within 5 minutes.

[0092] The state determination model table 182a is also called determination information. In addition, the state determination model table 182a includes information for determining the level of unconsciousness corresponding to the number of times the user wrinkles between the eyebrows. The state determination model table 182a includes information for determining the level of unconsciousness corresponding to the number of times the user licks his lips. The state determination model table 182a includes information for determining the level of unconsciousness corresponding to the number of times the user yawns.

[0093] Here, a method for determining that wrinkles have occurred between the eyebrows by the state determination unit 160 will be described. The state determination unit 160 calculates the cosine similarity Sn using equation (1).

[0094]

[0095] The average value Hm is the average value of the HOG feature values ​​extracted by the facial feature value extraction unit 150 for a plurality of frames in the high wakefulness state (i.e., the normal state). In addition, the average value Hm is calculated in advance before the state determination device 100 performs the determination of the unconscious state. In addition, for example, the average value Hm is stored in the facial feature value storage unit 181.

[0096] The HOG feature value Hn is a HOG feature value corresponding to n frames acquired by the acquisition unit 110 at a predetermined time.

[0097] Here, a diagram showing an example of the relationship between the cosine similarity Sn and time will be described.

[0098] Figure 7 This is a diagram for explaining the method of calculating the number of times wrinkles occur between eyebrows according to the first embodiment. Figure 7 The vertical axis of the curve graph represents the cosine similarity Sn. Figure 7 The horizontal axis of the graph represents time. When wrinkles appear between the eyebrows, the edges of the wrinkles appear strongly. When the edges of the wrinkles appear strongly, the cosine similarity Sn becomes a smaller value.

[0099] The state determination unit 160 determines that wrinkles have occurred between the eyebrows when the cosine similarity Sn is a value smaller than a preset threshold value S. When determining that wrinkles have occurred between the eyebrows, the state determination unit 160 counts up the number of times wrinkles have occurred between the eyebrows.

[0100] The method of determining that wrinkles are generated between the eyebrows is the same as the method of determining that lip licking is the same. In addition, the method of determining that wrinkles are generated between the eyebrows is the same as the method of determining that yawning is the same. Therefore, the description of the method of determining that lip licking is the same as the method of determining that yawning is omitted.

[0101] In this way, the state determination unit 160 calculates the number of times the user wrinkles between the eyebrows based on the facial feature amount extracted from the eyebrow area of ​​each of the multiple frames. The state determination unit 160 calculates the number of times the user licks his lips based on the facial feature amount extracted from the mouth area of ​​each of the multiple frames. The state determination unit 160 calculates the number of times the user yawns based on the facial feature amount extracted from the mouth area and cheek area of ​​each of the multiple frames.

[0102] The state determination unit 160 determines the level of unconsciousness based on the number of times wrinkles are formed between the eyebrows and the state determination model table 182a. The state determination unit 160 determines the level of unconsciousness based on the number of times lip licking is performed and the state determination model table 182a. The state determination unit 160 determines the level of unconsciousness based on the number of times yawning is performed and the state determination model table 182a.

[0103] Figure 8 This is a diagram showing a specific example of the unconsciousness level according to the first embodiment. Figure 8 The level of unconsciousness determined according to the number of wrinkles generated between the eyebrows is shown to be level 2. Figure 8 The level of unconsciousness determined by the number of times of licking lips is shown to be level 4. Figure 8 It is shown that the level of drowsiness determined based on the number of yawns is level 3.

[0104] The state determination unit 160 calculates the average value of the three unconsciousness levels. For example, the state determination unit 160 calculates level 3 (=(2+4+3) / 3) as the average value. The state determination unit 160 may round off the average value to the nearest decimal point.

[0105] When the average value is greater than or equal to a preset threshold, the state determination unit 160 determines that the user is in an unconscious state. For example, the threshold is 3. In addition, this threshold is also called a threshold level. In this way, when the average value is greater than or equal to level 3, the state determination unit 160 determines that the user is in an unconscious state. In addition, it is possible to change which level is determined as an unconscious state according to the evaluation method.

[0106] The state determination unit 160 stores the determination result in the determination result storage unit 183. The determination result is an average value. Here, the determination result storage unit 183 will be described. The determination result storage unit 183 stores a determination result table.

[0107] Fig. 9 1 is a diagram showing an example of a determination result table according to Embodiment 1. The determination result table 183a is stored in the determination result storage unit 183. The determination result table 183a has an item of unconsciousness level. The state determination unit 160 registers the calculated average value in the determination result table 183a.

[0108] In addition, the state determination unit 160 may determine the unconscious state based on information obtained by machine learning using Random Forest, SVM (Support Vector Machine), Adaboost, CNN (Convolutional Neural Network), etc. and facial feature quantity. In addition, this information is also referred to as determination information. That is, the determination information is information obtained by machine learning and used to determine whether the user is in an unconscious state.

[0109] return Figure 1 , the output unit 170 will be described. The output unit 170 outputs the determination result. The output unit 170 will be described in detail.

[0110] The output unit 170 outputs the unconsciousness level registered in the determination result table 183a. For example, the output unit 170 outputs the unconsciousness level to the display 105. In addition, the output unit 170 may output the unconsciousness level by sound. In addition, the unconsciousness level registered in the determination result table 183a is also referred to as information indicating an average value.

[0111] The output unit 170 may output that the user is unconscious when the unconsciousness level registered in the determination result table 183a is level 3 or higher. The output unit 170 may output that the user is not unconscious when the unconsciousness level registered in the determination result table 183a is level 2 or lower.

[0112] Next, the processing executed by the state determination device 100 will be described using a flowchart.

[0113] Fig.10 FIG. 1 is a flowchart showing the calculation process of the facial feature extraction area according to Embodiment 1. Fig.10 In the calculation process of the glabellar region, the following is described. Fig.10 This is an example of the processing executed by the facial feature amount extraction region calculation unit 140 .

[0114] (Step S11 ) The facial feature extraction region calculation unit 140 obtains the coordinates of the left and right inner corners of the eyes and the facial orientation from the facial feature point storage unit 180 .

[0115] (Step S12) The facial feature quantity extraction region calculation unit 140 calculates the middle point between the left inner corner of the eye and the right inner corner of the eye.

[0116] (Step S13) The facial feature extraction region calculation unit 140 calculates the center coordinates based on the middle point and the face orientation. Specifically, the facial feature extraction region calculation unit 140 calculates the center coordinates as coordinates obtained by parallel shifting the middle point.

[0117] (Step S14) The facial feature quantity extraction region calculation unit 140 calculates a rectangular region centered on the central coordinates.

[0118] (Step S15 ) The facial feature quantity extraction region calculation unit 140 acquires the rectangular region as the facial region extracted by the facial region extraction unit 120 .

[0119] (Step S16) The facial feature extraction region calculation unit 140 changes the size of the rectangular region calculated in step S14 based on the rectangular region as the facial region. For example, the facial feature extraction region calculation unit 140 increases the size of each side of the rectangular region by k times based on the rectangular region as the facial region. Thus, the glabellar region corresponding to the facial size is calculated.

[0120] The facial feature quantity extraction region calculation unit 140 can calculate the mouth region and the cheek region through the same processing as described above.

[0121] Fig.11 1 is a flowchart showing the facial feature value extraction process according to the first embodiment.

[0122] (Step S21) The facial feature extraction unit 150 obtains the three facial feature extraction regions calculated by the facial feature extraction region calculation unit 140. That is, the three facial feature extraction regions are the glabellar region, the mouth region, and the cheek region.

[0123] (Step S22) The facial feature extraction unit 150 extracts HOG features from the glabella region. Furthermore, the facial feature extraction unit 150 extracts HOG features from the mouth region. Furthermore, the facial feature extraction unit 150 extracts HOG features from the cheek region.

[0124] (Step S23) The facial feature extraction unit 150 stores the HOG feature extracted from the glabella region in the facial feature storage unit 181. In addition, the facial feature extraction unit 150 stores the HOG feature extracted from the mouth region in the facial feature storage unit 181. Furthermore, the facial feature extraction unit 150 stores the HOG feature extracted from the cheek region in the facial feature storage unit 181.

[0125] Thus, the HOG feature quantities extracted from the three facial feature quantity extraction regions are registered in the facial feature quantity table 181 a .

[0126] Fig.12 : is a flowchart showing the counting process of implementation mode 1. Fig.12 In the following, the case of counting the number of times wrinkles occur between the eyebrows is described.

[0127] (Step S31 ) The state determination unit 160 obtains the extracted HOG feature value from the facial feature value storage unit 181 based on the glabellar region extracted from one frame (for example, referred to as the first frame).

[0128] (Step S32) The state determination unit 160 calculates the cosine similarity Sn using equation (1).

[0129] (Step S33 ) The state determination unit 160 determines whether the cosine similarity Sn calculated in step S32 is smaller than the threshold value S.

[0130] For example, if the cosine similarity Sn calculated in step S32 is smaller than the threshold value S, the wrinkle edge between the eyebrows is strong. In other words, if the cosine similarity Sn calculated in step S32 is smaller than the threshold value S, the user has wrinkled the eyebrows.

[0131] If the determination condition is satisfied, the state determination unit 160 advances the process to step S34. If the determination condition is not satisfied, the state determination unit 160 advances the process to step S35.

[0132] (Step S34) The state determination unit 160 counts up the number of times wrinkles occur between the eyebrows.

[0133] (Step S35) The state determination unit 160 determines whether 5 minutes have passed since the counting process was started. If 5 minutes have passed, the state determination unit 160 ends the process. If 5 minutes have not passed, the state determination unit 160 proceeds to step S31. In addition, for example, in step S31, the state determination unit 160 acquires the extracted HOG feature value from the facial feature value storage unit 181 based on the glabellar region extracted from the second frame acquired by the acquisition unit 110 after the first frame.

[0134] In this way, the state determination unit 160 obtains each HOG feature extracted from the glabellar region in n frames obtained by the acquisition unit 110 within 5 minutes. The state determination unit 160 determines whether wrinkles are generated between the brows based on each HOG feature. Thus, the state determination unit 160 can obtain the number of times the user has generated wrinkles between the brows within 5 minutes.

[0135] Through Fig.11 The number of times the user licks his lips is obtained by the same counting method as shown in FIG. 1 . For example, the glabellar region recorded in step S31 is converted into the mouth region. Thus, the state determination unit 160 can obtain the number of times the user licks his lips within 5 minutes.

[0136] In addition, through Fig.11 The number of yawns is obtained by the same counting method as shown. For example, the glabellar region recorded in step S31 is transformed into the mouth region and the cheek region. And for example, in step S33, the state determination unit 160 advances the processing to step S34 when the cosine similarity Sn calculated based on the HOG feature corresponding to the mouth region and the cosine similarity Sn calculated based on the HOG feature corresponding to the cheek region are less than a preset threshold. Thus, the state determination unit 160 can obtain the number of times the user yawns within 5 minutes.

[0137] In addition, the above-mentioned 5 minutes is an arbitrary time. Therefore, the time of 5 minutes may be a time other than 5 minutes.

[0138] Fig.13 This is a flowchart showing the unconscious state determination process according to the first embodiment.

[0139] (Step S41) The state determination unit 160 determines the level of unconsciousness based on the state determination model table 182a and the number of times wrinkles are generated between the eyebrows. The state determination unit 160 determines the level of unconsciousness based on the state determination model table 182a and the number of times lip licking is performed. The state determination unit 160 determines the level of unconsciousness based on the state determination model table 182a and the number of times yawning is performed.

[0140] (Step S42) The state determination unit 160 calculates an average value based on the three unconsciousness levels. The state determination unit 160 may round off the average value to the nearest decimal point.

[0141] (Step S43) The state determination unit 160 determines the unconscious state based on the average value. For example, when the average value is level 3 or higher, the state determination unit 160 determines that the user is in the unconscious state.

[0142] (Step S44 ) The state determination unit 160 stores the determination result in the determination result storage unit 183 .

[0143] According to Embodiment 1, the state determination device 100 determines whether the user is in an unconscious state based on the user's behavior of wrinkling the brows, the user's behavior of licking the lips, and the user's behavior of yawning. The three behaviors in the unconscious state have little or no individual difference. The state determination device 100 determines the unconscious state based on the user's behaviors with little or no individual difference, and thus can determine the unconscious state with high accuracy.

[0144] In addition, the state determination unit 160 may determine the unconsciousness level based on the number of times the user creates wrinkles between the eyebrows and the state determination model table 182a, and determine that the user is in an unconscious state when the determined unconsciousness level is greater than a preset threshold level. For example, the preset threshold level is level 3. The output unit 170 may also output the determined unconsciousness level.

[0145] In addition, the state determination unit 160 may determine the unconsciousness level based on the number of times the user licks his lips and the state determination model table 182a, and determine that the user is in an unconscious state when the determined unconsciousness level is above a preset threshold level. For example, the preset threshold level is level 3. The output unit 170 may also output the determined unconsciousness level.

[0146] In addition, the state determination unit 160 may determine the level of unconsciousness based on the number of times the user yawns and the state determination model table 182a, and determine that the user is in an unconscious state when the determined level of unconsciousness is greater than a preset threshold level. For example, the preset threshold level is level 3. The output unit 170 may also output the determined level of unconsciousness.

[0147] In addition, in the above, it is described that the state determination unit 160 determines whether the user is in an unconscious state based on the average value of three unconsciousness levels. The state determination unit 160 may also determine whether the user is in an unconscious state based on the average value of two unconsciousness levels. For example, the state determination unit 160 determines whether the user is in an unconscious state based on the average value of the unconsciousness level based on the number of times the user wrinkles between the eyebrows and the unconsciousness level based on the number of times the user licks his lips. And for example, the state determination unit 160 determines whether the user is in an unconscious state based on the average value of the unconsciousness level based on the number of times the user wrinkles between the eyebrows and the unconsciousness level based on the number of times the user yawns. And for example, the state determination unit 160 determines whether the user is in an unconscious state based on the unconsciousness level based on the number of times the user licks his lips and the unconsciousness level based on the number of times the user yawns.

[0148] Implementation Method 2

[0149] Next, Embodiment 2 will be described. Regarding Embodiment 2, the matters different from Embodiment 1 will be mainly described, and the description of matters similar to Embodiment 1 will be omitted. Figures 1 to 13 .

[0150] Fig.14 1 is a functional block diagram showing the configuration of a state determination device according to Embodiment 2. The state determination device 100 a includes an extraction unit 10 a. The extraction unit 10 a includes a face condition determination unit 191 and a face feature quantity extraction region determination unit 192.

[0151] Furthermore, the state determination device 100 a includes an average facial feature point model storage unit 184 , a facial condition storage unit 185 , and an extraction region determination model storage unit 186 .

[0152] A part or all of the extraction unit 10a, the face condition determination unit 191, and the face feature quantity extraction region determination unit 192 may be implemented by the processor 101. In addition, a part or all of the extraction unit 10a, the face condition determination unit 191, and the face feature quantity extraction region determination unit 192 may be implemented as a module of a program executed by the processor 101. For example, the program executed by the processor 101 is also called a state determination program.

[0153] The average facial feature point model storage unit 184 , the facial condition storage unit 185 , and the extraction region determination model storage unit 186 may be realized as storage areas reserved in the volatile storage device 102 or the nonvolatile storage device 103 .

[0154] To Figure 1 The same structure as shown Fig.14 The structure, annotation and Figure 1 The same reference numerals are shown.

[0155] The facial condition determination unit 191 determines whether the user is wearing something based on a plurality of frames. For example, when the feature points of the eyes are not extracted as facial feature points, the facial condition determination unit 191 determines that the user is wearing sunglasses. And for example, when the feature points of the mouth are not extracted as facial feature points, the facial condition determination unit 191 determines that the user is wearing a mask.

[0156] Furthermore, the facial condition determination unit 191 may determine whether the user is wearing an accessory as described below. First, the average facial feature point model storage unit 184 will be described. The average facial feature point model storage unit 184 stores an average facial feature point model table.

[0157] Fig.15 1 is a diagram showing an example of an average facial feature point model table according to Embodiment 2. The average facial feature point model table 184a is stored in the average facial feature point model storage unit 184. Before the state determination device 100a performs the determination of the unconscious state, the average facial feature point model table 184a is stored in the average facial feature point model storage unit 184. The average facial feature point model table 184a is also called average facial feature point model information.

[0158] The average facial feature point model table 184a has items of feature points and average coordinates. The average facial feature point model table 184a indicates the average position of facial parts. For example, the average position of the outer end of the left eyebrow of many people's faces is registered in the average facial feature point model table 184a as (100, 100).

[0159] The facial condition determination unit 191 uses the average facial feature point model table 184a and the facial feature points to determine whether the user is wearing anything. For example, when the distance between the feature point position of the left eye and the average coordinate position of the left eye (i.e., the outer end of the left eye and the inner end of the left eye) registered in the average facial feature point model table 184a is greater than a threshold, the facial condition determination unit 191 determines that the reliability of the feature point position of the left eye is low. In addition, the facial condition determination unit 191 determines that the user is wearing sunglasses. Similarly, when the distance between the feature point position of the mouth and the average coordinates of the mouth registered in the average facial feature point model table 184a is greater than a threshold, the facial condition determination unit 191 determines that the user is wearing a mask.

[0160] Here, the facial condition determination unit 191 uses the Euclidean distance or the Mahalanobis distance when comparing the information registered in the average facial feature point model table 184a with the facial feature points. For another example, when the facial condition determination unit 191 uses the Euclidean distance or the Mahalanobis distance, it makes the distance between the outer ends of the left eye and the right eye extracted by the facial feature point extraction unit 130 and the distance between the outer ends of the left eye and the right eye registered in the average facial feature point model table 184a the same. In this way, the facial condition determination unit 191 uses the distance between the outer ends of the left eye and the right eye. In addition, the facial condition determination unit 191 may also use a distance other than the distance between the outer ends of the left eye and the right eye. For example, the facial condition determination unit 191 uses the distance between the outer ends of the left eye and the right eyebrow.

[0161] In addition, the facial condition determination unit 191 determines whether a shadow is included in any frame of the n frames. Furthermore, the facial condition determination unit 191 determines whether there is a color splash in any frame of the plurality of frames. For example, the color splash refers to whitening. Whitening refers to the appearance of a white portion in a frame. For example, when light is irradiated from the left by illumination, the left portion in the frame has a color splash.

[0162] The facial condition determination unit 191 controls not to set the shadow or out-of-color region as the feature extraction region when the frame includes the shadow or out-of-color region. Thus, the facial feature extraction region calculation unit 140 can calculate an appropriate region as the facial feature extraction region.

[0163] Furthermore, the face condition determination unit 191 determines that the state is normal when the user is not wearing any accessory and when there is no region including shadows and flying colors in the frame.

[0164] The face condition determination unit 191 stores the determination result in the face condition storage unit 185. Here, the face condition storage unit 185 will be described. The face condition storage unit 185 stores a face condition table.

[0165] Fig.16(A) to (C) are diagrams showing an example of a face condition table. The face condition table 185a has items of face conditions.

[0166] Fig.16 (A) indicates a state in which the face condition determination unit 191 registers “normal time” in the face condition table 185 a . Fig.16 (B) indicates that the facial condition determination unit 191 has registered the state of “wearing a mask” in the facial condition table 185 a . Fig.16 (C) indicates a state in which the face condition determination unit 191 has registered “irradiation from the left” in the face condition table 185 a .

[0167] When determining based on the facial orientation that the left half of the face does not exist in the frame, the facial condition determination unit 191 may register “tilted face” in the facial condition table 185 a .

[0168] Next, the extraction region determination model storage unit 186 will be described. The extraction region determination model storage unit 186 stores an extraction region determination model table.

[0169] Fig.17 1 is a diagram showing an example of an extraction region determination model table of Embodiment 2. The extraction region determination model table 186a is stored in the extraction region determination model storage unit 186. Before the state determination device 100a performs the determination of the unconscious state, the extraction region determination model table 186a is stored in the extraction region determination model storage unit 186. The extraction region determination model table 186a has items of facial conditions and facial feature quantity extraction regions. The extraction region determination model table 186a is also called extraction region determination model information. The extraction region determination model table 186a is information indicating which region in the facial region is to be used as the facial feature quantity extraction region according to the position where the worn article is worn. In addition, the extraction region determination model table 186a is information indicating which region in the facial region is to be used as the facial feature quantity extraction region according to the position where a shadow or a flying color exists.

[0170] When it is determined that the user is wearing an accessory, the facial feature extraction region determination unit 192 determines the facial feature extraction region based on the extraction region determination model table 186a. Specifically, the facial feature extraction region determination unit 192 determines the facial feature extraction region based on the extraction region determination model table 186a and the facial condition determined by the facial condition determination unit 191.

[0171] Furthermore, when determining that a shadow or a scattered color exists in any frame of the plurality of frames, the facial feature quantity extraction region determining unit 192 determines a facial feature quantity extraction region based on the extraction region determination model table 186 a .

[0172] For example, when the facial condition determination unit 191 determines that the user is wearing a mask, the facial feature extraction region determination unit 192 determines the facial feature extraction region to be the glabella region. Here, when the user is wearing a mask, it is difficult for the facial feature extraction region calculation unit 140 to calculate the mouth region and the cheek region. Therefore, the facial feature extraction region determination unit 192 controls not to set the mouth region and the cheek region as the facial feature extraction region. Furthermore, the facial feature extraction region determination unit 192 determines the facial feature extraction region to be the glabella region.

[0173] The facial feature quantity extraction region calculation unit 140 calculates the facial feature quantity extraction region determined by the facial feature quantity extraction region determination unit 192 based on the facial feature points extracted by the facial feature point extraction unit 130 .

[0174] Next, the processing executed by the state determination device 100a will be described using a flowchart.

[0175] Fig.18 : is a flowchart showing the facial condition determination process of the second embodiment. Fig.18 This is an example of the processing performed by the face condition determination unit 191 .

[0176] (Step S51 ) The face condition determination unit 191 acquires the face feature points from the face feature point storage unit 180 .

[0177] (Step S52 ) The face condition determination unit 191 determines whether or not there is a shadow or a flying color in the frame.

[0178] For example, when the color of the area with a face in the frame is black, the facial condition determination unit 191 determines that there is a shadow in the frame. Also, when the color of the area with a face in the frame is white, or when the facial feature points of a certain area in the frame cannot be obtained, the facial condition determination unit 191 determines that there is a flying color in the frame. Fig.18 In the example above, it is assumed that shadows or flying colors are caused by light shining from the left through the lighting.

[0179] Furthermore, when the facial condition determination unit 191 has acquired the position information indicating the installation location of the lighting, it may determine whether or not there is a shadow or a scattered color in the frame based on the position information.

[0180] If there is a shadow or a flying color in the frame, the face condition determination unit 191 advances the process to step S53. If there is no shadow or a flying color in the frame, the face condition determination unit 191 advances the process to step S54.

[0181] (Step S53) The face condition determination unit 191 registers "irradiation from the left" in the face condition table 185a. Then, the face condition determination unit 191 ends the processing.

[0182] (Step S54) The facial condition determination unit 191 determines whether the user is wearing anything. For example, when no eye feature points are extracted as facial feature points, the facial condition determination unit 191 determines that the user is wearing anything.

[0183] When the user is wearing an accessory, the face condition determination unit 191 advances the process to step S56. When the user is not wearing an accessory, the face condition determination unit 191 advances the process to step S55.

[0184] (Step S55) The face condition determination unit 191 registers "normal time" in the face condition table 185a. Then, the face condition determination unit 191 ends the processing.

[0185] (Step S56 ) The facial condition determination unit 191 determines whether the user is wearing a mask. For example, when no feature points of the mouth are extracted as facial feature points, the facial condition determination unit 191 determines that the user is wearing a mask.

[0186] When the user is wearing a mask, the face condition determination unit 191 advances the process to step S57. When the user is not wearing a mask, the face condition determination unit 191 advances the process to step S58.

[0187] (Step S57) The facial condition determination unit 191 registers "wearing a mask" in the facial condition table 185a. Then, the facial condition determination unit 191 ends the processing.

[0188] (Step S58) The facial condition determination unit 191 determines whether the user is wearing sunglasses. For example, when no eye feature points are extracted as facial feature points, the facial condition determination unit 191 determines that the user is wearing sunglasses.

[0189] If the user is wearing sunglasses, the facial condition determination unit 191 advances the process to step S59. If the user is not wearing sunglasses, the facial condition determination unit 191 ends the process.

[0190] (Step S59) The facial condition determination unit 191 registers "wearing sunglasses" in the facial condition table 185a. Then, the facial condition determination unit 191 ends the processing.

[0191] Fig.19 This is a flowchart showing the process of determining the facial feature extraction region according to the second embodiment.

[0192] (Step S61) The facial feature extraction region determination unit 192 acquires the facial condition from the facial condition table 185a.

[0193] (Step S62) The facial feature extraction area determination unit 192 determines the facial feature extraction area based on the extraction area determination model table 186a and the facial condition. For example, when the facial condition is "wearing a mask", the facial feature extraction area determination unit 192 determines the facial feature extraction area to be the glabella area. And for example, when the facial condition is "wearing sunglasses", the facial feature extraction area determination unit 192 determines the facial feature extraction area to be the mouth area and the cheek area. And for example, when the facial condition is "irradiated from the left", the facial feature extraction area determination unit 192 determines the facial feature extraction area to be the glabella area, the mouth area, and the right cheek area. That is, the facial feature extraction area determination unit 192 does not set the left cheek area as the facial feature extraction area.

[0194] According to Embodiment 2, the state determination device 100a can determine an unconscious state using the extractable facial feature extraction area even when the user is wearing an accessory. Furthermore, the state determination device 100a can determine an unconscious state using the extractable facial feature extraction area even when the user is irradiated with light.

[0195] The features in the respective embodiments described above can be combined with each other as appropriate.

[0196] Description of symbols

[0197] 10, 10a: extraction unit; 100, 100a: state determination device; 101: processor; 102: volatile storage device; 103: non-volatile storage device; 104: camera; 105: display; 110: acquisition unit; 120: facial region extraction unit; 130: facial feature point extraction unit; 140: facial feature quantity extraction region calculation unit; 150: facial feature quantity extraction unit; 160: state determination unit; 170: output unit; 180: facial feature point storage unit; 180a: facial feature point table; 181: facial feature quantity storage unit; 181a: facial feature point table; Feature table; 182: state judgment model storage unit; 182a: state judgment model table; 183: judgment result storage unit; 183a: judgment result table; 184: average facial feature point model storage unit; 184a: average facial feature point model table; 185: facial condition storage unit; 185a: facial condition table; 186: extraction area determination model storage unit; 186a: extraction area determination model table; 191: facial condition judgment unit; 192: facial feature extraction area determination unit; 200: middle point; 201: rectangular area; 210: middle point; 211: rectangular area.

Claims

1. A state determination device, the state determination device comprising: an extraction unit that extracts a facial region representing a region of the face from each of a plurality of frames sequentially acquired by photographing the face of a user, extracts facial feature points representing a portion of the face from the facial region, calculates a midpoint based on the facial feature points, and uses a rectangular region centered on the midpoint as a facial feature quantity extraction region, and extracts a facial feature quantity as a feature quantity from the facial feature quantity extraction region, the facial feature quantity extraction region being a region in the facial region that changes when the user is in an unconscious state; a state determination unit for determining whether the user is in the unconscious state based on the facial feature amount of each of the plurality of frames and pre-generated determination information; as well as an output unit that outputs the determination result, The facial feature extraction area is a mouth area within the facial area, The determination information includes information for determining a level of unconsciousness corresponding to the number of times the user licks his lips, The state determination unit calculates the number of times the user licks his lips based on the facial feature amount extracted from the mouth area of ​​each of the multiple frames, determines the unconsciousness level based on the number of times the user licks his lips and the determination information, and determines that the user is in the unconscious state when the determined unconsciousness level is above a predetermined threshold level.

2. The state determination device according to claim 1, wherein: The output unit outputs the determined unconsciousness level.

3. The state determination device according to claim 1, wherein: The facial feature extraction area is the glabellar area and the mouth area in the facial area, The determination information includes information for determining an unconsciousness level corresponding to the number of times the user creates wrinkles between eyebrows, and information for determining an unconsciousness level corresponding to the number of times the user licks lips. The state determination unit calculates the number of times the user has wrinkled between the eyebrows based on the facial feature amount extracted from the eyebrow area of ​​each of the plurality of frames, and determines the unconsciousness level based on the number of times the user has wrinkled between the eyebrows and the determination information; calculates the number of times the user has licked his lips based on the facial feature amount extracted from the mouth area of ​​each of the plurality of frames, and determines the unconsciousness level based on the number of times the user has licked his lips and the determination information; The state determination unit calculates an average value of the determined plurality of unconsciousness levels, and determines that the user is in the unconscious state when the average value is equal to or greater than a preset threshold value.

4. The state determination device according to claim 1, wherein: The facial feature extraction area is the mouth area and cheek area in the facial area, The determination information includes information for determining an unconsciousness level corresponding to the number of times the user licks lips, and information for determining an unconsciousness level corresponding to the number of times the user yawns, The state determination unit calculates the number of times the user licks lips based on the facial feature amount extracted from the mouth region of each of the plurality of frames, and determines the level of unconsciousness based on the number of times the user licks lips and the determination information; calculates the number of times the user yawns based on the facial feature amount extracted from the mouth region and the cheek region of each of the plurality of frames, and determines the level of unconsciousness based on the number of times the user yawns and the determination information; The state determination unit calculates an average value of the determined plurality of unconsciousness levels, and determines that the user is in the unconscious state when the average value is equal to or greater than a preset threshold value.

5. The state determination device according to claim 1, wherein: The facial feature extraction area is the glabellar area, the mouth area, and the cheek area in the facial area. The determination information includes information for determining an unconsciousness level corresponding to the number of times the user creates wrinkles between eyebrows, information for determining an unconsciousness level corresponding to the number of times the user licks lips, and information for determining an unconsciousness level corresponding to the number of times the user yawns. The state determination unit calculates the number of times the user wrinkles between eyebrows based on the facial feature amount extracted from the glabellar region of each of the plurality of frames, and determines the unconsciousness level based on the number of times the user wrinkles between eyebrows and the determination information; calculates the number of times the user licks lips based on the facial feature amount extracted from the mouth region of each of the plurality of frames, and determines the unconsciousness level based on the number of times the user licks lips and the determination information; calculates the number of times the user yawns based on the facial feature amount extracted from the mouth region and the cheek region of each of the plurality of frames, and determines the unconsciousness level based on the number of times the user yawns and the determination information; The state determination unit calculates an average value of the determined plurality of unconsciousness levels, and determines that the user is in the unconscious state when the average value is equal to or greater than a preset threshold value.

6. The state determination device according to claim 3, wherein: The output unit outputs information indicating the average value.

7. The state determination device according to claim 4, wherein: The output unit outputs information indicating the average value.

8. The state determination device according to claim 5, wherein: The output unit outputs information indicating the average value.

9. The state determination device according to claim 1, wherein: The determination information is information obtained through machine learning and used to determine whether the user is in the unconscious state.

10. The state determination device according to any one of claims 1 to 9, wherein: The state determination device also has: a facial condition determination unit configured to determine whether the user is wearing an accessory based on the plurality of frames; as well as A facial feature extraction area determination unit determines the facial feature extraction area based on extraction area determination model information when it is determined that the user is wearing the accessory, wherein the extraction area determination model information indicates which area of ​​the facial area is to be used as the facial feature extraction area based on the position where the accessory is worn.

11. The state determination device according to claim 10, wherein: The facial condition determination unit determines whether the user is wearing the accessory based on average facial feature point model information indicating average positions of facial parts and the positions of the facial feature points.

12. The state determination device according to any one of claims 1 to 9, wherein: The state determination device also has: a face condition determination unit that determines whether a shadow or a flying color exists in any frame of the plurality of frames; as well as a facial feature quantity extraction region determining unit that, when it is determined that the shadow or the out-of-color exists in any frame of the plurality of frames, determines the facial feature quantity extraction region based on extraction region determination model information, the extraction region determination model information indicating which region of the facial region is to be used as the facial feature quantity extraction region according to a position where the shadow or the out-of-color exists.

13. A state determination device, the state determination device comprising: an extraction unit that extracts a facial region representing a region of the face from each of a plurality of frames sequentially acquired by photographing the face of a user, extracts facial feature points representing a portion of the face from the facial region, calculates a midpoint based on the facial feature points, and uses a rectangular region centered on the midpoint as a facial feature quantity extraction region, and extracts a facial feature quantity as a feature quantity from the facial feature quantity extraction region, the facial feature quantity extraction region being a region in the facial region that changes when the user is in an unconscious state; a state determination unit for determining whether the user is in the unconscious state based on the facial feature amount of each of the plurality of frames and pre-generated determination information; as well as an output unit that outputs the determination result, The facial feature extraction area is the glabellar area, the mouth area and the cheek area in the facial area, The determination information includes information for determining an unconsciousness level corresponding to the number of times the user creates wrinkles between eyebrows, and information for determining an unconsciousness level corresponding to the number of times the user yawns. The state determination unit calculates the number of times the user has wrinkled between eyebrows based on the facial feature amount extracted from the glabellar region of each of the plurality of frames, and determines the level of unconsciousness based on the number of times the user has wrinkled between eyebrows and the determination information; calculates the number of times the user has yawned based on the facial feature amount extracted from the mouth region and the cheek region of each of the plurality of frames, and determines the level of unconsciousness based on the number of times the user has yawned and the determination information; The state determination unit calculates an average value of the determined plurality of unconsciousness levels, and determines that the user is in the unconscious state when the average value is equal to or greater than a preset threshold value.

14. The state determination device according to claim 13, wherein: The output unit outputs information indicating the average value.

15. The state determination device according to claim 13, wherein: The determination information is information obtained through machine learning and used to determine whether the user is in the unconscious state.

16. The state determination device according to any one of claims 13 to 15, wherein: The state determination device also has: a facial condition determination unit configured to determine whether the user is wearing an accessory based on the plurality of frames; as well as A facial feature extraction area determination unit determines the facial feature extraction area based on extraction area determination model information when it is determined that the user is wearing the accessory, wherein the extraction area determination model information indicates which area of ​​the facial area is to be used as the facial feature extraction area based on the position where the accessory is worn.

17. The state determination device according to claim 16, wherein: The facial condition determination unit determines whether the user is wearing the accessory based on average facial feature point model information indicating average positions of facial parts and the positions of the facial feature points.

18. The state determination device according to any one of claims 13 to 15, wherein: The state determination device also has: a face condition determination unit that determines whether a shadow or a flying color exists in any frame of the plurality of frames; as well as a facial feature quantity extraction region determining unit that, when it is determined that the shadow or the out-of-color exists in any frame of the plurality of frames, determines the facial feature quantity extraction region based on extraction region determination model information, the extraction region determination model information indicating which region of the facial region is to be used as the facial feature quantity extraction region according to a position where the shadow or the out-of-color exists.

19. A state determination method, wherein: The state determination device extracts a facial region representing an area of ​​the face from each of a plurality of frames sequentially obtained by photographing the face of the user, extracts facial feature points representing parts of the face from the facial region, calculates a middle point based on the facial feature points, and uses a rectangular region centered on the middle point as a facial feature quantity extraction region, extracts a facial feature quantity as a feature quantity from the facial feature quantity extraction region, and the facial feature quantity extraction region is a region in the facial region that changes when the user is in an unconscious state. The state determination device determines whether the user is in the unconscious state based on the facial feature values ​​of each of the plurality of frames and pre-generated determination information, The state determination device outputs a determination result, The facial feature extraction area is a mouth area within the facial area, The determination information includes information for determining a level of unconsciousness corresponding to the number of times the user licks his lips, When determining whether the user is in the unconscious state, the number of times the user licks his lips is calculated based on the facial feature amount extracted from the mouth area of ​​each of the multiple frames, and the unconsciousness level is determined based on the number of times the user licks his lips and the determination information. When the determined unconsciousness level is above a pre-set threshold level, it is determined that the user is in the unconscious state.

20. A state determination method, wherein: The state determination device extracts a facial region representing an area of ​​the face from each of a plurality of frames sequentially obtained by photographing the face of the user, extracts facial feature points representing parts of the face from the facial region, calculates a middle point based on the facial feature points, and uses a rectangular region centered on the middle point as a facial feature quantity extraction region, extracts a facial feature quantity as a feature quantity from the facial feature quantity extraction region, and the facial feature quantity extraction region is a region in the facial region that changes when the user is in an unconscious state. The state determination device determines whether the user is in the unconscious state based on the facial feature values ​​of each of the plurality of frames and pre-generated determination information, The state determination device outputs a determination result, The facial feature extraction area is the glabellar area, the mouth area and the cheek area in the facial area, The determination information includes information for determining an unconsciousness level corresponding to the number of times the user creates wrinkles between eyebrows, and information for determining an unconsciousness level corresponding to the number of times the user yawns. In a case of determining whether the user is in the unconscious state, the number of times the user has wrinkled between the eyebrows is calculated based on the facial feature amount extracted from the eyebrow area of ​​each of the plurality of frames, and the unconsciousness level is determined based on the number of times the user has wrinkled between the eyebrows and the determination information; the number of times the user has yawned is calculated based on the facial feature amount extracted from the mouth area and the cheek area of ​​each of the plurality of frames, and the unconsciousness level is determined based on the number of times the user has yawned and the determination information, An average value of the determined plurality of unconsciousness levels is calculated, and when the average value is equal to or greater than a preset threshold value, it is determined that the user is in the unconscious state.

21. A recording medium having a state determination program recorded thereon, the state determination program causing a state determination device to execute the following processing: extracting a facial region representing an area of ​​the face from each of a plurality of frames sequentially obtained by photographing the face of the user, extracting facial feature points representing a part of the face from the facial region, calculating a middle point from the facial feature points, and using a rectangular region centered on the middle point as a facial feature quantity extraction region, extracting a facial feature quantity as a feature quantity from the facial feature quantity extraction region, the facial feature quantity extraction region being a region in the facial region that changes when the user is in an unconscious state, determining whether the user is in the unconscious state based on the facial feature values ​​of each of the plurality of frames and pre-generated determination information, Output the judgment result. The facial feature extraction area is a mouth area within the facial area, The determination information includes information for determining a level of unconsciousness corresponding to the number of times the user licks his lips, When determining whether the user is in the unconscious state, the number of times the user licks his lips is calculated based on the facial feature amount extracted from the mouth area of ​​each of the multiple frames, and the unconsciousness level is determined based on the number of times the user licks his lips and the determination information. When the determined unconsciousness level is above a pre-set threshold level, it is determined that the user is in the unconscious state.

22. A recording medium having a state determination program recorded thereon, the state determination program causing a state determination device to execute the following processing: extracting a facial region representing an area of ​​the face from each of a plurality of frames sequentially obtained by photographing the face of the user, extracting facial feature points representing a part of the face from the facial region, calculating a middle point from the facial feature points, and using a rectangular region centered on the middle point as a facial feature quantity extraction region, extracting a facial feature quantity as a feature quantity from the facial feature quantity extraction region, the facial feature quantity extraction region being a region in the facial region that changes when the user is in an unconscious state, determining whether the user is in the unconscious state based on the facial feature values ​​of each of the plurality of frames and pre-generated determination information, Output the judgment result. The facial feature extraction area is the glabellar area, the mouth area and the cheek area in the facial area, The determination information includes information for determining an unconsciousness level corresponding to the number of times the user creates wrinkles between eyebrows, and information for determining an unconsciousness level corresponding to the number of times the user yawns. In a case of determining whether the user is in the unconscious state, the number of times the user has wrinkled between the eyebrows is calculated based on the facial feature amount extracted from the eyebrow area of ​​each of the plurality of frames, and the unconsciousness level is determined based on the number of times the user has wrinkled between the eyebrows and the determination information; the number of times the user has yawned is calculated based on the facial feature amount extracted from the mouth area and the cheek area of ​​each of the plurality of frames, and the unconsciousness level is determined based on the number of times the user has yawned and the determination information, An average value of the determined plurality of unconsciousness levels is calculated, and when the average value is equal to or greater than a preset threshold value, it is determined that the user is in the unconscious state.

Images