Judgment device, judgment method, program, and recording medium

Abstract

This invention provides a determination device, determination method, program, and recording medium that improve the accuracy of determining whether a person's eyes are open or closed based on an image. [Solution] The determination device includes an image acquisition unit that acquires a person's face image sent from an imaging device connected via a network, an attribute acquisition unit that acquires attribute information of the person based on the face image acquired by the image acquisition unit, and a determination unit that determines whether the person's eyes are open or closed based on the attribute information of the person acquired by the attribute acquisition unit and the face image.

Description

Technical Field

[0001] It relates to a determination device, a determination method, a program, and a recording medium.

Background Art

[0002] Techniques for determining the opening and closing of eyes from an image obtained by photographing a person's face image are generally widely used. In a face authentication system, as a countermeasure against identity theft using a photo or the like, the authenticated person may be instructed to blink, and biometric authentication may be performed by checking whether the blink was successful from the video. Patent Document 1 discloses a technique for performing biometric authentication from a face image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to improve the accuracy of determining the opening and closing of a person's eyes from an image.

Means for Solving the Problems

[0005] In order to solve the above problems, a determination device according to an aspect of the present invention includes an image acquisition unit that acquires a face image of a person, an attribute acquisition unit that acquires the attribute information of the person based on the face image acquired by the image acquisition unit, and a determination unit that determines the opening and closing of the person's eyes based on the attribute information of the person acquired by the attribute acquisition unit and the face image.

Effects of the Invention

[0006] According to the present invention, the accuracy of determining the opening and closing of a person's eyes from an image can be improved. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing an example of the hardware configuration of a judgment device. [Figure 2] This is a block diagram showing an example of the functional configuration of a judgment device. [Figure 3] This is a conceptual diagram illustrating how the judgment device is used. [Figure 4] This is a box plot showing the distribution of eye opening / closing scores for each attribute. [Figure 5] This is a schematic diagram of a face and a graph showing the changes in eye opening / closing scores. [Figure 6] This is a flowchart explaining the process involved in facial recognition. [Figure 7] This figure shows examples of parameters for each attribute. [Figure 8] This is a flowchart explaining the process involved in facial recognition. [Figure 9] This is a flowchart explaining the process for performing biological identification. [Figure 10] This is a flowchart explaining the process for updating biological detection parameters. [Figure 11] This is a flowchart explaining the process for calculating the eye opening / closing score. [Figure 12] This is a conceptual diagram illustrating the process used to calculate the eye opening / closing score. [Figure 13] This figure shows examples of parameters for each attribute. [Figure 14] This figure shows examples of parameters for each attribute. [Figure 15] This is a flowchart explaining the process for performing biological identification. [Figure 16] This is a flowchart explaining the process for updating biological detection parameters. [Modes for carrying out the invention]

[0008] Hereinafter, embodiments for implementing the present invention will be described in detail with reference to the accompanying drawings. The embodiments described below are examples of means for realizing the present invention and should be appropriately modified or changed depending on the configuration of the apparatus to which the present invention is applied and various conditions. The present invention is not limited to the following embodiments. Also, some of the embodiments described later may be appropriately combined to form a configuration.

[0009] (Hardware Configuration) FIG. 1 is a diagram showing the hardware configuration of the determination apparatus according to the present embodiment. The determination apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, a hard disk drive (HDD) 104, a communication unit 105, an input device 106, and an output device 107. The communication unit 105 is connected to a network 108.

[0010] The CPU 101 is a processor that reads out control programs recorded in the ROM 102 and executes various processes. The ROM 102 is a non-volatile memory in which control programs for executing various processes of the determination apparatus are stored. The RAM 103 is a volatile memory that provides a temporary storage area such as a main memory and a work area to the CPU 101. The HDD 104 is a storage unit used for long-term data retention.

[0011] The communication unit 105 is a circuit that communicates via the network 108. The input device 106 is a device for inputting instructions and data to the determination apparatus 100 from the outside. Specifically, it includes a camera for acquiring images, a keyboard, a mouse, a touch panel, etc. for receiving user input. The output device 107 is a device for outputting instructions and data from the determination apparatus 100 to the outside. Specifically, it includes display devices such as a display for displaying face authentication / biometric determination results and information to the user, and interfaces for outputting face authentication / biometric determination results and unlocking signals at the time of success to external devices.

[0012] Note that when the determination device 100 is not functionally necessary, it does not necessarily need to include all the parts shown in FIG. 1. For example, when all input / output with the outside is performed using another device interconnected by the network 108, the input device 106 and the output device 107 are not necessary. Also, it may include parts not shown in FIG. 1. For example, video processing may be executed by a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array).

[0013] As described above, the hardware configuration of the determination device 100 has hardware configuration elements similar to those of the hardware configuration installed in a personal computer (PC). Therefore, various functions realized by the determination device 100 can be implemented as software operating on the PC. The determination device 100 can realize various functions of the determination device 100 and the flowchart described later when the CPU 101 executes a program.

[0014] (Functional configuration) FIG. 2 is a block diagram showing the functional configuration installed in the determination device according to the present embodiment. Among the blocks shown here, functions realized by software can be realized by the CPU 101 loading the computer program stored in the ROM 102 and executing it using the RAM 103 as a work area.

[0015] The determination device 100 includes an image acquisition unit 202, a biometric determination unit 203, a face authentication unit 204, and an attribute acquisition unit 205. The face authentication unit 204 includes a feature amount calculation unit 206 and a feature amount comparison unit 207. Also, the determination device 100 is connected to the imaging device 201 via the network 108. Note that the determination device 100 and the imaging device 201 may be connected via an input / output interface included in the input device 106 and the output device 107 instead of the network 108.

[0016] The imaging device 201 captures images of subjects to be subject to facial recognition and biometric determination, and sends the images to the determination device 100 via the network 108. The images can be videos in any format, such as Motion JPEG or H.264, or still images in any format, such as JPEG.

[0017] The image acquisition unit 202 acquires images from the imaging device 201 via the network 108.

[0018] The biological detection unit 203 determines whether the subject is a living organism based on the acquired image. The specific determination method in this embodiment will be described later using a flowchart.

[0019] The face recognition unit 204 performs face recognition processing based on the acquired image. The face recognition unit 204 includes a feature calculation unit 206 and a feature comparison unit 207. The feature calculation unit 206 calculates face features from the face image using a pre-trained Deep Learning model or the like. The feature comparison unit 207 compares two face features and outputs a matching score correlated with the similarity. The face recognition processing here also includes the process of sending the image acquired by the image acquisition unit 202 to the face recognition server connected to the determination device 100 via the network 108, and obtaining a matching score from the face recognition server. In other words, some or all of the feature calculation unit 206 and the feature comparison unit 207 may be configured as an external authentication server.

[0020] The attribute acquisition unit 205 acquires attribute information indicating the attributes of the person being authenticated. Any method can be used to acquire the attribute information. For example, the attribute acquisition unit 205 may acquire the age, gender, race, etc. of the person being authenticated as attribute information from a face image using a pre-trained Deep Learning model or the like. Another example is that the face recognition unit 204 provides the attribute acquisition unit 205 with the registration information of the person being authenticated identified by face recognition, and the attribute acquisition unit 205 may acquire the age, gender, race, etc. included in the registration information of the person being authenticated as attribute information. Yet another example is that the person being authenticated uses a recording medium capable of personal identification, such as an IC card, with the determination device 200. Specifically, an identifier that identifies the person is recorded on the IC card, and information including the face image necessary for face recognition and attribute information such as the person's age, gender, and race is pre-registered with the determination device 200, linked to the identifier. As a result, when the person being authenticated has the determination device 200 read the IC card, the determination device 200 identifies the person, and the attribute acquisition unit 205 can acquire the attribute information linked to the identifier.

[0021] Figure 3 is a conceptual diagram illustrating how the determination device 100 is used. The person to be authenticated 301 has the imaging device 201 take a picture of their face. The determination device 100 acquires the image of the user 301's face from the imaging device 201 via the network 108 and performs face authentication processing.

[0022] (Eye opening / closing detection) A common method for determining eye openness from an image involves using image processing or machine learning to output a score representing the degree of eye opening or the probability of the eyes being open, and determining that the eyes are closed if the output score falls below a threshold. However, the output score can vary depending on the attributes of the person being authenticated (e.g., age, gender, race), making it difficult to determine with a general-purpose threshold. For example, as people age, their eyelids tend to droop, resulting in narrower eyes and a lower score for older adults. Therefore, a threshold suitable for determining eye openness in children may incorrectly determine that an image of an elderly person's eyes is closed, even if the eyes are open.

[0023] On the other hand, one approach is to learn images of each person with their eyes open and closed, and then determine a threshold for determining whether the eyes are open or closed. However, determining the threshold requires multiple frames, which is time-consuming. This could increase the authentication time when applied to biometric authentication in facial recognition.

[0024] Regarding authentication time, one possible approach is to store the determined threshold value in association with the subject's personal information and retrieve it during subsequent authentication attempts, thereby reducing waiting times for subsequent judgments. However, this requires a configuration to store the personal information and threshold values ​​in association. In addition, the number of threshold values ​​to be remembered will need to increase as the number of registered individuals to be authenticated increases. Furthermore, since the degree of eye opening may change due to aging or various other factors in registered individuals, periodic updates of the threshold values ​​will also be necessary. Managing all of these aspects could become a significant burden for users.

[0025] Therefore, in the eye opening / closing determination of the determination device according to this embodiment, attribute information of the subject is estimated from the image, and a threshold value used for determining the opening / closing of the eyes is determined based on the correspondence between predetermined attribute information and the threshold value.

[0026] Figure 4 is a box plot showing the distribution of eye opening / closing scores by attribute for a dataset of facial images. Attributes include gender (male / female), age (child / adult / elderly), and race (East Asian, Caucasian, Black). Individuals with attributes such as male and elderly tend to have lower scores. This is attributed to a correlation between attributes and the shape of the eyes and eyelids. For example, elderly individuals tend to have drooping eyelids and narrower eyes. Furthermore, the method used to calculate the score here involves comparing a template image of eye opening and closing with an image of the subject's eyes. Therefore, the attributes of the person whose original facial image was used to create the template image can be one of the reasons for the differences in score levels shown in Figure 4. Similarly, even when calculating scores using machine learning, differences in the output score depending on the attributes may occur due to the characteristics of the training dataset.

[0027] Figure 5 shows an example of the progression of eye opening / closing scores for two individuals. (a) and (b) are pairs of graphs showing the face of a person and the progression of their eye opening / closing score calculated from each frame of the video. Person 501 has the attributes of male, adult, and East Asian. Person 503 has the attributes of male, elderly, and Caucasian. For person 501, the progression of the score calculated using the flowchart method shown in Figure 11 below is shown in graph 502. Person 501 blinks at frame number 14, resulting in a low score. Similarly, the progression of the score for person 503 is shown in graph 504. Person 503 blinks at frame numbers 9 and 18, resulting in a low score. Here, person 503 has a lower score level compared to person 501. Thus, because there are differences in the score levels among individuals, in order to accurately determine eye opening and closing based on the score, it is necessary to apply different thresholds to each individual. For example, looking at Graph 502, setting a threshold of around 0.50 for person 501 allows us to determine eye opening and closing from the score. However, looking at Graph 504, setting a threshold of around 0.50 for person 503 is not appropriate; around 0.45 is more appropriate. As can be seen in Figure 4, where the trend of scores differs depending on attributes, attributes can be related to the difference in score levels among people. By using thresholds that match the attributes of the subject, the accuracy of the judgment can be improved compared to making judgments with a uniform threshold.

[0028] (Facial recognition processing) Figures 6 and 8-11 are flowcharts showing the operations performed by the determination device according to the embodiment. These will be used to explain the flow of the biological information detection process. Note that the operations shown in the following flowcharts are realized by the CPU 101 loading a program stored in the ROM 102 and executing it using the RAM 103 as the work area. The determination device 100 does not necessarily have to perform all the steps described in these flowcharts.

[0029] Figure 6 is a flowchart illustrating the processes performed by the determination device 100 when a user requests facial recognition and the determination device 100 performs facial recognition and biometric determination.

[0030] After processing begins (for example, after the user invokes the facial recognition function), in step S601, the image acquisition unit 202 acquires the user's facial image captured by the imaging device 201.

[0031] In step S602, the feature calculation unit 206 calculates the feature quantities (input face features) of the acquired face image.

[0032] In step S603, the feature comparison unit 207 compares the input facial features calculated in step S602 with the registered facial features of each registered information previously registered in the facial recognition database and calculates a matching score. In this embodiment, it is assumed that the facial recognition database has previously registered information for each person, including registered facial features, name, gender, race, and date of birth.

[0033] In step S604, the facial recognition unit 204 determines whether the highest matching score among those calculated in step S603 is equal to or greater than a predetermined threshold. If it is equal to or greater than the threshold, the registration information with the facial feature that calculated the highest matching score is considered to be the user's registration information, and the process proceeds to step S605. If it is below the threshold, the process is considered an authentication failure, and the process proceeds to step S610.

[0034] In step S605, the attribute acquisition unit 205 acquires at least one attribute from the user's registration information, such as gender, age, or race. Age is calculated from the current date and the date of birth in the registration information, and the user is classified as a child if they are between 0 and 12 years old, an adult if they are between 13 and 59 years old, and an elderly person if they are 60 years or older.

[0035] In this embodiment, age is divided into three stages: child, adult, and elderly. However, this is not limited to this, and more detailed age divisions may be used to improve accuracy. Also, in this embodiment, East Asian, Caucasian, and Black are used as races, but this is not limited to this, and more detailed race classifications may be used. Furthermore, the method is not limited to this embodiment, and attributes other than gender, age, and race, or combinations thereof, may be used.

[0036] In step S606, the biological determination unit 203 determines biological determination parameters based on attribute information. Any method is possible for determining biological determination parameters based on attribute information. In this embodiment, a correspondence table between attribute information and biological determination parameters, as shown in Figure 7, is stored in advance, and the biological determination parameters are determined by referring to it. The correspondence table may be created by first creating a distribution of scores for eye opening and closing for each attribute information using a dataset, and then determining the parameter that yields the highest accuracy in determining eye opening and closing for each combination of attribute information based on that distribution. In this embodiment, the biological determination parameter is a threshold for the eye opening and closing score.

[0037] In step S607, the biometric determination unit 203 performs biometric determination processing based on whether or not the user blinked. The specific processing in this embodiment will be described later with reference to Figure 9.

[0038] In step S608, the biometric detection unit 203 checks whether the biometric detection in step S607 was successful. If the biometric detection is successful, the process proceeds to step S609. If the biometric detection fails, the process proceeds to step S610 as an authentication failure.

[0039] In step S609, the determination device 100 performs processing upon successful authentication. This processing is optional. For example, if the determination device 100 is used as part of a gate access control system, it can inform the user of successful authentication via the output device 106 and send an unlocking signal to the linked gate system.

[0040] In step S610, the determination device 100 performs processing in the event of authentication failure. The processing here is optional. For example, it is possible to inform the user of the authentication failure via the output device 106.

[0041] Figure 6 shows an example of authentication using a captured facial image by comparing it with facial images in a database, as used in access control. It can also be used in facial recognition systems for identity verification purposes, such as comparing a captured facial image with a facial image stored on an IC card or other device possessed by the user.

[0042] (A form of facial recognition processing) Figure 8 is a flowchart illustrating the processes executed by the determination device 100 when a user requests facial recognition and registers authentication information from an IC card or other device, etc., to the determination device 100 during facial recognition and biometric determination. Only the differences from Figure 6 will be explained.

[0043] In step S801, the user has the determination device 100 read their IC card or the like, and registers the stored facial image and attribute information, including registration information.

[0044] In step S602, the feature calculation unit 206 calculates the feature quantities of the face image acquired in step S601 (input face feature quantities). It also calculates the feature quantities of the face image registered in step S801 (registered face feature quantities).

[0045] In step S802, the feature comparison unit 207 compares the input face features calculated in step S602 with the registered face features also calculated in step S602, and calculates a matching score.

[0046] In step S604, the facial recognition unit 204 determines whether the matching score calculated in step S603 is equal to or greater than a predetermined threshold. If it is equal to or greater than the threshold, the registration information registered in step S801 is considered to belong to the user, and the process proceeds to step S605. If it is less than the threshold, the process is considered an authentication failure, and the process proceeds to step S610. The subsequent processing is the same as in Figure 6.

[0047] (Biometric identification process) Figure 9 is a flowchart illustrating the process when the biological determination unit 203 performs the biological determination process in step S607.

[0048] In step S901, the biometric detection unit 203 instructs the user to blink via the output device 106. For example, it displays a message such as "Please blink" on the display.

[0049] In step S902, the image acquisition unit 202 acquires a face image of the user's face captured by the imaging device 201.

[0050] In step S903, similar to step S801, the biometric determination unit 203 calculates an eye opening / closing score based on the facial image acquired in step S902. The eye opening / closing score is an evaluation value regarding the state of opening and closing of a person's eyes, and the specific process for calculating the eye opening / closing score will be described later with reference to Figure 11.

[0051] In step S904, the biometric determination unit 203 compares the score calculated in step S903 with the threshold determined as a biometric determination parameter in step S606. If the score is greater than or equal to the threshold, the process proceeds to step S905. If the score is less than the threshold, the process proceeds to step S906. In step S905, the biometric determination unit 203 determines that the user's eyes are open. In step S906, the biometric determination unit 203 determines that the user's eyes are closed.

[0052] In step S907, the biometric determination unit 203 determines whether the user successfully blinked. For example, if it was determined in step S906 that the user's eyes were closed, it may be determined that the blink was successful. Alternatively, the state of the user's eyes may be stored chronologically, and it may be determined that the blink was successful when the state transitioned from open → closed → open. If it is determined that the user successfully blinked, the process proceeds to step S909. If it is not determined that the user successfully blinked, the process proceeds to step S908. The biometric determination unit 203 may also store the eye opening / closing score, the user's attributes, and the determination result in association for use in updating the correspondence table of attribute information and biometric determination parameters shown in Figure 10.

[0053] In step S908, the biometric detection unit 203 checks whether the biometric detection process has timed out. For example, if the user has not successfully blinked within 10 seconds of being instructed to blink in step S901, a timeout may be set. If a timeout occurs, the process proceeds to step S910. If no timeout occurs, the process returns to step S902, acquires the next image, and performs the same biometric detection process.

[0054] In step S909, the biological detection unit 203 determines that the subject is a living organism, assuming that the user has successfully performed biological detection.

[0055] In step S910, the biometric determination unit 203 determines that the user did not succeed in biometric determination.

[0056] (Biological detection parameter update process) Figure 10 is a flowchart illustrating the process by which the biometric determination unit 203 updates the correspondence table between attribute information and biometric determination parameters. The biometric determination unit 203 may update the correspondence table between attribute information and biometric determination parameters at any time, for example, once a week, in order to improve accuracy.

[0057] In step S1001, the biometric determination unit 203 refers to the eye opening / closing score, user attribute information, and the eye opening / closing determination result saved in step S907 during the biometric determination process in step S607.

[0058] This allows us to obtain the distribution of scores for each attribute information in the actual authentication history when the eyes are open and when they are closed.

[0059] In step S1002, the biological determination unit 203 determines an appropriate threshold for each attribute piece of information. Any method can be used to determine the threshold. For example, the threshold may be a value exactly midway between the score level when the eyes are open (e.g., the average score for those judged as open) and the score level when the eyes are closed (similarly, the average score for those judged as closed).

[0060] In step S1003, the biological determination unit 203 overwrites the parameter table for each attribute information shown in Figure 7 with the threshold value determined in step S1002.

[0061] Figure 11 is a flowchart illustrating the process by which the biometric determination unit 203 calculates the eye opening / closing score based on the facial image in step S903. Figure 12 is a conceptual diagram of the eye opening / closing score calculation process.

[0062] In step S1101, the biometric detection unit 203 extracts an image of the region corresponding to the person's eyes from the face image to create an input image. More specifically, it cuts out an image of the eye region and creates an input image by adjusting the image size or upscaling it as needed. Figure 1201 shows an example of an input image.

[0063] In step S1102, the biometric detection unit 203 calculates HOG (Histgrams of Oriented Gradients) features for the input image. Figure 1204 in Figure 12 shows the HOG features calculated for the input image 1201.

[0064] In step S1103, the biodetector 203 acquires HOG features of the open-eye template image (first template image) and the closed-eye template image (second template image). The open-eye template image is a template image that the detection device 100 has in advance, showing an image of a person with their eyes open. Each template image can be stored in a storage unit such as the HDD 104. The open-eye template image can be created by any method. For example, a typical image of a person with their eyes open can be created using computer graphics, or it can be created based on an image of any person with their eyes open. Figure 1203 in Figure 12 is an example of an open-eye template image. Similarly, the closed-eye template image is a template image that shows an image of a person with their eyes closed, created by any method, and Figure 1201 in Figure 12 is an example of this. The biodetector 203 calculates HOG features for these. Figures 1205 and 1206 in Figure 12 show the HOG features calculated for 1202 and 1203, respectively.

[0065] The determination device 100 may not store open and closed eye template images, but instead store HOG features calculated from them. In that case, in step S1103, the biological determination unit 203 acquires the stored HOG features instead of calculating them from the template images.

[0066] In step S1104, the biological detection unit 203 calculates the Euclidean distance (D1) between the HOG features of the input image and the closed eye template image.

[0067] In step S1105, the biometric detection unit 203 calculates the Euclidean distance (D2) between the HOG features of the input image and the eye-open template image.

[0068] In step S1106, the biological determination unit 203 calculates the score S using the score calculation formula shown in equation (1).

number

[0069] When the eyes in the input image are open, D2 becomes small and D1 becomes large because the shape of the image is similar to the open eye template. As a result, S becomes large according to the scoring formula. Conversely, when the eyes in the input image are closed, S becomes small. Therefore, S is a score in which a larger value indicates a higher probability that the eyes in the input image are open.

[0070] Alternatively, multiple sets of open-eye and closed-eye template images may be prepared, a score may be calculated for each set, and the average of these scores may be used as the final score S. Furthermore, the score may be calculated for either the user's right eye or left eye, or it may be calculated for both and then averaged. As mentioned above, D1 and D2 are inversely related, so the evaluation value can be calculated by comparing the input image with either the open-eye template image or the closed-eye template image. In other words, either step S1104 or step S1105 may be omitted.

[0071] Furthermore, the process shown in Figure 11 is just one example of a process for calculating the eye opening / closing score, and in this invention, the eye opening / closing score may be calculated using other known methods. For example, it is possible to detect the upper and lower eyelids using a Deep Learning model capable of detecting facial organ points or an algorithm based on the brightness gradient of an image, and then calculate the eye opening / closing score based on the distance between the upper and lower eyelids. Alternatively, a learner that learns multiple images of eyes open and closed, as shown in 1202 and 1203, and outputs the likelihood that the input image is an image of eyes open as a score may be used. For example, this can be implemented using algorithms such as Deep Learning or SVM (Support Vector Machine).

[0072] As described above, evaluation values ​​regarding the open / closed state of a person's eyes are calculated using various algorithms. The biometric determination unit 203 uses the calculated evaluation value as a threshold and adjusts the threshold based on the attribute information of the person being determined.

[0073] In this embodiment, attribute information is obtained by referring to the registration information for facial recognition. However, in step S605, the attribute acquisition unit 205 can also estimate the user's attribute information based on the acquired facial image. Any known method can be used for attribute estimation. For example, the age, gender, race, etc. of the person being authenticated may be obtained as attribute information from the facial image using a pre-trained Deep Learning model. This allows the same method to be used in eye-open / close detection devices that do not involve facial recognition.

[0074] (Variation 1) In this embodiment, a method was described in which appropriate threshold scores are used for each attribute of the biometric information as biometric detection parameters. Here, a modified example is described in which appropriate open-eye and closed-eye template images are used for each attribute of the biometric information as biometric detection parameters.

[0075] In this modified version, suitable images are used for each attribute information as the eye open and closed template images (examples shown in Figures 1202 and 1203) used to calculate the eye open / closed score shown in Figure 11. This makes it possible to calculate a score that can accurately determine eye open / closed regardless of attribute information.

[0076] In step S606 of Figure 6, the biometric determination unit 203 determines biometric determination parameters based on attribute information. In this modified example, a correspondence table between attribute information and biometric determination parameters, as shown in Figure 14, is stored in advance, and the biometric determination parameters are determined by referring to it. In creating the correspondence table in Figure 14, a template image capable of suitably calculating a score for each attribute information is created by actually cutting out a facial image of the person for which that attribute information is located.

[0077] In step S1103 of Figure 11, template images corresponding to the attribute information determined in step S606 are selected as open and closed template images. More specifically, multiple template images corresponding to the attribute information are stored in a storage unit such as the HDD 104, and the biological determination unit 203 selectively reads out the template image to be used for determining the opening and closing of the eye based on the attribute information.

[0078] (Modification 2) In this embodiment, a method was described in which a threshold value for the eye opening / closing score appropriate for each attribute is used as a biological determination parameter. Here, a modified example is shown in which a coefficient of a correction formula that corrects the score is used as a biological determination parameter.

[0079] In this modified example, if the distribution of eye opening / closing scores for each attribute information obtained by the judgment device shown in Figure 4 is known, a correction formula is prepared for each attribute information so that the score distribution of each attribute information is roughly the same, and the correction formula corresponding to the user's attribute information is applied. In other words, by correcting the evaluation value (eye opening / closing score) based on the attribute information acquired by the attribute acquisition unit 205, the same effect as when the threshold is changed according to the attribute information can be obtained. A more detailed explanation follows below.

[0080] In step S606 of Figure 6, the biological determination unit 203 determines biological determination parameters based on attribute information. In this modified example, a correspondence table between attribute information and biological determination parameters, as shown in Figure 13, is stored in advance, and the biological determination parameters are determined by referring to it.

[0081] The table in Figure 13 was created by calculating coefficients a and b such that, based on the score distribution for each attribute information shown in Figure 4, the mean of S' for that attribute information becomes 0.55 and the standard deviation becomes 0.025 when the score S is corrected to the corrected score S' using the correction formula (2).

number

[0082] This is just one example of how to create a correspondence table between attribute information and biological judgment parameters. The correspondence table can be created in any way that allows for appropriate judgment using parameters for each attribute information. For example, based on the distribution of scores for each attribute information as shown in Figure 4, the correlation between attribute information and scores can be confirmed by calculating the correlation coefficient, and the coefficient of the score correction formula can be calculated from the attribute information using a predetermined mathematical formula based on that value.

[0083] Figure 15 is a flowchart illustrating the process when the biological determination unit 203 performs the biological determination process in step S606 in Modification 1. The processes other than S1501 and S1502 are the same as the processes in Figure 9.

[0084] In step S1501, the biological determination unit 203 corrects the eye opening / closing score obtained in step S903. Specifically, the score obtained in step S903 is denoted as S, and the corrected score S' is calculated based on the coefficients a and b determined as biological determination parameters in step S606 and equation (2).

[0085] In step S904, the biological determination unit 203 compares the corrected score calculated in S1501 with a predetermined threshold (for example, 0.50). If the score is equal to or greater than the threshold, the process proceeds to step S905. If the score is less than the threshold, the process proceeds to step S906.

[0086] Figure 16 is a flowchart illustrating the process when the biological determination unit 203 updates the biological determination parameters in Modification Example 1. The process in step S1001 is the same as the process in Figure 10, so its explanation is omitted.

[0087] In step S1601, the biological determination unit 203 determines the coefficients of the correction formula for correcting the score. Any method may be used to determine the coefficients. For example, the coefficients a and b of the correction formula may be determined from the distribution of scores with eyes open for each attribute information obtained in step S1001, such that the mean of the distribution of the corrected score S' is 0.55 and the standard deviation is 0.025.

[0088] In S1602, the biological determination unit 203 overwrites the table in Figure 13 with the coefficients a and b of the correction formula determined in S1601.

[0089] <Other Embodiments> The present invention can be realized by a process of reading and executing a program that implements one or more of the functions of Embodiment 1 described above. This program is supplied to a system or device via a network or storage medium, read and executed by one or more processors in the computer of that system or device. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0090] The disclosures herein include the following determination devices, determination methods, programs, and storage media.

[0091] (Item 1) An image acquisition unit that acquires facial images of people, An attribute acquisition unit acquires attribute information of the person based on the face image acquired by the image acquisition unit, A determination device comprising: a determination unit that determines whether a person's eyes are open or closed based on the attribute information of the person obtained by the attribute acquisition unit and the facial image.

[0092] (Item 2) The determination unit, Based on the attribute information acquired by the attribute acquisition unit, a determination parameter for determining whether the person's eyes are open or closed is determined. The determination device according to item 1, characterized in that it determines whether the person's eyes are open or closed based on the determination parameters and the facial image.

[0093] (Item 3) The determination device according to item 1, characterized in that the determination unit calculates an evaluation value regarding the open / closed state of the person's eyes based on the face image acquired by the image acquisition unit.

[0094] (Item 4) The determination device according to item 3, characterized in that the determination unit determines an evaluation value regarding the open / closed state of the person's eyes based on the face image acquired by the image acquisition unit and the attribute information acquired by the attribute acquisition unit.

[0095] (Item 5) The determination device according to item 4, characterized in that the determination unit determines the evaluation value by correcting the evaluation value calculated based on the face image acquired by the image acquisition unit based on the attribute information acquired by the attribute acquisition unit.

[0096] (Item 6) The aforementioned determination parameter includes a threshold value for the evaluation value, The determination device according to item 3, characterized in that the determination unit determines the threshold based on the attribute information acquired by the attribute acquisition unit.

[0097] (Item 7) The determination device according to item 6, characterized in that the determination unit determines the threshold based on the correlation coefficient between the attribute information and the evaluation value.

[0098] (Item 8) The determination unit, The image acquisition unit extracts an image of the region corresponding to the person's eyes from the face image acquired by the image acquisition unit. The determination device according to item 1, characterized in that it determines whether a person's eyes are open or closed based on at least one of a first template image, which is an image of the eyes in an open state, and a second template image, which is an image of the eyes in a closed state, and an image of the region corresponding to the eyes.

[0099] (Item 9) The system further includes a storage unit that stores a plurality of template images for at least one of the first template image and the second template image, The determination device according to item 8, characterized in that the determination unit selects at least one of the first template image and the second template image to be used for determining the opening and closing of the person's eyes from the storage unit, based on the attribute information acquired by the attribute acquisition unit.

[0100] (Item 10) The system further includes an authentication unit that authenticates whether the person is the registered person based on the facial image acquired by the image acquisition unit and the information about the registered person stored in the storage unit. The determination device according to item 1, characterized in that the attribute acquisition unit acquires attribute information from information relating to the registered person when the authentication unit determines that the person is the registered person.

[0101] (Item 11) The determination device according to any one of items 1 to 11, characterized in that the attribute information includes at least one of the person's age, gender, and race.

[0102] (Item 12) The image acquisition process involves obtaining a person's facial image, An attribute acquisition step, which acquires attribute information of the person based on the face image acquired by the image acquisition unit, A determination method characterized by comprising a determination step of determining whether the person's eyes are open or closed based on the person's attribute information obtained in the attribute acquisition step and the face image.

[0103] (Item 13) A program to cause a computer to execute the judgment method described in item 12.

[0104] (Item 14) A computer-readable storage medium containing the program described in item 13. [Explanation of symbols]

[0105] 100 Judgment device 201 Imaging device 302 Image acquisition unit 203 Biological Testing Unit 204 Facial Recognition Section 205 Attribute acquisition section 206 Feature Calculation Unit 207 Feature Comparison Section

Claims

1. An image acquisition unit that acquires facial images of people, An attribute acquisition unit acquires attribute information of the person based on the face image acquired by the image acquisition unit, A determination device comprising: a determination unit that determines whether a person's eyes are open or closed based on the attribute information of the person obtained by the attribute acquisition unit and the facial image.

2. The determination unit, Based on the attribute information acquired by the attribute acquisition unit, a determination parameter for determining whether the person's eyes are open or closed is determined. The determination device according to claim 1, characterized in that it determines whether the person's eyes are open or closed based on the determination parameters and the facial image.

3. The determination device according to claim 1, characterized in that the determination unit calculates an evaluation value regarding the open / closed state of the person's eyes based on the face image acquired by the image acquisition unit.

4. The determination device according to claim 3, characterized in that the determination unit determines an evaluation value regarding the open / closed state of the person's eyes based on the face image acquired by the image acquisition unit and the attribute information acquired by the attribute acquisition unit.

5. The determination device according to claim 4, characterized in that the determination unit determines the evaluation value by correcting the evaluation value calculated based on the face image acquired by the image acquisition unit based on the attribute information acquired by the attribute acquisition unit.

6. The determination device according to claim 3, characterized in that the determination unit determines a threshold value for the evaluation value based on the attribute information obtained by the attribute acquisition unit.

7. The determination device according to claim 6, characterized in that the determination unit determines the threshold based on the correlation coefficient between the attribute information and the evaluation value.

8. The determination unit, The image acquisition unit extracts an image of the region corresponding to the person's eyes from the face image acquired by the image acquisition unit. The determination device according to claim 1, characterized in that it determines whether a person's eyes are open or closed based on at least one of a first template image, which is an image of the eyes in an open state, and a second template image, which is an image of the eyes in a closed state, and an image of the region corresponding to the eyes.

9. The system further includes a storage unit that stores a plurality of template images for at least one of the first template image and the second template image. The determination device according to claim 8, characterized in that the determination unit selects at least one of the first template image and the second template image to be used for determining the opening and closing of the person's eyes from the storage unit based on the attribute information acquired by the attribute acquisition unit.

10. The system further includes an authentication unit that authenticates whether the person is the registered person based on the facial image acquired by the image acquisition unit and the information about the registered person stored in the storage unit. The determination device according to claim 1, characterized in that the attribute acquisition unit acquires attribute information from information relating to the registered person when the authentication unit determines that the person is the registered person.

11. The determination device according to any one of claims 1 to 10, characterized in that the attribute information includes at least one of the person's age, gender, and race.

12. The image acquisition process involves obtaining a person's facial image, An attribute acquisition step, which acquires attribute information of the person based on the face image acquired in the image acquisition step, A determination method characterized by comprising a determination step of determining whether the person's eyes are open or closed based on the person's attribute information obtained in the attribute acquisition step and the face image.

13. A program for causing a computer to execute the determination method described in claim 12.

14. A computer-readable storage medium storing the program described in claim 13.

Images