Terminal device, method, and program

Abstract

This terminal device comprises: a reflection image processing unit (192) that restores a reflection image of a display screen, the reflection image being reflected on a pupil from a captured face image of a user; a motion information acquisition unit (193) that acquires information about the motion of a user; and an authentication determination unit (194) that determines an authentication value by collating the reflection image with an image of the display screen when the face image of the user is captured, confirms the identity of the user when the authentication value is smaller than a predetermined authentication threshold, and determines the identity of the user on the basis of the information about the motion of the user when the authentication value is larger than the predetermined authentication threshold and smaller than a predetermined authentication allowable value that is different from the authentication threshold.

Description

Terminal device, method, and program

[0001] The present invention relates to a terminal device, a method, and a program.

[0002] Facial recognition technology is one of the authentication technologies for preventing fraudulent use. Facial recognition technology captures a subject's face with a camera and authenticates the subject by comparing the captured facial image with a comparison source image. In recent years, a problem has arisen in which a third party who is not the subject uses a photograph, video, or the like of the subject to impersonate the subject and perform authentication. In response to this problem, for example, Patent Document 1 discloses a technology that determines whether an image reflected in the subject's eyes is different from a specific image for authentication that was displayed on an image display means when the image was captured by an imaging means, and if it is determined to be different, determines that the person is an impersonation.

[0003] Japanese Patent Application Laid-Open No. 2007-072861

[0004] In the technology disclosed in Patent Document 1, a specific image for authentication is displayed on an image display means, and it is determined whether the specific image differs from the image reflected in the subject's eyes, and if it is determined that they differ, it is determined to be impersonation. Therefore, in order to determine impersonation, there is a problem in that it is necessary to display the specific image for authentication in addition to the image, video, etc. that are normally displayed on the image display means.

[0005] Furthermore, when using a terminal device, the subject may not always be facing the display screen, but may assume various postures, such as looking down or turning their entire face in a different direction. Furthermore, when the subject is holding a terminal device and looking at the display screen, the subject rarely holds the terminal device in the same position or at the same angle throughout the entire process. Therefore, depending on the subject's face, body orientation, and the position and angle of the terminal device, the specific image displayed on the image display unit may not be reflected in the subject's eyes. In this case, the technology disclosed in Patent Document 1 has the problem that even if the subject himself / herself is using the terminal device, the specific image displayed on the image display unit may not be reflected in the subject's eyes, meaning that the specific image for authentication cannot be detected from the image of the subject's eyes, which could result in the subject being identified as an impersonator.

[0006] The present invention solves the above-mentioned problems, and aims to provide a terminal device, method, and program that can stably authenticate a user without the need to display a specific image for authentication on an image display means.

[0007] In order to achieve the above object, the terminal device of the present invention comprises: a reflected image processing unit that restores a reflected image of the display screen reflected in the user's eyes from a photographed facial image of the user; a motion information acquisition unit that acquires motion information by the user; and an authentication determination unit that obtains an authentication value by comparing the reflected image with an image of the display screen at the time the facial image of the user was photographed, determines that the user is the user if the authentication value is smaller than a predetermined authentication threshold, and determines whether the user is the user based on the motion information by the user if the authentication value is larger than the predetermined authentication threshold and smaller than a predetermined authentication tolerance value different from the authentication threshold.

[0008] According to the terminal device of the present invention, a user can be authenticated based on the image normally displayed on the screen and the user's action information, so that the user can be stably authenticated without the need to display a specific image for authentication on the image display means.

[0009] Diagram showing the outline of the usage state of the user's terminal device according to Embodiment 1 Diagram for explaining the position of the image reflected in the pupil when the user using the terminal device shown in FIG. 1 faces forward Diagram for explaining the position of the image reflected in the pupil when the user shown in FIG. 2A moves only the eyeball in the -X axis direction Diagram for explaining the position of the image reflected in the pupil when the user shown in FIG. 2A moves the entire face in the +Z axis direction Diagram for explaining the position of the image reflected in the pupil when the user shown in FIG. 2A moves the entire face in the +X axis direction Diagram for explaining the position of the image reflected in the user's pupil when the inclination of the terminal device shown in FIG. 1 is changed Diagram showing the outline of the terminal device according to Embodiment 1 Diagram showing the configuration of the terminal device shown in FIG. 3 Diagram showing an example of the hardware configuration of the terminal device shown in FIG. 3 Diagram of the information processing block of the terminal device shown in FIG. 3 Diagram showing the reflected image authentication value table stored in the terminal storage unit of the terminal device shown in FIG. 4 Diagram showing the table of the authentication operation information database acquired from the terminal device shown in FIG. 4 Diagram showing the authentication count table stored in the terminal storage unit of the terminal device shown in FIG. 4 Diagram showing the standby time setting table stored in the terminal storage unit of the terminal device shown in FIG. 4 Diagram for explaining the outline of the authentication method according to Embodiment 1 Flowchart of the authentication process according to Embodiment 1 Continued flowchart of the flowchart of the authentication process shown in FIG. 9A Flowchart of the reflected image authentication value acquisition process according to Embodiment 1 Flowchart of the operation information authentication process according to Embodiment 1 Diagram showing the outline of the terminal device according to Embodiment 2 Diagram showing the combined authentication count table stored in the terminal storage unit of the terminal device shown in FIG. 12 Diagram for explaining the outline of the authentication method according to Embodiment 2 Flowchart of the combined authentication process according to Embodiment 2 Continued flowchart of the flowchart of the combined authentication process shown in FIG. 15A Diagram showing the configuration of the terminal device according to Embodiment 3 Diagram showing the reflected image authentication value table according to Embodiment 3 Flowchart of the linked authentication process according to Embodiment 3 Continued flowchart of the flowchart of the linked authentication process shown in FIG. 18A Flowchart of the linked reflected image authentication value acquisition process according to Embodiment 3 Flowchart of the linked authentication value setting process according to Embodiment 3

[0010] A terminal device, a method, and a program according to an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the same or corresponding parts in the drawings are designated by the same reference numerals.

[0011] The terminal device 1 of this embodiment 1 restores the image of the display screen reflected in the eyes from the facial image of the user using it, and authenticates the user U by comparing it with the image of the display screen at the time the user's facial image was captured.If the authentication based on the reflected image R determines that the image is gray, the terminal device 1 can authenticate the user U based on the behavior information of the user U detected by the terminal device 1.

[0012] As shown in FIG. 1, a user U holds the terminal device 1 in his left hand H. Ｌ While holding it, right hand H Ｒ When operating with the right eye E of the user U, Ｒ Right pupil P Ｒ And left eye E Ｌ Left pupil P Ｌ The right pupil P Ｒ and left pupil P Ｌ The left hand H is the black part of the eye including the iris. Ｌ While holding it, right hand H Ｒ These are collectively called the hand H, and the right eye E Ｒ and left eye E Ｌ are collectively called eye E, and the right pupil P Ｒ and left pupil P Ｌ are collectively referred to as pupil P. In the following description, an XYZ Cartesian coordinate system is set up, with the width direction of user U shown in Figure 1 as the X-axis direction, the height direction as the Z-axis direction, and the direction perpendicular to the X-axis and Z-axis directions as the Y-axis direction, and this system will be referred to as appropriate. In the following description, the direction in which the arrows on each coordinate axis travel is referred to as the + direction, and the direction opposite to the arrow's travel is referred to as the - direction.

[0013] When a user U holds a terminal device 1 in their hand H and looks at the display screen, an image displayed on the display screen of the terminal device 1 is reflected in the pupil P of their eye E. Hereinafter, the image displayed on the display screen of the terminal device 1 that is reflected in the pupil P is referred to as a reflected image R. For example, when the user U shown in FIG. 1 holds the terminal device 1 and looks at the display screen, the reflected image R is reflected in the pupil P of the user U's eye E so as to cover the entire pupil P, as shown in FIG. 2A. Here, if the user U moves only their eyeball in the −X-axis direction without moving their face, for example, the pupil P moves in the −X-axis direction, and part of the reflected image R is reflected in the pupil P, as shown in FIG. 2B.

[0014] Also, suppose that user U moves, for example, their entire face in the +Z-axis direction from the state shown in Fig. 2A. In this case, as shown in Fig. 2C, the reflected image R moves in the -Z-axis direction and is reflected in the part of the pupil P on the -Z-axis side. Also, suppose that user U moves, for example, their entire face in the +X-axis direction from the state shown in Fig. 2A. In this case, as shown in Fig. 2D, the reflected image R moves in the -X-axis direction and is reflected in the part of the pupil P on the -X-axis side.

[0015] Also, assume that the user U changes the tilt of the terminal device 1 that he or she is holding in his or her hand H. In this case, the angle between the display screen and the pupil P of the user U changes according to the tilt of the terminal device 1, and therefore the shape of the reflected image R reflected in the pupil P changes. For example, as shown in FIG. 2E , the height of the reflected image R in the Z-axis direction decreases, and the shape changes from a rectangle to a trapezoid, becoming a deformed reflected image Rm, which is reflected in a position on the −Z-axis direction of the pupil P.

[0016] In this way, the shape and position of the reflected image R reflected in the pupil P of the user U vary depending on the position of the face and eyeballs of the user U and the tilt of the terminal device 1 when the user U is looking at the display screen of the terminal device 1. In some cases, as shown in Fig. 2D , the reflected image R may not be reflected in the pupil P. In this case, it is difficult to authenticate the user U by comparing the reflected image R with the image normally displayed on the display screen.

[0017] Therefore, in the first embodiment, the user U is authenticated based on the reflected image R, and even if the user U cannot be authenticated based on the reflected image R, the user U can be authenticated based on the operation information of the user U detected by the terminal device 1. As a result, even if the user U cannot be authenticated based on the reflected image R, the terminal device 1 is not immediately locked so that it cannot be used, and the user U can use the terminal device 1 stably.

[0018] Furthermore, the authentication process for user U performed in terminal device 1 is executed in the background while terminal device 1 is operating, and updates the authentication value for authenticating reflected image R and the information on the operation of user U. This improves the accuracy of authentication, preventing unauthorized use by others, such as impersonation, and allowing user U to use terminal device 1 safely.

[0019] 3 shows a front view of the terminal device 1 in the first embodiment. The terminal device 1 is a so-called smartphone and includes an in-camera 11A on the front side that captures the face of a user U, a speaker 12A, a microphone 12B for calls, a tilt detection unit 13 that detects the tilt of the terminal device 1, a touch panel that also serves as an operation input unit 14 and a display unit 20, a left fingerprint sensor 15A and a right fingerprint sensor 15B that detect the fingerprint of the user U, a position detection unit 16 that detects the current position of the terminal device 1, and a right pressure detection unit 17A and a left pressure detection unit 17B that detect the pressure with which the user U grips the terminal device 1. The terminal device 1 also includes a main camera 11B on the back side that can capture images of people, landscapes, objects, etc. as seen from the user U's perspective.

[0020] Hereinafter, the in-camera 11A and the main camera 11B will be collectively referred to as the imaging unit 11. Hereinafter, the speaker 12A and the microphone 12B, which is a microphone for calls, will be collectively referred to as the audio input / output unit 12. Hereinafter, the left fingerprint sensor 15A and the right fingerprint sensor 15B will be collectively referred to as the fingerprint detection unit 15. Hereinafter, the right pressure detection unit 17A and the left pressure detection unit 17B will be collectively referred to as the pressure detection unit 17.

[0021] 4 is a block diagram showing the configuration of the terminal device 1. The terminal device 1 includes a communication unit 10, an imaging unit 11, an audio input / output unit 12, a tilt detection unit 13, an operation input unit 14, a fingerprint detection unit 15, a position detection unit 16, a pressure detection unit 17, a terminal storage unit 18, a terminal control unit 19, and a display unit 20.

[0022] The communication unit 10 includes a data communication unit that communicates with an external server, cloud, etc. via a communication network (not shown) and transmits and receives various data, and a voice communication unit that transmits and receives radio signals for telephone communication with a base station (not shown). The data communication unit can be configured using a wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The voice communication unit can be configured using a communication device that transmits and receives radio signals for telephone communication with a base station.

[0023] The imaging unit 11 includes an in-camera 11A and a main camera 11B shown in Fig. 3. The imaging unit 11 may be any of various cameras capable of capturing still images or videos and acquiring the captured still images or videos, such as a camera using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a video camera.

[0024] The audio input / output unit 12 includes a speaker 12A and a microphone 12B shown in Fig. 3. The speaker 12A outputs audio received during a voice call, music data acquired from an external device via a communication network, etc. The microphone 12B is a device that picks up the audio of the user U.

[0025] The tilt detection unit 13 is a device that can detect the tilt, shaking, etc. of the terminal device 1. The tilt detection unit 13 can be configured using various sensors that can detect the tilt of the terminal device 1, such as an acceleration sensor, an angle sensor, and a magnetic sensor that detects geomagnetism. The number and types of sensors that configure the tilt detection unit 13 may be either single or multiple.

[0026] The operation input unit 14 is a device that can input operations from the user shown in Fig. 3. The fingerprint detection unit 15 is a sensor that detects the fingerprint of the user U. The fingerprint detection unit 15 includes a left fingerprint sensor 15A and a right fingerprint sensor 15B shown in Fig. 3. Note that the fingerprint detection unit 15 is not limited to a fingerprint sensor, and any sensor, device, etc. that can detect the fingerprint of the user U may be used.

[0027] The position detection unit 16 is a device that can detect the current position of the terminal device 1. The position detection unit 16 can be configured using a device that can detect the current position of the terminal device 1, such as a GPS (Global Positioning System). The pressure detection unit 17 is a sensor that detects the pressure with which the user U grips the terminal device 1. The pressure detection unit 17 includes a right pressure detection unit 17A and a left pressure detection unit 17B shown in FIG. 3 . Note that the pressure detection unit 17 is not limited to a pressure sensor, and any sensor, device, or the like that can detect the pressure with which the user U grips the terminal device 1 may be used.

[0028] The terminal storage unit 18 includes an authentication processing program 180 for performing authentication processing for the user U, a reflected image processing program 181 for restoring the reflected image R reflected in the pupil P of the user U, a reflected image authentication value table 182 that stores authentication values ​​for authenticating the reflected image R, an authentication motion information database 183 that compiles motion information of the user U acquired by the terminal device 1, an authentication count table 184 that sets the number of authentication failures and the number of judgments, and a wait time setting table 185 that sets the wait time until the authentication processing is executed. The terminal storage unit 18 also stores programs for various applications that are executed by the terminal device 1.

[0029] The authentication processing program 180 is a program that performs processing to authenticate the user U based on the reflected image R reflected in the pupil P of the user U and the user's motion information acquired by the terminal device 1. The reflected image processing program 181 is a program that performs processing to detect the eyes E from the facial image of the user U captured by the imaging unit 11 and restore the reflected image R reflected in the pupil P. The reflected image authentication value table 182 is a table that stores the authentication value for authenticating the reflected image R in the authentication processing program 180, the authentication threshold value for judgment, and the authentication tolerance value.

[0030] The authentication motion information database 183 is a database for storing motion information of the user U who operates the terminal device 1, conditions for passing authentication, etc. Here, the motion information of the user U includes the pressure with which the user U holds the terminal device 1, the angle of inclination of the terminal device 1, the distance between the screen of the display unit 20 and the user's face, the number of connections to a specific communication network / device, etc.

[0031] The authentication count table 184 is a table that sets the number of authentication failures based on the reflected image R, the number of failures based on the user U's motion information, and the number of judgments for determining whether authentication is unsuccessful based on the number of failures. The waiting time setting table 185 is a table that sets the waiting time until the next execution of the authentication process, which is repeatedly executed. The reflected image authentication value table 182, the authentication motion information database 183, the authentication count table 184, and the waiting time setting table 185 will be described in detail below.

[0032] The terminal control unit 19 executes various programs stored in the terminal storage unit 18. The terminal control unit 19 also acquires and processes various data from the communication unit 10, the photographing unit 11, the voice input / output unit 12, the tilt detection unit 13, the operation input unit 14, the fingerprint detection unit 15, the position detection unit 16, and the pressure detection unit 17, and stores the data in various databases and tables in the terminal storage unit 18. The terminal control unit 19 can also cause the photographing unit 11 to photograph the face of the user U at any timing by sending an instruction to the photographing unit 11 to photograph the face.

[0033] The display unit 20 displays the processing contents of various programs executed by the terminal control unit 19. The display unit 20 can also display images such as still images and videos captured by the imaging unit 11, and data input from the operation input unit 14. The display unit 20 is layered on the operation input unit 14, and forms the touch panel shown in FIG.

[0034] Next, an example of the hardware configuration of the terminal device 1 will be described with reference to Fig. 5. The terminal device 1 includes a processor 31 that executes various programs, a memory 32 for expanding the various programs, a display controller 33 that outputs various display data, a display device 34 that displays the various display data, an I / O port 35 for connecting the imaging unit 11, the audio input / output unit 12, etc., a storage device 36 that stores various programs and various data, and a communication device 37 that communicates with the outside and transmits and receives various data. The processor 31, memory 32, display controller 33, display device 34, I / O port 35, storage device 36, and communication device 37 are interconnected via a data bus 38.

[0035] The processor 31 reads various programs stored in the storage device 36, loads them into the memory 32, and executes them. The processor 31 can be configured using a processing device such as a CPU (Central Processing Unit) or an MPU (Micro-processing Unit). The memory 32 can be configured using storage elements and storage media such as RAM (Random Access Memory) and volatile or non-volatile semiconductor memory such as flash memory.

[0036] The display controller 33 is a controller that outputs various display data to the display device 34. The display controller 33 can be configured using a video signal output device such as a video card, a GPU (Graphics Processing Unit), or a graphics board. The display device 34 can be configured using a display device such as an LCD (Liquid Crystal Display), an organic EL (Electroluminescence) monitor, or the like.

[0037] The I / O port 35 is a connection port that can connect the imaging unit 11, the audio input / output unit 12, the tilt detection unit 13, the operation input unit 14, the fingerprint detection unit 15, the position detection unit 16, and the pressure detection unit 17. The I / O port 35 can be configured using various ports that can connect devices, such as a USB (Universal Serial Bus) port or an IEEE 1394 port.

[0038] The storage device 36 is a device that stores various programs executed by the processor 31 and various data to be used by the various programs. The storage device 36 can be configured using a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0039] The communication device 37 includes a data communication unit that communicates with an external server, cloud, etc. via a communication network (not shown) and transmits and receives various data, and a voice communication unit that transmits and receives wireless signals for telephone communication with a base station (not shown). The data communication unit can be configured using a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The voice communication unit can be configured using a communication device that transmits and receives wireless signals for telephone communication with a base station.

[0040] The processor 31 executes the authentication processing program 180 and the reflected image processing program 181 stored in the terminal storage unit 18 of the terminal device 1 shown in Fig. 4, thereby realizing the information processing block shown in Fig. 6 in the terminal control unit 19. As a result, the terminal device 1 can restore an image reflected in the eyes from an image of the face of the user using the terminal device 1 and authenticate the user U by comparing the restored image with the image normally displayed on the screen, and can also authenticate the user U based on the user U's movement information.

[0041] The information processing block includes an authentication information acquisition unit 191 that acquires images and various data from the communication unit 10, the photographing unit 11, etc.; a reflected image processing unit 192 that restores the reflected image R reflected in the pupil P of the user U and performs various processes related to the reflected image R; a motion information acquisition unit 193 that acquires motion information of the user U; an authentication judgment unit 194 that authenticates whether the user is the real person; a display processing unit 195 that displays various data, images, authentication results, etc. on the display unit 20; and an authentication information update unit 196 that updates the information of various databases and tables stored in the terminal memory unit 18 in response to instructions from the authentication judgment unit 194.

[0042] The authentication information acquisition unit 191 acquires a photograph of the user U from the photographing unit 11, and also acquires motion information of the user U from the communication unit 10, the photographing unit 11, the tilt detection unit 13, etc. The reflected image processing unit 192 acquires an image of the face of the user U from the photograph acquired from the authentication information acquisition unit 191, and restores the reflected image R reflected in the pupil P. The reflected image processing unit 192 also performs various processes related to the reflected image R, such as correction and size change.

[0043] In the first embodiment, the reflected image R is an image of the display content displayed on the screen of the display unit 20 of the terminal device 1. When the user U is looking at the display unit 20 of the terminal device 1, not only the display content displayed on the screen of the display unit 20 of the terminal device 1 but also images of objects placed around the user U, the background of the location, etc. are reflected in the pupil P of the user U. For this reason, the reflected image processing unit 192 extracts the image of the display content displayed on the screen of the display unit 20 of the terminal device 1 from the various images reflected in the pupil P and outputs it as the reflected image R.

[0044] The motion information acquisition unit 193 acquires the detection results of the communication unit 10, the photographing unit 11, the tilt detection unit 13, etc. from the authentication information acquisition unit 191 as motion information of the user U. The authentication determination unit 194 authenticates the user based on the reflected image R restored by the reflected image processing unit 192 and the motion information of the user U acquired by the motion information acquisition unit 193. For example, authentication using the reflected image R is performed by comparing the reflected image R with the image displayed on the screen of the display unit 20 of the terminal device 1 when the face of the user U was photographed by the photographing unit 11, and authenticating the user based on an authentication value obtained from the comparison result. Note that the comparison between the reflected image R and the image displayed on the screen of the display unit 20 of the terminal device 1 may be performed using any method that can compare images, such as pattern matching or difference comparison. Furthermore, for example, authentication based on the user U's motion information authenticates the user U when the detection results of the communication unit 10, the photographing unit 11, the tilt detection unit 13, etc. acquired from the authentication information acquisition unit 191 satisfy predetermined passing conditions.

[0045] The display processing unit 195 displays or reflects various data, images, authentication results of the authentication assessment unit 194, etc. on the display unit 20 as necessary. The display processing unit 195 also transmits the various data, images, etc. displayed on the display unit 20 to the authentication assessment unit 194. The authentication information update unit 196 updates the data stored in the various databases and tables stored in the terminal storage unit 18 based on instructions from the authentication assessment unit 194.

[0046] Next, the configuration of each of the reflected-image authentication value table 182, authentication action information database 183, authentication count table 184, and waiting time setting table 185 stored in the terminal storage unit 18 shown in Fig. 4 will be described below with reference to Figs. 7A to 7D. First, Fig. 7A shows the table configuration of the reflected-image authentication value table 182. The reflected-image authentication value table 182 includes an average value of the authentication value used to authenticate the reflected image R by the authentication processing program 180, an authentication threshold value that is a threshold value for judgment, and an authentication tolerance value that includes an authentication tolerance range value that indicates when the authentication threshold is gray for user U.

[0047] In the first embodiment, authentication determination based on the reflected image R is performed using an authentication value. The authentication value is a value found based on the result of comparing the reflected image R with the display content displayed on the display unit 20 shown in FIG. 4. In the first embodiment, the authentication value approaches 0 when the reflected image R and the display content displayed on the display unit 20 are similar, and approaches 1 when they are not similar. The average value of the authentication values ​​included in the reflected image authentication value table 182 is the average value of the authentication values ​​found based on the result of comparing the reflected image R with the display content displayed on the display unit 20 shown in FIG. 4.

[0048] The authentication threshold is a reference value for determining that user U is the user himself / herself if the authentication value obtained based on the result of comparing the reflected image R with the display content displayed on the display unit 20 shown in FIG. 4 is equal to or less than this value. The authentication threshold is a value that varies according to the authentication status of user U, and has a predetermined upper limit. The upper limit is a value that, if exceeded, determines that user U should not be authenticated as the user himself / herself based on the reflected image R. For example, if the default value of the authentication threshold is 0.4, which is between 0 when the reflected image R and the display content displayed on the display unit 20 are similar and 1 when they are not similar, the upper limit of the authentication threshold is set to 0.45.

[0049] The authentication tolerance value is a standard value for determining that user U is not the user himself / herself if the authentication value obtained based on the result of comparing the reflected image R with the display content displayed on the display unit 20 is equal to or greater than this value. As described above, the authentication tolerance value is a value that includes the authentication threshold value and the authentication tolerance range value that indicates when user U is in a gray area, and is a value that fluctuates depending on fluctuations between the authentication threshold value and the authentication tolerance range value. A predetermined upper limit is set for the authentication tolerance value, which is called the maximum authentication tolerance value. The maximum authentication tolerance value is a value above which user U should be determined to be a different person. For example, the maximum authentication tolerance value is set to 0.5, which is halfway between 0, which approaches when the reflected image R and the display content displayed on the display unit 20 are similar, and 1, which approaches when they are not similar.

[0050] The value between the authentication threshold and the authentication tolerance value is called the authentication tolerance range value. When it is unclear whether user U is the real user, that is, when the authentication value falls within this range, it is generally considered safe to assume that user U is the real user. The authentication tolerance range value is, for example, 0.08, which is less than 10% of the difference between 0, which approaches similarity, and 1, which approaches dissimilarity. Note that when the authentication threshold reaches its upper limit, the authentication tolerance range value becomes the maximum authentication tolerance value minus the authentication threshold upper limit. For example, if the upper limit of the authentication threshold is 0.45 and the maximum authentication value is 0.5, the authentication tolerance range value would be 0.05. Therefore, when the authentication threshold is at its upper limit, the authentication tolerance range value is smaller than when the authentication threshold is not at its upper limit.

[0051] Then, if the authentication value is within the authentication tolerance range, whether the user U is the real user is determined not only based on the reflected image R, but also based on the motion information of the user U. As a result, for example, if the authentication value is slightly greater than the authentication threshold as a result of comparing the reflected image R with the display content displayed on the display unit 20, the user U is not immediately determined to be an unauthorized user, thereby reducing troublesome situations for the user U, such as frequent locking or the display of a message indicating that authentication is not possible. When authenticating based on the motion information of the user U, if the motion information of the user U matches the pass criteria, the user U is authenticated as the authorized user, and if it does not match, the user U is not authenticated as the authorized user.

[0052] In the reflected image authentication value table 182 shown in FIG. 7A, for example, the average authentication value is set to "0.44", the authentication threshold is set to "0.40", and the authentication tolerance is set to "0.45".

[0053] Next, the tables of the authentication motion information database 183 will be described with reference to Fig. 7B. The authentication motion information database 183 is a database for storing motion information of the user U operating the terminal device 1, conditions for passing authentication, etc. The motion information of the user U operating the terminal device 1 is information related to the operation details of the user U on the terminal device 1, motion information of the terminal device 1, etc., obtained from the detection results of the communication unit 10, the imaging unit 11, the tilt detection unit 13, etc., shown in Fig. 6. As shown in Fig. 7B, the table of the authentication motion information database 183 includes items such as type of motion, acquired information, latest status, and passing conditions.

[0054] The type of action is the type of operation performed by the user U on the terminal device 1, and the type of action of the terminal device 1. The acquired information is information acquired according to the type of action from the imaging unit 11, the tilt detection unit 13, etc. shown in Fig. 4. The latest status stores the latest action information when the user U was successfully authenticated. The pass condition includes a pass condition that is a criterion for determining whether or not to authenticate the user U.

[0055] For example, as shown in FIG. 7B , when the type of action is "gripping state," this is the state in which the terminal device 1 is being gripped by the user U. When the terminal device 1 is being gripped by the user U, the pressure can be detected by the pressure detection unit 17 of the terminal device 1 shown in FIG. 3 . Therefore, the acquired information is "pressure," and is the pressure value acquired from the pressure detection unit 17. The latest status is the average value of the pressure value acquired from the pressure detection unit 17 when the user U was successfully authenticated and the pressure value stored in the existing latest status column. The passing condition is that the pressure value acquired from the pressure detection unit 17 is equal to or greater than a threshold value.

[0056] Also, for example, as shown in FIG. 7B , if the type of action is "tilt," this is the current tilt of the terminal device 1. The current tilt of the terminal device 1 can be indicated by an angle detected by the tilt detection unit 13 shown in FIG. 4. Therefore, the acquired information is "angle," and is the tilt angle value acquired from the tilt detection unit 13. The latest status is the average value of the tilt angle value acquired from the tilt detection unit 13 when user U was successfully authenticated and the tilt value stored in the existing latest status field. The pass condition is that the difference between the angle acquired from the tilt detection unit 13 and the angle stored in the latest status field is within plus or minus 30 degrees.

[0057] 7B, for example, when the type of action is "distance between the face and the terminal device," the distance between the face and the terminal device can be obtained from the facial image of user U captured by the photographing unit 11 shown in FIG. 4. Therefore, the acquired information is "distance," and is the distance value obtained from the facial image of user U captured by the photographing unit 11. The latest status is the average value of the distance value obtained from the facial image of user U captured by the photographing unit 11 when user U was successfully authenticated and the distance value stored in the existing latest status field. The passing condition is that the difference between the distance value obtained from the facial image of user U captured by the photographing unit 11 and the distance value stored in the latest status field is within plus or minus 20 mm.

[0058] Also, for example, as shown in FIG. 7B , if the type of operation is “communication connection,” this indicates the current communication destination of the terminal device 1, such as the Internet, LAN, or WAN. The current communication destination of the terminal device 1 is the communication destination communicated with via the communication unit 10 shown in FIG. 4 . Therefore, the acquired information is the communication destination communicated with via the communication unit 10, such as “123WAN” shown in FIG. 7B . The latest status here is the number of times the terminal device 1 has connected to “123WAN.” If the current communication destination of the communication unit 10 is “123WAN,” the number stored in the existing latest status field is incremented by one; if the communication destination is not “123WAN,” nothing is done. The pass condition is that the terminal device 1 has been connected to “123WAN” a set number of times or more. If the terminal device 1 has been connected to “123WAN” a set number of times or more, the user U is authenticated as a trustworthy connection destination. Furthermore, if the terminal device 1 has not been connected to "123WAN" a set number of times or more, the user U is not authenticated as "123WAN" is an unreliable connection destination.

[0059] 7B , if the type of operation is "location information," this indicates the current location of the terminal device 1. For example, this may be the location of an event stored in a schedule book, or the address of the user U's home, office, etc. Therefore, the acquired information may be, for example, latitude and longitude indicating the location of an event stored in a schedule book, or the address of the user U's home, office, etc. The latest status is the average value of the latitude and longitude detected by the position detection unit 16 shown in FIG. 4 when user U is successfully authenticated and the latitude and longitude stored in the existing latest status field. The passing condition is that the difference in distance between the latitude and longitude of the current location of the terminal device 1 detected by the position detection unit 16 and the latitude and longitude stored in the latest status field is within 100 meters.

[0060] Also, for example, as shown in FIG. 7B, when the type of operation is "device connection," this indicates the device currently connected to the terminal device 1. The terminal device 1 can connect to other devices via the communication unit 10 shown in FIG. 4, for example, by infrared rays, Bluetooth (registered trademark), etc. Therefore, the acquired information is the name, ID, etc. of the connected device, for example, the device name "ABC" in FIG. 7B. The latest status is that the connection status with device "ABC" is saved when user U was authenticated. The pass condition is that the terminal device 1 is connected to device "ABC" via the communication unit 10.

[0061] Next, the authentication count table 184 is shown in Fig. 7C. The authentication count table 184 is a table that sets the number of authentication failures based on the reflected image R, the number of authentication failures based on the motion information of the user U, and the number of judgments for determining whether the authentication is unsuccessful based on the number of authentication failures. As shown in Fig. 7C, the authentication count table 184 includes an authentication type, the number of failures, and the number of judgments. The authentication type is the type of authentication to be performed. In the first embodiment, the reflected image R and the motion information are included because authentication based on the reflected image R and authentication based on the motion information are performed.

[0062] The number of failures is the number of times authentication based on the reflected image R and authentication based on the motion information have failed. For example, in FIG. 7C , the number of failures is one for authentication based on the reflected image R and two for authentication based on the motion information. The number of determinations is the number of times used to determine whether authentication has been unsuccessful due to the number of authentication failures. For example, in FIG. 7C , the number of determinations is set to three.

[0063] Next, the waiting time setting table 185 is shown in Fig. 7D. The waiting time setting table 185 is a table for setting the waiting time until the next execution of the authentication process, which is repeatedly executed. For example, in Fig. 7D, the waiting time is set to 0.5 seconds.

[0064] Next, a description will be given of the flow of authentication of user U in terminal device 1 of Embodiment 1. In terminal device 1, an image reflected in the eyes of the user is restored from an image of the face of the user using the terminal device 1, and the image is compared with the image normally displayed on the screen to authenticate user U. If the authentication based on the reflected image R determines that the image is gray, user U is authenticated based on movement information of user U detected by terminal device 1.

[0065] However, the user U does not necessarily remain motionless while facing the screen when using the terminal device 1. For example, the user U may perform various actions, such as changing the state of holding the terminal device 1 in the hand to a state of holding it in the palm of the hand, or changing the tilt by 90 degrees or more. For this reason, there may be cases where the user U cannot be authenticated based on the user U's action information.

[0066] If the terminal device 1 is disabled immediately after one authentication failure when user U cannot be authenticated based on the user U's motion information, there is a risk that the terminal device 1 will become unusable frequently, resulting in a loss of convenience for the terminal device 1. Therefore, in the first embodiment, the number of authentication failures based on the reflected image R or the number of authentication failures based on the motion information of user U, which are stored in the authentication count table 184 of Fig. 7C, is counted, and the terminal device 1 is disabled when the cumulative number of failures exceeds the determination number.

[0067] For example, the number of times of judgment is set to three as set in the authentication count table 184 of FIG. １ Assume that authentication based on the reflected image R is successful at time t. In this case, the determination result is OK, and the user U can use the terminal device 1. ２ Assume that authentication based on the reflected image R fails in the above example. In this case, the determination result is NG, but since this is the first authentication failure, the user U can use the terminal device 1.

[0068] Time t ３ Assume that authentication based on the reflected image R is successful at time t. In this case, the judgment result is OK, and the number of failures of authentication based on the reflected image R is set to 0. ４In this example, authentication based on the reflected image R is set to gray. In this case, the terminal device 1 performs authentication based on the motion information of the user U. Here, it is assumed that the authentication is successful. In this case, the determination result is OK, and the user U can use the terminal device 1.

[0069] Time t ５ At time t, authentication based on the reflected image R is set to gray. In this case, the terminal device 1 executes authentication based on the motion information of the user U. Here, it is assumed that the authentication has failed. In this case, the determination result is NG, but since this is the first authentication failure, the user U can use the terminal device 1. ６ Assume that authentication based on the reflected image R is successful in the above step 10. In this case, the determination result is OK, and the number of authentication failures based on the user U's motion information is set to zero.

[0070] Time t ７ In this example, authentication based on the reflected image R is set to gray. In this case, the terminal device 1 executes authentication based on the operation information of the user U. Here, it is assumed that the authentication has failed. In this case, the determination result is NG, but since this is the first authentication failure, the user U can use the terminal device 1.

[0071] Time t ８ In this example, authentication based on the reflected image R is set to gray. In this case, the terminal device 1 executes authentication based on the operation information of the user U. Here, it is assumed that the authentication has failed. In this case, the determination result is NG, but since this is the second authentication failure, the user U can use the terminal device 1.

[0072] Time t ９ In this case, authentication based on the reflected image R is set to gray. In this case, the terminal device 1 executes authentication based on the operation information of the user U. Here, it is assumed that the authentication has failed. In this case, the determination result is NG, and since this is the third authentication failure, the terminal device 1 is locked. As a result, the user U is unable to use the terminal device 1.

[0073] When the terminal device 1 according to the first embodiment completes the initialization process after power-on or returns from sleep mode, it enters a locked state in which operation of each function is not permitted until authentication is successful. When this locked state is entered, the terminal control unit 19 shown in FIG. 4 executes the authentication processing program 180 stored in the terminal storage unit 18 to determine whether the user U is the real user. Thereafter, authentication is performed in the background when each function is operated. The processing of the authentication processing program 180 executed by the terminal control unit 19 will be described below with reference to the flowcharts of each process shown in FIGS. 9A to 11.

[0074] First, refer to the flowchart of the authentication process shown in Figure 9A. The authentication assessment unit 194 of the terminal control unit 19 shown in Figure 6 sets the number of failures to 0 (step S101). Specifically, the number of failures of authentication using reflected images and the number of failures of authentication using motion information shown in Figure 7C are set to 0. The authentication assessment unit 194 of the terminal control unit 19 executes reflected image authentication value acquisition processing (step S102). The reflected image authentication value acquisition processing will be described below with reference to the flowchart in Figure 10.

[0075] The authentication determination unit 194 of the terminal control unit 19 sets the number of retries to 0 (step S201). The authentication information acquisition unit 191 shown in Fig. 6 causes the photographing unit 11 to take a facial photograph of the user U operating the terminal device 1. Specifically, the authentication information acquisition unit 191 causes the in-camera 11A to take a facial photograph of the user U facing the front of the terminal device 1. The authentication information acquisition unit 191 acquires the facial photograph of the user taken by the photographing unit 11 (step S202).

[0076] 6, the authentication determination unit 194 acquires data of the display image that was displayed on the display unit 20 when the facial photograph of the user U was taken by the photographing unit 11 (step S203). The authentication information acquisition unit 191 determines whether the acquired facial photograph of the user U is blurred (step S204).

[0077] If the facial photograph of the user U is blurred (step S204; YES), the authentication information acquisition unit 191 determines whether the number of retries is less than a predetermined number (step S205). This predetermined number is a predetermined number of retries that can be made, and is assumed to be stored in the terminal storage unit 18 shown in FIG.

[0078] If the number of retries is less than the specified number (step S205; YES), the authentication information acquisition unit 191 adds 1 to the number of retries (step S206). The authentication information acquisition unit 191 causes the photographing unit 11 to retry taking a photograph of the user's face (step S207). If the number of retries is equal to or greater than the specified number (step S205; NO) in step S205, the authentication information acquisition unit 191 sets the authentication value to a value greater than the authentication tolerance value (step S208). For example, if the authentication tolerance value is 0.45 as shown in FIG. 7A, the authentication information acquisition unit 191 sets the authentication value to a value such as 0.50.

[0079] Furthermore, in step S204, if the facial photograph of user U is not blurred (step S204; NO), the authentication information acquisition unit 191 transmits the facial photograph of user U taken by the photographing unit 11 to the reflected image processing unit 192 of the terminal control unit 19 shown in Fig. 6. The reflected image processing unit 192 determines whether or not the face of user U can be detected from the received facial photograph of user U (step S209). If the face of user U cannot be detected from the facial photograph of user U (step S209; NO), the reflected image processing unit 192 causes the authentication information acquisition unit 191 to execute steps S205 to S208.

[0080] If the face of user U can be detected from the facial photograph of user U (step S209; YES), the reflected-image processing unit 192 determines whether or not the pupil P of the eye E can be detected from the detected image of the face of user U (step S210). If the pupil P of the eye E cannot be detected from the detected image of the face of user U (step S210; NO), the reflected-image processing unit 192 causes the authentication information acquisition unit 191 to execute step S208.

[0081] If the pupil P of the eye E can be detected from the detected image of the face of the user U (step S210; YES), the reflected image processing unit 192 restores the reflected image R reflected in the pupil P (step S211). The reflected image processing unit 192 transmits the restored reflected image R to the authentication assessment unit 194.

[0082] The authentication assessment unit 194 compares the received reflected image R with the display image data acquired from the display processing unit 195 in step S203 (step S212). The authentication assessment unit 194 obtains an authentication value from the comparison result (step S213). The authentication assessment unit 194 then terminates the reflected image authentication value acquisition process.

[0083] Returning to Fig. 9A, the authentication assessment unit 194 determines whether the authentication value acquired in step S102 is greater than the authentication threshold value (step S103). The authentication threshold value is the value stored in the reflected-image authentication value table 182 of Fig. 7A. If the authentication value is greater than the authentication threshold value (step S103; YES), the authentication assessment unit 194 determines whether the authentication value is smaller than the authentication tolerance value (step S104). If the authentication value is greater than the authentication threshold value (step S104; NO), the authentication assessment unit 194 adds 1 to the number of authentication failures using the reflected image R (step S105).

[0084] Now, proceed to Fig. 9B. The authentication assessment unit 194 determines whether the number of authentication failures based on the reflected image R or the motion information is equal to or greater than the determination count (step S106). If the number of authentication failures based on the reflected image R or the motion information is less than the determination count (step S106; NO), the authentication assessment unit 194 determines whether the waiting time has elapsed (step S107). The waiting time is the waiting time set in the waiting time setting table 185 shown in Fig. 7D.

[0085] If the waiting time has not elapsed (step S107; NO), the authentication assessment unit 194 repeats step S107. If the waiting time has elapsed (step S107; YES), the authentication assessment unit 194 returns to step S102 and executes step S102 and subsequent steps.

[0086] Furthermore, in step S106, if the number of authentication failures based on the reflected image R or the motion information is equal to or greater than the determination number (step S106; YES), the authentication determination unit 194 clears the number of authentication failures based on the reflected image R and the motion information and resets it to 0 (step S108). The authentication determination unit 194 causes the display processing unit 195 shown in Fig. 6 to display on the display unit 20 that authentication has failed (step S109). The authentication determination unit 194 causes the display processing unit 195 to display on the display unit 20 a message requesting entry of a password (step S110).

[0087] The authentication determination unit 194 determines whether the password entered by the user U via the operation input unit 14 shown in Fig. 6 is correct (step S111). If the password is correct (step S111; YES), the authentication determination unit 194 executes step S107. If the password is incorrect (step S111; NO), the authentication determination unit 194 causes the display processing unit 195 to display a login screen on the display unit 20 (step S112). This login screen is a personal authentication means by the OS included in the terminal device 1, such as password entry or fingerprint authentication.

[0088] If the authentication value is smaller than the authentication threshold value in step S104 (step S104; YES), the authentication assessment unit 194 executes a motion information authentication process (step S113). The motion information authentication process will be described below with reference to the flowchart shown in FIG.

[0089] The motion information acquisition unit 193 shown in Fig. 6 acquires a pressure value from the pressure detection unit 17 via the authentication information acquisition unit 191 (step S301). This pressure value is the pressure value when the terminal device 1 is being held by the user U. The authentication determination unit 194 receives the pressure value from the motion information acquisition unit 193 and determines whether it is equal to or greater than a threshold value (step S302). The threshold value is, for example, a threshold value set as a pass condition when the motion type is "gripping state" in the authentication motion information database 183 shown in Fig. 7B.

[0090] If the pressure value is equal to or greater than the threshold (step S302; YES), the authentication assessment unit 194 sets the assessment result to indicate that the user U has been authenticated as the user (step S303). If the pressure value is less than the threshold (step S302; NO), the motion information acquisition unit 193 acquires the tilt angle of the terminal device 1 from the tilt detection unit 13 via the authentication information acquisition unit 191 (step S304). The authentication assessment unit 194 receives the tilt angle of the terminal device 1 from the motion information acquisition unit 193 and determines whether it meets the pass criteria for the "tilt" motion type in the authentication motion information database 183 shown in FIG. 7B (step S305). The pass criteria in the authentication motion information database 183 is that the difference between the angle acquired from the tilt detection unit 13 and the angle stored in the "latest status" column is within ±30 degrees.

[0091] If the difference between the angle acquired from the tilt detection unit 13 and the angle stored in the latest status field is within ±30 degrees (step S305; YES), the authentication determination unit 194 executes step S303. If the difference between the angle acquired from the tilt detection unit 13 and the angle stored in the latest status field is greater than ±30 degrees (step S305; NO), the motion information acquisition unit 193 acquires a facial image of the user U from the imaging unit 11 via the authentication information acquisition unit 191. The motion information acquisition unit 193 calculates the distance between the face of the user U and the terminal device 1 based on the facial image of the user U (step S306).

[0092] The authentication determination unit 194 receives the distance between the face of the user U and the terminal device 1 from the motion information acquisition unit 193, and determines whether the distance value is within the pass condition for the motion type "distance between face and terminal device" in the authentication motion information database 183 shown in Fig. 7B (step S307). The pass condition in the authentication motion information database 183 is that the difference between the distance value calculated from the face image of the user U captured by the imaging unit 11 and the distance value saved in the latest status column is within plus or minus 20 mm.

[0093] If the difference between the distance value calculated from the facial image and the distance value stored in the latest status field is within ±20 mm (step S307; YES), the authentication assessment unit 194 executes step S303. If the difference between the distance value calculated from the facial image and the distance value stored in the latest status field is greater than ±20 mm (step S307; NO), the operation information acquisition unit 193 acquires the current connection destination of the terminal device 1 from the communication unit 10 via the authentication information acquisition unit 191. For example, if the current communication connection destination of the terminal device 1 is "123WAN" shown in the acquired information in FIG. 7B and has been connected more than the number of times set as the pass condition, the authentication assessment unit 194 determines that "123WAN" is a trustworthy connection destination (step S308; YES). The authentication assessment unit 194 executes step S303.

[0094] Furthermore, for example, if the current communication connection destination of terminal device 1 is "123WAN" shown in the acquired information of Fig. 7B and has been connected fewer times than the number of times set as the pass condition, or if the current communication connection destination of terminal device 1 is not "123WAN" shown in the acquired information of Fig. 7B, the authentication assessment unit 194 determines that terminal device 1 is not connected to a trustworthy connection destination (step S308; NO). The operation information acquisition unit 193 acquires the current location of terminal device 1 from the position detection unit 16 via the authentication information acquisition unit 191 (step S309).

[0095] The authentication assessment unit 194 receives the current location of the terminal device 1 from the motion information acquisition unit 193, and determines whether the current value is within the pass condition for the motion type "location information" in the authentication motion information database 183 shown in Fig. 7B (step S310). The pass condition in the authentication motion information database 183 is that the difference in distance between the latitude and longitude of the current location of the terminal device 1 detected by the position detection unit 16 and the latitude and longitude stored in the latest status column is within 100 meters.

[0096] If the difference between the latitude and longitude of the current location of the terminal device 1 and the latitude and longitude stored in the latest status field is within 100 meters (step S310; YES), the authentication determination unit 194 executes step S303. If the difference between the latitude and longitude of the current location of the terminal device 1 and the latitude and longitude stored in the latest status field is greater than 100 meters (step S310; NO), the operation information acquisition unit 193 acquires the name, ID, etc. of the currently connected device from the communication unit 10 via the authentication information acquisition unit 191. The authentication determination unit 194 determines whether the terminal device 1 is currently connected to another device (step S311).

[0097] If the name and ID of the currently connected device can be acquired from the operation information acquisition unit 193, the authentication assessment unit 194 determines that the terminal device 1 is currently connected to another device (step S311; YES). The authentication assessment unit 194 executes step S303. If the name and ID of the currently connected device cannot be acquired from the operation information acquisition unit 193, the authentication assessment unit 194 determines that the terminal device 1 is not currently connected to another device (step S311; NO). The authentication assessment unit 194 sets the determination result to the effect that user U could not be authenticated as the user himself / herself (step S312). The authentication assessment unit 194 ends the operation information authentication process.

[0098] Returning to Fig. 9A, the authentication assessment unit 194 determines whether or not the user U has been authenticated (step S114). If the user U has not been authenticated (step S114; NO), the authentication assessment unit 194 executes steps S105 to S112. If the user U has been authenticated (step S114; YES), or if the authentication value in step S103 is smaller than the authentication threshold value (step S103; NO), the process proceeds to Fig. 9B, where the authentication assessment unit 194 clears the number of authentication failures based on the reflected image R and the motion information and resets it to 0 (step S115).

[0099] The authentication assessment unit 194 calculates the average value of the authentication values ​​(step S116). Specifically, the average value is calculated by adding the value stored in the average authentication value column of the reflected-image authentication value table 182 shown in Fig. 7A to the authentication value acquired in step S102 and dividing the result by two.

[0100] The authentication assessment unit 194 updates the authentication threshold (step S117). Specifically, if the authentication value acquired in step S102 is greater than the value stored in the average authentication value column of the reflected-image authentication value table 182 shown in FIG. 7A by a predetermined value or more, the authentication assessment unit 194 increases the authentication threshold. For example, from 0.40 to 0.42. Conversely, if the authentication value acquired in step S102 is smaller than the value stored in the average authentication value column of the reflected-image authentication value table 182 shown in FIG. 7A by a predetermined value or more, the authentication assessment unit 194 decreases the authentication threshold. For example, from 0.40 to 0.38.

[0101] The authentication assessment unit 194 updates the authentication tolerance value (step S118). Specifically, if the authentication threshold value was increased in step S117, the authentication assessment unit 194 increases the authentication tolerance value. For example, if the authentication threshold value changes from 0.40 to 0.42, the authentication assessment unit 194 increases the authentication tolerance value from 0.45 to 0.47. Furthermore, if the authentication threshold value was decreased in step S117, the authentication assessment unit 194 decreases the authentication tolerance value. For example, if the authentication threshold value changes from 0.40 to 0.38, the authentication assessment unit 194 decreases the authentication tolerance value from 0.45 to 0.43.

[0102] The authentication assessment unit 194 causes the authentication information update unit 196 shown in Fig. 6 to update the reflected-image authentication value table 182 and the authentication action information database 183 shown in Fig. 7B (step S119). Specifically, the authentication assessment unit 194 causes the authentication information update unit 196 to store the average value of the authentication values ​​calculated in step S116 in the average value of authentication values ​​column of the reflected-image authentication value table 182. The authentication assessment unit 194 causes the authentication information update unit 196 to store the authentication threshold value calculated in step S117 in the authentication threshold column of the reflected-image authentication value table 182. The authentication assessment unit 194 causes the authentication information update unit 196 to store the authentication tolerance value calculated in step S118 in the authentication tolerance value column of the reflected-image authentication value table 182.

[0103] Furthermore, the authentication assessment unit 194 causes the authentication information update unit 196 to update each latest status column in the authentication action information database 183. In this way, by updating each value stored in the reflected image authentication value table 182 and each latest status column in the authentication action information database 183, the authentication system for user U is improved. The authentication assessment unit 194 executes step S107 and repeats step S102 and subsequent steps.

[0104] As described above, the terminal device 1 according to the first embodiment authenticates the user U by restoring the image reflected in the eyes from the image of the face of the user using the terminal device 1 and comparing it with the image normally displayed on the display screen, and if the reflected image R is determined to be gray in authentication based on the reflected image R, the terminal device 1 can authenticate the user U based on the motion information of the user U detected by the terminal device 1. This makes it possible to prevent impersonation of the user U who is looking at the display screen of the terminal device 1.

[0105] Furthermore, in the first embodiment, when the number of authentication failures based on the reflected image R or the number of authentication failures based on the motion information of the user U, stored in the authentication count table 184 of Fig. 7C, exceeds the determination count, the terminal device 1 is made unusable. As a result, when the user U cannot be authenticated based on the motion information of the user U, the terminal device 1 is not immediately locked out of use after one authentication failure, so the terminal device 1 does not become unusable frequently, and the convenience of the terminal device 1 can be ensured.

[0106] Furthermore, the authentication process for user U performed by terminal device 1 is executed in the background while terminal device 1 is running, and the authentication value, authentication threshold, and authentication tolerance used for authentication using a reflected image, as well as the latest status used for authentication using motion information, are updated, thereby improving the accuracy of authentication. This ensures security without imposing a burden on user U.

[0107] (Embodiment 2) In embodiment 1, if the authentication based on the reflected image R determines that the image is gray, the user U is authenticated based on the motion information of the user U detected by the terminal device 1. However, this is not limiting, and if authentication based on both the reflected image R and the motion information of the user U is successful, the user U may be authenticated as the person in question. This authentication process is described below as the authentication process of embodiment 2.

[0108] 12 is a block diagram showing the configuration of a terminal device 1A according to Embodiment 2. The terminal device 1A includes a communication unit 10, an imaging unit 11, an audio input / output unit 12, a tilt detection unit 13, an operation input unit 14, a fingerprint detection unit 15, a position detection unit 16, a pressure detection unit 17, a terminal storage unit 18A, a terminal control unit 19, and a display unit 20.

[0109] The terminal storage unit 18A includes a combined authentication processing program 186 for performing authentication processing for user U, a reflected image processing program 181 for restoring a reflected image R reflected in the pupil P of user U, a reflected image authentication value table 182 in which an authentication value for authenticating the reflected image R is set, an authentication operation information database 183 that compiles operation information of user U acquired by the terminal device 1A, a combined authentication count table 187 in which the number of authentication failures and the number of judgments are set, and a waiting time setting table 185 in which the waiting time until the authentication processing is executed is set.

[0110] The combined authentication processing program 186 is a program that authenticates user U as the identity of the user when both authentication based on the reflected image R and authentication based on the user U's motion information are successful. The combined authentication count table 187 is a table that sets the number of failures in authentication based on the reflected image R and authentication based on the user U's motion information, and the number of judgments used to determine whether authentication is unsuccessful due to the number of failures. The combined authentication count table 187 includes items for the number of failures and the number of judgments, as shown in the table configuration in FIG. 13. For example, in FIG. 13, 1 is stored in the number of failures and 3 is stored in the number of judgments.

[0111] Next, the flow of authentication of user U in the second embodiment will be described. When authentication based on both the reflected image R and the user U's motion information is successful, the terminal device 1A authenticates the user U as the person in question. When using the terminal device 1A, the user U does not necessarily remain motionless while facing the display screen. For example, the user U may turn his / her face away from the display screen of the terminal device 1A, or may change from holding the terminal device 1A in his / her hand to placing it in his / her palm. For this reason, there may be cases where the user U cannot be authenticated based on the reflected image R and the user U's motion information.

[0112] If user U cannot be authenticated based on the reflected image R and the motion information of user U, and terminal device 1A is made unusable immediately after one authentication failure, there is a risk that terminal device 1A will become unusable frequently, and the convenience of terminal device 1A will be lost. Therefore, in the second embodiment, the number of failures in authenticating user U based on the reflected image R and the motion information of user U is counted, and when the cumulative number of failures exceeds the number of determinations stored in combined authentication count table 187 of Fig. 13 , terminal device 1A is made unusable.

[0113] For example, the number of times of judgment is set to three as set in the combined authentication count table 187 of FIG. 13. As shown in FIG. １ Assume that authentication based on the reflected image R and the motion information of the user U is successful at time t ２ Assume that authentication based on the reflected image R is successful, but authentication based on the motion information of the user U is unsuccessful. In this case, the determination result is NG, but since this is the first authentication failure, the user U can use the terminal device 1A.

[0114] Time t ３ Assume that authentication based on the reflected image R and the motion information of the user U is successful at time t. In this case, the determination result is OK, and the number of authentication failures is set to 0. ４Assume that authentication based on the reflected image R fails, but authentication based on the motion information of the user U succeeds. In this case, the determination result is NG, but since this is the first authentication failure, the user U can use the terminal device 1A.

[0115] Time t ５ Assume that authentication based on the reflected image R is successful, but authentication based on the motion information of the user U is unsuccessful at time t. In this case, the determination result is NG, but since this is the second authentication failure, the user U can use the terminal device 1A. ６ Assume that authentication based on the reflected image R and the motion information of the user U is successful. In this case, the determination result is OK, and the number of authentication failures is set to zero.

[0116] Time t ７ Assume that authentication based on the reflected image R and the motion information of the user U fails at time t. In this case, the determination result is NG, but since this is the first authentication failure, the user U can use the terminal device 1A. ８ Assume that authentication based on the reflected image R is successful, but authentication based on the motion information of the user U is unsuccessful. In this case, the determination result is NG, but since this is the second authentication failure, the user U can use the terminal device 1A.

[0117] Time t ９ Assume that authentication based on the reflected image R fails, but authentication based on the user U's motion information succeeds. In this case, the determination result is NG, and since this is the third authentication failure, the terminal device 1A is locked. This prevents the user U from using the terminal device 1A.

[0118] When the terminal device 1A according to the second embodiment completes the initialization process after power-on or returns from sleep mode, it enters a locked state in which operation of each function is not permitted until authentication is successful. When this locked state is entered, the terminal control unit 19 shown in FIG. 12 executes the combined authentication processing program 186 stored in the terminal storage unit 18A to determine whether the user U is the real user. Thereafter, authentication is performed in the background when each function is operated. The processing of the combined authentication processing program 186 executed by the terminal control unit 19 will be described below with reference to the flowcharts of each process shown in FIGS. 15A and 15B.

[0119] First, refer to the flowchart of the combined authentication process shown in Fig. 15A. The authentication assessment unit 194 of the terminal control unit 19 shown in Fig. 6 sets the number of failures to 0 (step S401). Specifically, the number of failures in the combined authentication count table 187 shown in Fig. 13 is set to 0. The authentication assessment unit 194 of the terminal control unit 19 executes a process to obtain an reflected-image authentication value (step S402). The reflected-image authentication value acquisition process executes the process shown in the flowchart of Fig. 10.

[0120] Next, the authentication determination unit 194 of the terminal control unit 19 executes a motion information authentication process (step S403). The motion information authentication process executes the process of the flowchart in Fig. 11. The authentication determination unit 194 of the terminal control unit 19 determines whether the authentication value acquired in the reflected-image authentication process in step S402 is smaller than the authentication threshold value and whether user U was authenticated based on the motion information in the motion information authentication process in step S403 (step S404). The authentication threshold value is the value stored in the reflected-image authentication value table 182 in Fig. 7A.

[0121] If the authentication value is greater than the authentication threshold, or if user U could not be authenticated, or if both of these conditions are met (step S404; NO), the authentication assessment unit 194 adds 1 to the number of failures due to the reflected image R (step S405). The authentication assessment unit 194 then determines whether the number of authentication failures is equal to or greater than the determination count (step S406). If the number of authentication failures is less than the determination count (step S406; NO), the authentication assessment unit 194 proceeds to FIG. 15B and determines whether a waiting time has elapsed (step S407). The waiting time is the waiting time set in the waiting time setting table 185 shown in FIG. 7D.

[0122] If the waiting time has not elapsed (step S407; NO), the authentication assessment unit 194 repeats step S407. If the waiting time has elapsed (step S407; YES), the authentication assessment unit 194 returns to step S402 and executes step S402 and subsequent steps.

[0123] 15A, if the number of authentication failures is equal to or greater than the determination number (step S406; YES), the authentication determination unit 194 clears the number of authentication failures to 0 (step S408). The authentication determination unit 194 then causes the display processing unit 195 shown in FIG. 6 to display on the display unit 20 that authentication was not successful (step S409).

[0124] Now, the process proceeds to FIG. 15B. The authentication assessment unit 194 causes the display processing unit 195 to display on the display unit 20 a message requesting entry of a password (step S410). The authentication assessment unit 194 determines whether the password entered by the user U via the operation input unit 14 shown in FIG. 6 is correct (step S411). If the password is correct (step S411; YES), the authentication assessment unit 194 executes step S407. If the password is incorrect (step S411; NO), the authentication assessment unit 194 causes the display processing unit 195 to display a login screen on the display unit 20 (step S412). This login screen is a personal authentication means by the OS included in the terminal device 1, such as password entry or fingerprint authentication.

[0125] Also, in step S404, if the authentication value obtained in the reflected image authentication process in step S402 is smaller than the authentication threshold value and user U can be authenticated using the action information in the action information authentication process in step S403 (step S404; YES), the authentication judgment unit 194 clears the number of authentication failures using the reflected image R and action information and sets it back to 0 (step S413).

[0126] The authentication assessment unit 194 calculates the average value of the authentication values ​​(step S414). Specifically, the average value is calculated by adding the value stored in the average authentication value column of the reflected-image authentication value table 182 shown in Fig. 7A to the authentication value acquired in step S402 and dividing the result by two.

[0127] The authentication assessment unit 194 updates the authentication threshold (step S415). Specifically, if the authentication value acquired in step S402 is greater than the value stored in the average authentication value column of the reflected-image authentication value table 182 shown in FIG. 7A by a predetermined value or more, the authentication assessment unit 194 increases the authentication threshold. For example, from 0.40 to 0.42. Conversely, if the authentication value acquired in step S402 is smaller than the value stored in the average authentication value column of the reflected-image authentication value table 182 shown in FIG. 7A by a predetermined value or more, the authentication assessment unit 194 decreases the authentication threshold. For example, from 0.40 to 0.38.

[0128] The authentication assessment unit 194 updates the authentication tolerance value (step S416). Specifically, if the authentication threshold value was increased in step S415, the authentication assessment unit 194 increases the authentication tolerance value. For example, if the authentication threshold value changes from 0.40 to 0.42, the authentication assessment unit 194 increases the authentication tolerance value from 0.45 to 0.47. Furthermore, if the authentication threshold value was decreased in step S415, the authentication assessment unit 194 decreases the authentication tolerance value. For example, if the authentication threshold value changes from 0.40 to 0.38, the authentication assessment unit 194 decreases the authentication tolerance value from 0.45 to 0.43.

[0129] The authentication assessment unit 194 causes the authentication information update unit 196 shown in Fig. 6 to update the reflected-image authentication value table 182 shown in Fig. 7A and the authentication action information database 183 shown in Fig. 7B (step S417). Specifically, the authentication assessment unit 194 causes the authentication information update unit 196 to store the average value of the authentication values ​​calculated in step S414 in the average value of authentication values ​​column of the reflected-image authentication value table 182. The authentication assessment unit 194 causes the authentication information update unit 196 to store the authentication threshold value calculated in step S415 in the authentication threshold column of the reflected-image authentication value table 182. The authentication assessment unit 194 causes the authentication information update unit 196 to store the authentication tolerance value calculated in step S416 in the authentication tolerance value column of the reflected-image authentication value table 182.

[0130] Furthermore, the authentication assessment unit 194 causes the authentication information update unit 196 to update each latest status column in the authentication action information database 183. In this way, by updating each value stored in the reflected image authentication value table 182 and each latest status column in the authentication action information database 183, the authentication system for user U is improved. The authentication assessment unit 194 executes step S407 and repeats step S402 and subsequent steps.

[0131] As described above, in addition to the effects of embodiment 1, the terminal device 1A according to embodiment 2 can authenticate user U as the user when both authentication based on the reflected image R and authentication based on the user U's behavior information are possible.

[0132] (Embodiment 3) While the user U is holding the terminal device 1, 1A, it is rare that the user U holds the terminal device 1, 1A in the same position and at the same angle. For example, if the user U moves the hand holding the terminal device 1, 1A up, down, left, or right, the position of the terminal device 1, 1A changes in accordance with the movement of the hand. This changes the relative position between the terminal device 1, 1A and the user U's face, and therefore the distance between the terminal device 1, 1A and the user U's face. When the distance between the terminal device 1, 1A and the user U's face changes, the area occupied by the reflected image R on the user U's pupil P changes. For example, if the distance between the terminal device 1, 1A and the user U's face becomes shorter, the area occupied by the reflected image R on the user U's pupil P becomes larger. Furthermore, if the distance between the terminal device 1, 1A and the user U's face becomes longer, the area occupied by the reflected image R on the user U's pupil P becomes smaller.

[0133] Furthermore, when the tilt of the terminal device 1, 1A held by the user U is changed, the relative angle between the display screen of the display unit 20 of the terminal device 1, 1A and the pupil P of the user U changes. This changes the tilt of the reflected image R reflected on the pupil P of the user U, resulting in, for example, a change in the inclination of the four corners of the reflected image R or a deformation such as the shape of the reflected image R changing from a square to a trapezoid. Therefore, in the third embodiment, the user U is authenticated taking into consideration changes in the size and tilt of the reflected image R linked to changes in the tilt of the terminal device 1, 1A or the distance between the terminal device 1, 1A and the face of the user U.

[0134] 16 is a block diagram showing the configuration of a terminal device 1B according to Embodiment 3. The terminal device 1B includes a communication unit 10, an imaging unit 11, an audio input / output unit 12, a tilt detection unit 13, an operation input unit 14, a fingerprint detection unit 15, a position detection unit 16, a pressure detection unit 17, a terminal storage unit 18B, a terminal control unit 19, and a display unit 20.

[0135] The terminal storage unit 18B includes an interlocking authentication processing program 188 for performing authentication processing for the user U, a reflected image processing program 181 for restoring the reflected image R reflected in the pupil P of the user U, a reflected image authentication value table 182A that stores an authentication value for authenticating the reflected image R and the reflected image R at the time of the previous authentication, an authentication operation information database 183 that compiles the user's operation information acquired by the terminal device 1B, an authentication count table 184 that sets the number of authentication failures and the number of judgments, and a waiting time setting table 185 that sets the waiting time until the authentication processing is executed.

[0136] The linked authentication processing program 188 is a program that authenticates the user U by taking into consideration changes in the reflected image R that are linked to changes in the tilt of the terminal device 1B or the distance between the terminal device 1B and the face of the user U. For example, when the tilt of the terminal device 1B changes, it is determined whether the tilt of the reflected image R has changed from the tilt of the reflected image R acquired during the previous authentication.

[0137] If the tilt of the reflected image R has changed, the tilt of the reflected image R is corrected. The tilt of the reflected image R is corrected, for example, by a method such as tilt correction using affine transformation or by recognizing the four corners of the reflected image R and correcting them to be straight. Then, the tilt-corrected reflected image R is compared with the display image that was displayed on the screen of the display unit 20 when the face of the user U was photographed by the photographing unit 11 to obtain an authentication value, and the user U is authenticated using this authentication value. Furthermore, if the tilt of the reflected image R has not changed, it is determined that there is a possibility of spoofing, and the authentication value is set to a value greater than the authentication tolerance value.

[0138] Furthermore, when the distance between terminal device 1B and the face of user U changes, it is determined whether the size of reflected image R acquired during the previous authentication has changed. This is because, when the distance between terminal device 1B and the face of user U becomes shorter, the area occupied by reflected image R on pupil P becomes larger, and when the distance becomes longer, the area occupied by reflected image R on pupil P becomes smaller, so the area occupied by reflected image R on pupil P of user U changes in inverse proportion to the distance, i.e., the size of reflected image R changes.

[0139] If the size of the reflected image R has changed, the rate of change is calculated from the difference from the size of the reflected image R acquired during the previous authentication, and the size of the display image that was displayed on the screen of the display unit 20 when the face of the user U was photographed by the photographing unit 11 is corrected. Then, the corrected display image is compared with the reflected image R to obtain an authentication value, and the user U is authenticated using this authentication value. If the size of the reflected image R has not changed, there is a possibility of impersonation, and the authentication value is set to a value greater than the authentication tolerance value.

[0140] The reflected-image authentication value table 182A is a table that stores the authentication value for authenticating the reflected image R and the reflected image R at the time of the previous authentication. Specifically, as shown in Fig. 17 , the reflected-image authentication value table 182A includes an average value of the authentication values ​​for authenticating the reflected image R in the linked authentication processing program 188, an authentication threshold value that is a threshold value for judgment, an authentication tolerance value that includes an authentication tolerance range value that indicates when the user U is in a gray area in addition to the authentication threshold value, and a reflected image that stores the image data of the reflected image R.

[0141] The image data stored in the "Reflected Image" field of the reflected image authentication value table 182A is the image data of the reflected image R acquired during the previous authentication. In the third embodiment, the linked authentication processing program 188 compares the image data of the reflected image R stored in the reflected image authentication value table 182A with the image data of the reflected image R acquired during authentication to determine changes in tilt and size. Then, the user U is authenticated depending on whether the tilt and size of the reflected image R have changed.

[0142] Next, the processing of the interlocking authentication processing program 188 executed by the terminal control unit 19 of the terminal device 1B shown in FIG. 16 will be described. When the terminal device 1B according to the third embodiment completes the initialization processing of the post-power-on processing or returns from the sleep state, it enters a locked state in which operation of each function is not permitted until authentication is successful. When it enters this locked state, the terminal control unit 19 shown in FIG. 16 executes the interlocking authentication processing program 188 stored in the terminal storage unit 18B and determines whether the user U is the real user. Thereafter, authentication is performed in the background when operating each function. The processing of the interlocking authentication processing program 188 executed by the terminal control unit 19 will be described below with reference to the flowcharts of FIGS. 18A to 20.

[0143] First, refer to the flowchart of the linked authentication process shown in Figure 18A. The authentication assessment unit 194 of the terminal control unit 19 shown in Figure 6 sets the number of failures to 0 (step S501). Specifically, the number of failures of authentication using reflected images and the number of failures of authentication using motion information shown in Figure 7C are set to 0. The authentication assessment unit 194 of the terminal control unit 19 executes linked reflected-image authentication process (step S502). The linked reflected-image authentication value acquisition process will be described below with reference to the flowchart in Figure 19.

[0144] Steps S601 to S611 of the linked reflected-image authentication value acquisition process are the same as steps S201 to S211 of the reflected-image authentication process shown in Fig. 10. The authentication assessment unit 194 executes linked authentication value setting process (step S612). The linked authentication value setting process will be described below with reference to the flowchart in Fig. 20.

[0145] The motion information acquisition unit 193 shown in FIG. 6 acquires the tilt angle of the terminal device 1B from the tilt detection unit 13 via the authentication information acquisition unit 191 (step S701). The authentication assessment unit 194 determines whether the tilt angle of the terminal device 1B has changed (step S702). Specifically, the authentication assessment unit 194 receives the tilt angle of the terminal device 1B from the motion information acquisition unit 193. The authentication assessment unit 194 acquires angle data stored in the "latest information" for the motion type "tilt" in the authentication motion information database 183. The authentication assessment unit 194 compares the angle value received from the motion information acquisition unit 193 with the angle data value acquired from the authentication motion information database 183, and if there is a difference, it determines that the tilt angle of the terminal device 1B has changed. If there is no difference, the authentication assessment unit 194 determines that the tilt angle of the terminal device 1B has not changed.

[0146] If the tilt angle of terminal device 1B has changed (step S702; YES), the authentication assessment unit 194 determines whether the tilt of reflected image R has changed (step S703). Specifically, the authentication assessment unit 194 acquires the image data of reflected image R acquired during the previous authentication, which is stored in the "Reflected Image" field of the reflected image authentication value table 182A shown in FIG. 17. The authentication assessment unit 194 compares the image data of reflected image R restored in step S611 of FIG. 19 with the image data of reflected image R acquired from the reflected image authentication value table 182A, and if there is a difference, it determines that the tilt of reflected image R has changed. On the other hand, if there is no difference, the authentication assessment unit 194 determines that the tilt of reflected image R has not changed.

[0147] If the tilt of the reflected image R has changed (step S703; YES), the authentication assessment unit 194 causes the reflected image processing unit 192 to correct the tilt of the reflected image R (step S704). The tilt of the reflected image R is corrected by, for example, a method such as tilt correction using an affine transformation or recognizing the four corners of the reflected image R and correcting them to be straight.

[0148] The authentication assessment unit 194 compares the corrected reflected image R with the display image data acquired from the display processing unit 195 in step S603 of FIG. 19 (step S705). The authentication assessment unit 194 calculates an authentication value from the comparison result (step S706). The authentication assessment unit 194 updates the data (step S707). Specifically, the authentication assessment unit 194 stores the image data of reflected image R restored in step S611 of FIG. 19 in "Reflected Image" in the reflected image authentication value table 182A shown in FIG. 17.

[0149] Furthermore, if the tilt angle of terminal device 1B has not changed in step S702 (step S702; NO), the authentication assessment unit 194 causes the motion information acquisition unit 193 to calculate the distance between the face of user U and terminal device 1B based on the facial image of user U acquired in step S602 of Fig. 19 (step S708). The authentication assessment unit 194 determines whether the distance between the face of user U and terminal device 1B has changed (step S709).

[0150] Specifically, first, the authentication assessment unit 194 acquires the distance value stored in the "latest information" for the type of action "distance between face and terminal device" in the authentication motion information database 183. The authentication assessment unit 194 compares the distance value acquired from the authentication motion information database 183 with the distance value between the face of user U and terminal device 1B calculated in step S708, and if there is a difference, it determines that the distance between the face of user U and terminal device 1B has changed. If there is no difference, the authentication assessment unit 194 determines that the distance between the face of user U and terminal device 1B has not changed.

[0151] If the distance between the face and the terminal device 1B has not changed (step S709; NO), the authentication assessment unit 194 proceeds to step S705 and executes steps S705 to S707. If the distance between the face and the terminal device 1B has changed (step S709; YES), the authentication assessment unit 194 determines whether the size of the reflected image R has changed (step S710).

[0152] Specifically, the authentication assessment unit 194 acquires the image data of the reflected image R acquired during the previous authentication, which is stored in the "Reflected Image" field of the reflected image authentication value table 182A shown in Fig. 17. The authentication assessment unit 194 compares the image data of the reflected image R restored in step S611 of Fig. 19 with the image data of the reflected image R acquired from the reflected image authentication value table 182A, and if there is a difference, it determines that the size of the reflected image R has changed. If there is no difference, the authentication assessment unit 194 determines that the size of the reflected image R has not changed.

[0153] If the size of the reflected image R has changed (step S710; YES), the authentication assessment unit 194 calculates a change rate from the difference in size between the image data of the reflected image R restored in step S611 of Fig. 19 and the image data of the reflected image R acquired from the reflected image authentication value table 182A (step S711). The authentication assessment unit 194 causes the reflected image processing unit 192 shown in Fig. 6 to correct the size of the image data of the reflected image R restored in step S611 using the change rate calculated in step S711 (step S712).

[0154] The authentication assessment unit 194 proceeds to step S705 and executes steps S705 and S706. The authentication assessment unit 194 updates the data (step S707). Specifically, the authentication assessment unit 194 stores the image data of reflected image R restored in step S611 of Fig. 19 in "Reflected Image" in the reflected-image authentication value table 182A shown in Fig. 17. The authentication assessment unit 194 also updates the distance value stored in "Latest Information" for the "Distance Between Face and Terminal Device" type of action in the authentication action information database 183 to the distance value calculated in step S708.

[0155] If the tilt of the reflected image R has not changed in step S703 (step S703; NO), and if the size of the reflected image R has not changed in step S710 (step S710; NO), the authentication assessment unit 194 sets the authentication value to a value greater than the authentication tolerance value (step S713). For example, if the authentication tolerance value is 0.45 as shown in FIG. 7A, the authentication information acquisition unit 191 sets the authentication value to a value such as 0.50.

[0156] The authentication assessment unit 194 ends the linked authentication value setting process and returns to Fig. 19. The authentication assessment unit 194 ends the linked reflected image authentication value acquisition process and returns to Fig. 18A. The authentication assessment unit 194 executes step S503 to step S519 shown in Fig. 18B of the linked authentication process. Step S503 to step S519 shown in Fig. 18B are the same as step S103 shown in Fig. 9A to step S119 shown in Fig. 9B.

[0157] As described above, in addition to the effects of the first and second embodiments, the terminal device 1B according to the third embodiment can authenticate the user U by taking into consideration the change in the size and tilt of the reflected image R linked to the change in the tilt of the terminal device 1B or the distance between the terminal device 1B and the face of the user U. This makes it possible to determine impersonation from the change in the size and tilt of the reflected image R.

[0158] (Variation 1) In the above-described first to third embodiments, the user U is authenticated using authentication based on the reflected image R and authentication based on the motion information of the user U. However, the present invention is not limited to this, and authentication may be performed by combining various authentication methods using biometric information such as finger vein authentication, iris authentication, and voice authentication.

[0159] (Variation 2) In each of the above first to third embodiments, the authentication process for user U can be performed by the authentication process program 180 shown in Fig. 4, the combined authentication process program 186 shown in Fig. 12, and the interlocked authentication process program 188 shown in Fig. 16. All or part of the steps performed by the authentication process program 180, the combined authentication process program 186, and the interlocked authentication process program 188 may be realized by semiconductor chips such as an ASIC (Application Specific Integrated Circuit) or a system LSI (Large-scale Integration), circuits formed of various circuit elements, or the like.

[0160] (Variation 3) In the above-described first to third embodiments, the pressure of the user U gripping the terminal device 1, 1A, or 1B, the distance between the user U's face and the terminal device 1, 1A, or 1B, the number of connections to a specific connection destination, and the connection status with a specific device are used as criteria for determining authentication based on the user U's motion information. This is not a limitation, and other methods may be used or may be included. For example, it is determined whether another device owned by the user is connected to the terminal device 1, 1A, or 1B via Bluetooth (registered trademark), and if connected, the user is authenticated. In order to use devices connected via Bluetooth (registered trademark), the devices must be "paired" with each other. For this reason, device connections via Bluetooth (registered trademark) are highly personal, and can be used as auxiliary authentication to authenticate the user. Furthermore, it may be possible to determine whether or not the user is the user himself / herself based on the pattern, regularity, etc. of the user U's behavior route acquired by the position detection unit 16 shown in Figures 4, 12, and 16, and to authenticate the user as the user himself / herself if the pattern, regularity, etc. of the behavior route match.

[0161] (Variation 4) In the above-described first to third embodiments, in authentication based on the reflected image R, the facial image of the user U is photographed once by the photographing unit 11. However, the present invention is not limited to this, and the photographing unit 11 may photograph the user U multiple times, and the reflected image R obtained from each facial image may be used for authentication.

[0162] (Variation 5) In the above-described first to third embodiments, if authentication of the user U is successful and the user U continues to operate the terminal device 1, 1A, or 1B, it can be determined that the user himself / herself is continuing to operate the terminal device 1, 1A, or 1B. In this case, the authentication threshold and authentication tolerance used for authenticating the reflected image R may be set looser. Also, the authentication interval may be lengthened. By doing so, it is possible to perform the minimum necessary authentication of the user himself / herself in the background, while conserving the use of resources in the terminal device 1, 1A, or 1B.

[0163] (Variation 6) In the above-described first to third embodiments, authentication is performed in the background at a predetermined authentication interval. However, the present invention is not limited to this; authentication may be performed randomly in the background without setting the timing and interval of authentication. For example, authentication may be performed whenever any of the various sensors mounted on the terminal devices 1, 1A, and 1B detects a spatial change, such as a change in the position or inclination, of the terminal devices 1, 1A, and 1B. Authentication may also be performed when the user U performs an operation for performing special processing or an irregular operation on the terminal devices 1, 1A, and 1B.

[0164] (Variation 7) In the above-described first to third embodiments, the pressure detection unit 17 detects the pressure with which the user U grips the terminal device 1, 1A, 1B. This is not limiting, and for example, the pressure detection unit 17 may be provided below a touch panel in which the display unit 20 is stacked on the operation input unit 14, and the pressure, contact area, etc. of the operation performed by the user U on the screen may be detected to determine the operating state of the user U. Furthermore, the operating state of the user U may be determined by detecting both the pressure with which the user U grips the terminal device 1, 1A, 1B by the pressure detection unit 17 and the pressure, contact area, etc. of the operation performed by the user U on the screen.

[0165] (Variation 8) In the above-described first to third embodiments, the processor 31 can be configured using a processing device such as a CPU, an MPU, etc. However, the present invention is not limited to this, and the processor 31 may include, for example, a GPU, which is the display controller 33, and a memory for processing, in addition to a processing device such as a CPU or an MPU.

[0166] (Modification 9) In the above-described first to third embodiments, an image obtained by extracting an image of the display content displayed on the screen of the display unit 20 of the terminal device 1 from among various images reflected in the pupil P is used as the reflected image R. However, the present invention is not limited to this, and various images reflected in the pupil P (all images reflected in the pupil P) may be used as is as the reflected image R. In this case, it is possible to determine changes in the distance between the terminal device 1 and the face of the user U, the tilt of the terminal device 1, the current position of the terminal device 1, etc., from not only changes in the size of the display content displayed on the screen of the display unit 20 of the terminal device 1 reflected in the pupil P but also changes in the images of objects placed around the user U, the background of the location, etc.

[0167] For example, the reference time is t ｓ At the reference time t ｓ After the time α has elapsed from α The elapsed time t α At the reference time t ｓ When the user U is farther away than at the reference time t , the range occupied by the image of the display content displayed on the screen of the display unit 20 of the terminal device 1 is smaller than the range occupied by the image of the items placed around the user U and the background of that place. ｓ It will be more than when

[0168] Also, the elapsed time t α At the reference time t ｓ When the user U is closer than at the reference time t , the range occupied by the image of the display content displayed on the screen of the display unit 20 of the terminal device 1 is smaller than the range occupied by the image of the items placed around the user U and the background of that place. ｓ Therefore, it is possible to determine a change in the distance between the terminal device 1 and the face of the user U from a change in the range occupied by the image of the display content displayed on the screen of the display unit 20 of the terminal device 1, and the range occupied by the images of objects placed around the user U, the background of that location, etc., among the images reflected in the pupil P.

[0169] Also, the elapsed time t αIn this case, the position and tilt of the image of the display content displayed on the screen of the display unit 20 of the terminal device 1 in the image reflected in the pupil P are ｓ If the tilt of the terminal device 1 has changed from the reference time t ｓ It can be judged that the time has changed from the time of elapsed time t α In the image reflected in the pupil P, the contents of the image such as the objects placed around the user U and the background of the place are ｓ If the current position of the terminal device 1 has changed from the reference time t ｓ That is, the terminal device 1 changes at the reference time t ｓ It can be determined that the position is different from that of the previous time.

[0170] As a result, even if the terminal device 1 does not include the tilt detection unit 13 and position detection unit 16 shown in Figure 4, or if the tilt detection unit 13 and position detection unit 16 become unusable due to a malfunction, operational error, etc., it is possible to determine changes in the distance between the terminal device 1 and the face of the user U, the tilt of the terminal device 1, the current position of the terminal device 1, etc., by using various images reflected in the pupil P (all images reflected in the pupil P), and the user U can be authenticated using the determination results.

[0171] (Variation 10) In the above-described third embodiment, before comparing the reflected image R with the image on the display screen when the user's face image was captured to determine an authentication value, the tilt and size of the reflected image R are corrected. Similarly, in the first and second embodiments, the tilt and size of the reflected image R may be corrected before comparing the reflected image R with the image on the display screen when the user's face image was captured to determine an authentication value.

[0172] (Variation 11) In the above-described embodiment 3, similar to embodiment 1, it is assumed that user U can be authenticated as the person in question when either authentication based on the reflected image R or authentication based on the motion information of user U is successful. However, it is also possible to assume that user U can be authenticated as the person in question when both authentication based on the reflected image R and authentication based on the motion information of user U are successful, similar to embodiment 2. In this case, the reflected-image authentication value acquisition process in step S402 of the combined authentication process in FIG. 15A is replaced with the linked reflected-image authentication value acquisition process shown in FIG. 19.

[0173] Furthermore, in the first to third embodiments of the present invention, the terminal devices 1, 1A, and 1B can be realized using a normal computer system, rather than a dedicated system. For example, a program for realizing each function of the terminal devices 1, 1A, and 1B may be stored and distributed on a computer-readable recording medium, such as a CD-ROM (Compact Disc Read Only Memory) or a DVD-ROM (Digital Versatile Disc Read Only Memory), and a computer may be configured to realize each of the above-described functions by installing the program on the computer. Furthermore, if each function is realized by sharing the work between an operating system (OS) and an application, or by cooperation between an OS and an application, only the application may be stored on the recording medium.

[0174] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to illustrate the present invention and do not limit the scope of the present invention. In other words, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and within the meaning of the disclosure equivalent thereto are considered to be within the scope of the present invention.

[0175] The present invention can be suitably used in a terminal device.

[0176] 1, 1A, 1B Terminal device, 10 Communication unit, 11 Photography unit, 11A In-camera, 11B Main camera, 12 Audio input / output unit, 12A Speaker, 12B Microphone, 13 Tilt detection unit, 14 Operation input unit, 15 Fingerprint detection unit, 15A Left fingerprint sensor, 15B Right fingerprint sensor, 16 Position detection unit, 17 Pressure detection unit, 17A Right pressure detection unit, 17B Left pressure detection unit, 18, 18A, 18B Terminal storage unit, 19 Terminal control unit, 20 Display unit, 31 Processor, 32 Memory, 33 Display controller, 34 Display device, 35 I / O port, 36 Storage device, 37 Communication device, 38 Data bus, 180 Authentication processing program, 181 Reflected image processing program, 182, 182A Reflected image authentication value table, 183 Authentication operation information database, 184 authentication count table, 185 waiting time setting table, 186 combined authentication processing program, 187 combined authentication count table, 188 linked authentication processing program, 191 authentication information acquisition unit, 192 reflected image processing unit, 193 operation information acquisition unit, 194 authentication determination unit, 195 display processing unit, 196 authentication information update unit.

Claims

1. A terminal device comprising: a reflected image processing unit that restores a reflected image of the display screen reflected in the user's eyes from a captured facial image of the user; a motion information acquisition unit that acquires motion information by the user; and an authentication determination unit that compares the reflected image with an image of the display screen at the time the user's facial image was captured to determine an authentication value, determines that the user is the user if the authentication value is smaller than a predetermined authentication threshold, and determines whether the user is the user based on the motion information by the user if the authentication value is larger than the predetermined authentication threshold and smaller than a predetermined authentication tolerance value different from the authentication threshold.

2. A terminal device comprising: a reflected image processing unit that restores an image of the display screen reflected in the user's eyes from a captured facial image of the user; a motion information acquisition unit that acquires motion information by the user; and an authentication determination unit that compares the reflected image with the image of the display screen at the time the facial image of the user was captured to determine an authentication value, and if the authentication value is smaller than a predetermined authentication threshold and the user can be authenticated as the user based on the motion information by the user, determines that the user is the user.

3. A terminal device according to claim 1 or 2, further comprising a tilt detection unit that detects the angle of tilt of the terminal device, wherein the authentication determination unit determines whether the angle of tilt of the terminal device detected by the tilt detection unit has changed from the angle at the time of the previous authentication, and if the angle of tilt of the terminal device has changed, determines whether the tilt of the reflected image has changed from the tilt of the reflected image at the time of the previous authentication, and if the tilt of the reflected image has changed, compares the reflected image with the image of the display screen when the user's face image was captured to determine the authentication value, and if the tilt of the reflected image has not changed, sets the authentication value to a value greater than a predetermined authentication tolerance value.

4. The terminal device described in any one of claims 1 to 3, wherein the authentication judgment unit determines the distance between the user and the terminal device based on the captured facial image of the user, determines whether the distance has changed from the distance at the time of the previous authentication, and if the distance has changed, determines whether the size of the reflected image has changed from the size of the reflected image at the time of the previous authentication, and if the size of the reflected image has changed, compares the reflected image with the image of the display screen at the time the user's facial image was captured to determine the authentication value, and if the size of the reflected image has not changed, sets the authentication value to a value greater than a predetermined authentication tolerance value.

5. A terminal device according to any one of claims 1 to 4, wherein the authentication assessment unit counts the number of failures as authentication failures when the authentication value is greater than a predetermined authentication tolerance value or when the user cannot be authenticated as the person in question based on the user's operation information, and locks the terminal device when the number of failures reaches or exceeds a predetermined number of assessments.

6. A method executed by a terminal device, comprising the steps of: restoring an image of the display screen reflected in the user's eyes from a captured facial image of the user; acquiring information about movements made by the user; comparing the reflected image with an image of the display screen at the time the facial image of the user was captured to determine an authentication value; determining that the user is the user if the authentication value is smaller than a predetermined authentication threshold; and determining whether the user is the user based on the information about the movements made by the user if the authentication value is larger than the predetermined authentication threshold and smaller than a predetermined authentication tolerance value different from the authentication threshold.

7. A method executed by a terminal device, comprising the steps of: restoring an image of a display screen reflected in the user's eyes from a captured facial image of the user; acquiring information about the user's movements; and collating the reflected image with the image of the display screen at the time the user's facial image was captured to determine an authentication value; and determining that the user is the user if the authentication value is smaller than a predetermined authentication threshold and the user can be authenticated based on the user's movement information.

8. A program for causing a computer to execute the following processes: a process of restoring an image of a display screen reflected in the user's eyes from a photographed image of the user's face; a process of acquiring information about movements made by the user; a process of finding an authentication value by comparing the reflected image with an image of the display screen at the time the user's face image was photographed, determining that the user is the user if the authentication value is smaller than a predetermined authentication threshold; and a process of determining whether the user is the user based on the information about movements made by the user if the authentication value is larger than the predetermined authentication threshold and smaller than a predetermined authentication tolerance value different from the authentication threshold.

9. A program for causing a computer to execute the following processes: a process of restoring an image of a display screen reflected in the user's eyes from a photographed image of the user's face; a process of acquiring information about the user's movements; a process of finding an authentication value by comparing the reflected image with the image of the display screen at the time the user's face image was photographed, and determining that the user is the person in question if the authentication value is smaller than a predetermined authentication threshold and the user can be authenticated based on the user's movement information.

Images