Authentication device, engine generation device, authentication method, engine generation method, and recording medium

Abstract

An authentication device 3 includes an authentication means 311 for authenticating an object person, using a person image IMG_P generated by a visible-light camera 1 performing image-capturing of the object person at a first time ta, and a determining means 312 for determining whether or not the object person is a living body, using a plurality of thermal images IMG_T generated by a thermal camera performing image-capturing of the object person at a second time tb1 that is the closest to the first time out of a plurality of times of the terminal camera 2 performing image-capturing of the object person, and a third time tb2 before or after the second time.

Description

[Technical Field]

[0001] This disclosure relates to the technical fields of, for example, an authentication device, an authentication method, and a recording medium capable of authenticating a subject appearing in a person image, as well as an engine generation device, an engine generation method, and a recording medium capable of generating a determination engine capable of determining whether a subject appearing in a person image is a living body. [Background technology]

[0002] An example of an authentication device capable of authenticating a subject who appears in a person image is described in Patent Document 1. Patent Document 1 describes a device that authenticates the subject using a facial image of the subject obtained from a camera, and determines whether the subject is a living body using a temperature distribution of the subject's face obtained from a thermograph.

[0003] Other prior art documents related to this disclosure include Patent Documents 2 to 5. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-115460 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-078052 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-067371 [Patent Document 4] International Publication No. 2009 / 107237 Brochure [Patent Document 5] Japanese Patent Application Laid-Open No. 2005-259049 Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure aims to provide an authentication device, an engine generation device, an authentication method, an engine generation method, and a recording medium that aim to improve upon the techniques described in prior art documents. [Means for solving the problem]

[0006] One aspect of the authentication device includes an authentication means for authenticating a subject using a human image generated by a visible light camera capturing an image of the subject at a first time, and a determination means for determining whether the subject is a living body using multiple thermal images generated by the thermal camera capturing an image of the subject at a second time closest to the first time and a third time before and after the second time among multiple times at which the thermal camera captured an image of the subject.

[0007] One aspect of the engine generation device is an engine generation device that generates a determination engine for determining whether a subject is a living organism using a thermal image generated by capturing an image of the subject with a thermal camera, and includes: an extraction means that extracts at least one sample image as an extracted image from a learning dataset that includes a plurality of sample images that show the body surface temperature distribution of the sample person and in which a region of interest that should be noted in order to determine whether the sample person is a living organism; an image generation means that generates a learning image by changing the positional relationship between the region of interest set in the extracted image and the part of interest of the sample person that should be noted in order to determine whether the sample person is a living organism, based on the imaging environment in which the thermal camera captures the subject; and an engine generation means that generates the determination engine by performing machine learning using the learning image.

[0008] One aspect of the authentication method includes authenticating a subject using a human image generated by a visible light camera capturing an image of the subject at a first time, and determining whether the subject is a living body using multiple thermal images generated by the thermal camera capturing an image of the subject at a second time closest to the first time and a third time before and after the second time among multiple times at which the subject was captured by a thermal camera.

[0009] One aspect of the engine generation method is an engine generation method that generates a determination engine for determining whether a subject is a living organism using a thermal image generated by capturing an image of the subject with a thermal camera, and includes: extracting at least one sample image as an extracted image from a learning dataset that includes a plurality of sample images that show the body surface temperature distribution of the sample person and in which a region of interest that should be noted in order to determine whether the sample person is a living organism; generating a learning image by changing the positional relationship between the region of interest set in the extracted image and the part of interest of the sample person that should be noted in order to determine whether the sample person is a living organism, based on the imaging environment in which the thermal camera captures the subject; and generating the determination engine by performing machine learning using the learning image.

[0010] One aspect of the recording medium is a recording medium having recorded thereon a computer program that causes a computer to execute an authentication method including: authenticating a subject using a person image generated by a visible light camera capturing an image of the subject at a first time; and determining whether the subject is a living body using multiple thermal images generated by the thermal camera capturing an image of the subject at a second time closest to the first time among multiple times at which the thermal camera captured an image of the subject, and at a third time before and after the second time.

[0011] Another aspect of the recording medium is an engine generation method for generating a determination engine for determining whether a subject is a living body using a thermal image generated by capturing an image of the subject with a thermal camera, the method including: extracting at least one sample image as an extracted image from a training dataset including a plurality of sample images that show the body surface temperature distribution of the sample person and in which a region of interest that should be noted for determining whether the sample person is a living body is set; generating a training image by changing the positional relationship between the region of interest set in the extracted image and the part of interest of the sample person that should be noted for determining whether the sample person is a living body based on the imaging environment in which the thermal camera captures the subject; and generating the determination engine by performing machine learning using the training image.The recording medium has recorded thereon a computer program that causes a computer to execute the engine generation method. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of an authentication device in the first embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of an engine generation device according to the second embodiment. [Figure 3] FIG. 3 is a block diagram showing the configuration of an authentication system according to the third embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of an authentication device in the third embodiment. [Figure 5] FIG. 5 is a flowchart showing the flow of authentication operations performed by the authentication device in the third embodiment. [Figure 6] FIG. 6 shows an example of a portrait image. [Figure 7] FIG. 7 is a timing chart showing the relationship between the authentication time and the time of interest (particularly, the closest time). [Figure 8] FIG. 8 shows the relationship between the face area of ​​a person image and the attention area of ​​a thermal image. [Figure 9]FIG. 9 is a timing chart showing the relationship between the authentication time and the time of interest (particularly the time before and after) in the first modified example. [Figure 10] FIG. 10 shows the relationship between the face area of ​​a person image and the attention area of ​​a thermal image. [Figure 11] FIG. 11 is a flowchart showing the flow of authentication operations in the second modified example. [Figure 12] FIG. 12 shows how the region of interest moves within a thermal image. [Figure 13] Each of FIGS. 13(a) and 13(b) is a graph showing the temperature distribution in a pixel row of a thermal image. [Figure 14] FIG. 14 shows a number of thermal images corresponding to a number of times of interest. [Figure 15] FIG. 15 is a block diagram showing the configuration of an authentication system according to the fourth embodiment. [Figure 16] FIG. 16 is a block diagram showing the configuration of an engine generation device according to the fourth embodiment. [Figure 17] FIG. 17 is a flowchart showing the flow of the engine generation operation performed by the engine generation device in the fourth embodiment. [Figure 18] FIG. 18 shows an example of the data structure of the training data set. [Figure 19] 19(a) and 19(b) show examples of training images generated from extracted images. [Figure 20] 20(a) and 20(b) show examples of training images generated from extracted images. [Figure 21] FIG. 21 shows an example of a training image. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of an authentication device, an engine generation device, an authentication method, an engine generation method, and a recording medium will be described.

[0014] (1) First embodiment First, a first embodiment of an authentication device, an engine generation device, an authentication method, an engine generation method, and a recording medium will be described. Hereinafter, with reference to Fig. 1, the authentication device, the authentication method, and the recording medium in the first embodiment will be described using an authentication device 1000 to which the authentication device, the authentication method, and the recording medium in the first embodiment are applied. Fig. 1 is a block diagram showing the configuration of the authentication device 1000 in the first embodiment.

[0015] 1, the authentication device 1000 includes an authentication unit 1001 and a determination unit 1002. The authentication unit 1001 authenticates the subject using a human image generated by capturing an image of the subject with a visible light camera at a first time. The determination unit 1002 determines whether the subject is a living body or not using multiple thermal images generated by capturing an image of the subject with the thermal camera at a second time closest to the first time and a third time before and after the second time among multiple times at which the thermal camera captured an image of the subject.

[0016] According to such an authentication device 1000, it is possible to determine with higher accuracy whether a subject is a living body or not, compared to a comparative authentication device that determines whether a subject is a living body or not without taking into account the first time when the visible camera captured the subject's image.

[0017] (2) Second embodiment Next, a second embodiment of an authentication device, an engine generation device, an authentication method, an engine generation method, and a recording medium will be described. Hereinafter, with reference to Fig. 2, the engine generation device, the engine generation method, and the recording medium according to the second embodiment will be described using an engine generation device 2000 to which the engine generation device, the engine generation method, and the recording medium according to the second embodiment are applied. Fig. 2 is a block diagram showing the configuration of the engine generation device 2000 according to the second embodiment.

[0018] The engine generation device 2000 is a device capable of generating a determination engine for determining whether a subject is a living body or not using a thermal image generated by capturing an image of the subject with a thermal camera. The determination engine may be used, for example, by an authentication device that determines whether a subject is a living body or not using a thermal image.

[0019] To generate a determination engine, the engine generation device 2000 includes an extraction unit 2001, an image generation unit 2002, and an engine generation unit 2003, as shown in FIG. 2. The extraction unit 2001 extracts at least one sample image as an extracted image from a training dataset including multiple sample images each showing the body surface temperature distribution of a sample person and each including a region of interest that should be noted for determining whether the sample person is a living body. The image generation unit 2002 generates a training image using the extracted image. Specifically, the image generation unit 2002 generates the training image by changing the positional relationship between the region of interest set in the extracted image and the part of interest of the sample person that should be noted for determining whether the sample person is a living body, based on the imaging environment in which the thermal camera captures the subject. The engine generation unit 2003 generates a determination engine by performing machine learning using the training image.

[0020] The engine generation device 2000 described above can generate a determination engine capable of determining with high accuracy whether a subject is a living body. Specifically, information about the imaging environment in which a thermal camera captures an image of the subject is reflected in the training image. Therefore, the engine generation device 2000 can generate a determination engine that reflects information about the imaging environment by performing machine learning using the training image that reflects information about the imaging environment. For example, the engine generation device 2000 can generate a determination engine ENG for determining whether a subject is a living body using a thermal image generated by a thermal camera capturing an image of the subject in a specific imaging environment by performing machine learning using the training image that reflects information about the specific imaging environment. As a result, by using a determination engine that reflects information about the specific imaging environment, the authentication device can determine with high accuracy whether a subject is a living body from a thermal image generated by a thermal camera capturing an image of the subject in a specific imaging environment, compared to using a determination engine that does not reflect information about the specific imaging environment. In this way, the engine generation device 2000 can generate a determination engine that can determine with high accuracy whether a subject is a living body from a thermal image generated by a thermal camera capturing an image of the subject in a specific imaging environment.

[0021] (3) Third embodiment Next, a third embodiment of the authentication device, engine generation device, authentication method, engine generation method, and recording medium will be described. Hereinafter, the authentication device, authentication method, and recording medium of the third embodiment will be described using an authentication system SYS3 to which the authentication device, authentication method, and recording medium of the third embodiment are applied.

[0022] (3-1) Configuration of authentication system SYS3 First, the configuration of the authentication system SYS3 in the third embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the authentication system SYS3 in the third embodiment.

[0023] As shown in Fig. 3, the authentication system SYS3 includes a visible camera 1, a thermal camera 2, and an authentication device 3. The visible camera 1 and the authentication device 3 may be able to communicate with each other via a communication network NW. The thermal camera 2 and the authentication device 3 may be able to communicate with each other via the communication network NW. The communication network NW may include a wired communication network. The communication network NW may include a wireless communication network.

[0024] The visible camera 1 is an imaging device capable of optically capturing an image of a subject located within the imaging range of the visible camera 1. In particular, the visible camera 1 is an imaging device capable of optically capturing an image of the subject by detecting visible light from the subject. The visible camera 1 captures an image of the subject and generates a person image IMG_P representing the subject captured by the visible camera 1. The person image IMG_P representing the subject is typically an image in which the subject P appears. Note that the "person image IMG_P in which the subject appears" may include an image generated by the visible camera 1 capturing an image of a subject who does not wish to be captured by the visible camera 1. The "person image IMG_P in which the subject appears" may include an image generated by the visible camera 1 capturing an image of a subject who wishes to be captured by the visible camera 1. The visible camera 1 transmits the generated person image IMG_P to the authentication device 3 via the communication network NW.

[0025] The thermal camera 2 is an imaging device capable of capturing an image of a subject located within the imaging range of the thermal camera 2. The thermal camera 2 captures an image of the subject to generate a thermal image IMG_T showing the body surface temperature distribution of the subject captured by the thermal camera 2. The thermal image IMG_T may be an image showing the body surface temperature distribution of the subject by color or gradation. The thermal image IMG_T showing the body surface temperature of the subject may typically be an image in which the subject P is substantially captured by the body surface temperature distribution of the subject. Note that the "thermal image IMG_T capturing the subject" may include an image generated by the thermal camera 2 capturing an image of a subject who does not wish to be captured by the thermal camera 2. The "thermal image IMG_T capturing the subject" may include an image generated by the thermal camera 2 capturing an image of a subject who wishes to be captured by the thermal camera 2. The thermal camera 2 transmits the generated thermal image IMG_T to the authentication device 3 via the communication network NW.

[0026] The visible camera 1 and the thermal camera 2 are aligned so that the visible camera 1 and the thermal camera 2 can capture images of the same subject. In other words, the visible camera 1 and the thermal camera 2 are aligned so that the imaging range of the visible camera 1 and the imaging range of the thermal camera 2 at least partially overlap. For this reason, a subject who appears in a person image IMG_P generated by the visible camera 1 during a certain time period will usually also appear in the thermal image IMG_T generated by the thermal camera 2 during the same time period. In other words, the same subject appears in the person image IMG_P generated by the visible camera 1 and the thermal image IMG_T generated by the thermal camera 2 during a certain time period.

[0027] The authentication device 3 acquires a person image IMG_P from the visible camera 1. The authentication device 3 uses the acquired person image IMG_P to perform an authentication operation to authenticate the subject person appearing in the person image IMG_P. That is, the authentication device 3 uses the acquired person image IMG_P to determine whether the subject person appearing in the person image IMG_P is the same as a person registered in advance (hereinafter referred to as a "registered person"). If it is determined that the subject person appearing in the person image IMG_P is the same as the registered person, it is determined that authentication of the subject has been successful. On the other hand, if it is determined that the subject person appearing in the person image IMG_P is not the same as the registered person, it is determined that authentication of the subject has failed.

[0028] Here, a malicious person may cause the visible camera 1 to capture an image (for example, a photograph with an image printed on it or a display with an image displayed on it) in which the registered person appears, in order to impersonate the registered person. In this case, the authentication device 3 may determine that the authentication of the target person has been successful, even though the registered person is not actually in front of the visible camera 1, in the same way as if the registered person were present in front of the visible camera 1. In other words, a malicious person may impersonate the registered person. Therefore, as part of the authentication operation, the authentication device 3 determines whether the target person appearing in the person image IMG_P is a living body. Specifically, the authentication device 3 acquires a thermal image IMG_T from the thermal camera 2. The authentication device 3 uses the acquired thermal image IMG_T to determine whether the target person appearing in the thermal image IMG_T is a living body. As described above, the same target person appears in the person image IMG_P generated by the visible camera 1 and the thermal image IMG_T generated by the thermal camera 2 during a certain time period. Therefore, the operation of determining whether or not a subject appearing in the thermal image IMG_T is a living body is equivalent to the operation of determining whether or not a subject appearing in the person image IMG_P is a living body.

[0029] Such an authentication system SYS3 may be used, for example, to manage the entry and exit of subjects to and from a restricted area. Specifically, a restricted area is an area in which subjects who meet certain entry conditions are permitted to enter, while subjects who do not meet the certain entry conditions are not permitted (i.e., prohibited) to enter. In this case, the authentication device 3 may authenticate the subject by determining whether the subject appearing in the person image IMG_P is identical to a person permitted to enter the restricted area (e.g., a person pre-registered as a person who meets the entry conditions). In this case, if it is determined that the subject is identical to the person permitted to enter the restricted area (i.e., if authentication is successful), the authentication device 3 may permit the subject to enter the restricted area. As an example, the authentication device 3 may set the state of an entry / exit restriction device (e.g., a gate device or a door device) that can restrict the subject's passage to an open state that allows the subject to pass through the entry / exit restriction device. On the other hand, if it is determined that the subject is not identical to the person permitted to enter the restricted area (i.e., if authentication fails), the authentication device 3 may prohibit the subject from entering the restricted area. As an example, the authentication device 3 may set the state of the entrance / exit restriction device to a closed state in which the target person cannot pass through the entrance / exit restriction device. Furthermore, even if the target person is successfully authenticated, if the target person reflected in the person image IMG_P is determined to be not a living body, the authentication device 3 may prohibit the target person from entering the restricted area.

[0030] When the authentication system SYS3 is used to manage the entry and exit of subjects to and from a restricted area, each of the visible camera 1 and the thermal camera 2 may capture an image of a subject attempting to enter the restricted area. As an example, each of the visible camera 1 and the thermal camera 2 may be disposed near an entry / exit control device and capture an image of a subject positioned near the entry / exit control device in order to enter the restricted area. In this case, each of the visible camera 1 and the thermal camera 2 may capture an image of a subject moving toward the entry / exit control device. Each of the visible camera 1 and the thermal camera 2 may capture an image of a subject moving toward the visible camera 1 and the thermal camera 2 disposed near the entry / exit control device. Alternatively, each of the visible camera 1 and the thermal camera 2 may capture an image of a subject standing still in front of the entry / exit control device. Each of the visible camera 1 and the thermal camera 2 may capture an image of a subject standing still in front of the visible camera 1 and the thermal camera 2 disposed near the entry / exit control device.

[0031] (3-2) Configuration of authentication device 3 Next, the configuration of the authentication device 3 will be described with reference to Fig. 4. Fig. 3 is a block diagram showing the configuration of the authentication device 3.

[0032] 4, the authentication device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The authentication device 3 may further include an input device 34 and an output device 35. However, the authentication device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0033] The arithmetic device 31 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic device 31 reads a computer program. For example, the arithmetic device 31 may read a computer program stored in the storage device 32. For example, the arithmetic device 31 may read a computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the authentication device 3. The arithmetic device 31 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the authentication device 3 via the communication device 33 (or another communication device). The arithmetic device 31 executes the read computer program. As a result, logical functional blocks for executing operations to be performed by the authentication device 3 (for example, the above-mentioned authentication operation) are realized within the arithmetic device 31. That is, the arithmetic device 31 can function as a controller for realizing logical function blocks for executing the operations (in other words, processing) that the authentication device 3 should perform.

[0034] Fig. 4 shows an example of logical functional blocks implemented in the computing device 31 for performing authentication operations. As shown in Fig. 4, an authentication unit 311, a biometric determination unit 312, and an entrance / exit management unit 313 are implemented in the computing device 31.

[0035] The authentication unit 311 acquires a person image IMG_P from the visible camera 1 via the communication network NW using the communication device 33. Furthermore, the authentication unit 311 uses the acquired person image IMG_P to determine whether the subject person appearing in the person image IMG_P is the same as a registered person. Information about the registered persons may be stored in the storage device 32 as a registered person DB 321.

[0036] The liveness determination unit 312 acquires a thermal image IMG_T from the thermal camera 2 via the communication network NW using the communication device 33. Furthermore, the liveness determination unit 312 uses the acquired thermal image IMG_T to determine whether or not the subject appearing in the thermal image IMG_T (i.e., the subject appearing in the person image IMG_P) is a living body. For example, the liveness determination unit 312 may determine that the subject appearing in the thermal image IMG_T is a living body when the similarity between the body surface temperature distribution of the subject appearing in the thermal image IMG_T and a body surface temperature distribution previously registered as the body surface temperature distribution of a living body (particularly a human) (hereinafter referred to as a "registered body surface temperature distribution") is higher than a predetermined threshold. Note that this threshold may be a fixed value. Alternatively, the threshold may be changeable. For example, the threshold may be changeable by a user of the authentication system SYS3.

[0037] The information on registered body surface temperature distribution may be stored in the storage device 32 as a registered body surface temperature distribution DB 322. The information on registered body surface temperature distribution may include information on the body surface temperature distribution of a general living organism (especially a human) (e.g., the average body surface temperature distribution of a human). The information on registered body surface temperature distribution may include information on the body surface temperature distribution of a registered person (i.e., a registered person registered in advance in the registration DB 321) used for face authentication (i.e., the body surface temperature distribution of a specific person).

[0038] The entrance / exit management unit 313 controls the state of the entrance / exit restriction device that can restrict the passage of a target person attempting to enter a restricted area based on the judgment results of the authentication unit 311 and the judgment results of the biometric judgment unit 312.

[0039] However, when the authentication system SYS3 is not used to manage the entrance and exit of subjects to and from the restricted area, the authentication device 3 does not need to include the entrance and exit management unit 313. Alternatively, even when the authentication system SYS3 is used to manage the entrance and exit of subjects to and from the restricted area, the authentication device 3 does not need to include the entrance and exit management unit 313.

[0040] The storage device 32 can store desired data. For example, the storage device 32 may temporarily store a computer program executed by the arithmetic device 31. The storage device 32 may temporarily store data that the arithmetic device 31 temporarily uses when the arithmetic device 31 is executing a computer program. The storage device 32 may store data that the authentication device 3 stores long-term. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 32 may include a non-temporary recording medium.

[0041] In the third embodiment, as described above, the storage device 32 stores a registered person DB 321 that is mainly referenced by the authentication unit 311 to authenticate the subject, and a registered body surface temperature distribution DB 322 that is mainly referenced by the biometric determination unit 312 to determine whether the subject is a living person.

[0042] The communication device 33 can communicate with each of the visible camera 1 and the thermal camera 2 via the communication network NW. In the third embodiment, the communication device 33 receives (i.e., acquires) a person image IMG_P from the visible camera 1 via the communication network NW. Furthermore, the communication device 33 receives (i.e., acquires) a thermal image IMG_T from the thermal camera 2 via the communication network NW.

[0043] The input device 34 is a device that accepts information input to the authentication device 3 from outside the authentication device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the authentication device 3. For example, the input device 34 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the authentication device 3.

[0044] The output device 35 is a device that outputs information to the outside of the authentication device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) that can output sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) that can print desired information on paper.

[0045] (3-3) Authentication Operation Performed by Authentication Device 3 Next, the flow of the authentication operation performed by the authentication device 3 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of the authentication operation performed by the authentication device 3.

[0046] As shown in FIG. 5, the communication device 33 acquires a person image IMG_P from the visible camera 1 via the communication network NW (step S10). The visible camera 1 normally continues to capture an image of the imaging range at a constant imaging rate. For example, the visible camera 1 continues to capture an image of the imaging range at an imaging rate of N1 times per second (where N1 is an integer equal to or greater than 1). Therefore, the communication device 33 may acquire a plurality of person images IMG_P, which are time-series data. The plurality of person images IMG_P acquired by the communication device 33 may be stored in the storage device 32.

[0047] Furthermore, the communication device 33 acquires a thermal image IMG_T from the thermal camera 2 via the communication network NW (step S11). The thermal camera 2 normally continues to capture an image of the imaging range at a constant imaging rate. For example, the thermal camera 2 continues to capture an image of the imaging range at an imaging rate of N2 (where N2 is an integer greater than or equal to 1) times per second. Therefore, the communication device 33 may acquire multiple thermal images IMG_T, which are time-series data. The multiple thermal images IMG_T acquired by the communication device 33 may be stored in the storage device 32.

[0048] When the person image IMG_P is acquired in step S11, the authentication unit 311 uses the person image IMG_P acquired in step S10 to authenticate the subject person appearing in the person image IMG_P (step S12). In the third embodiment, an example will be described in which the authentication unit 311 authenticates the subject person using the subject person's face. That is, an example will be described in which the authentication unit 311 performs face authentication. However, the authentication unit 311 may authenticate the subject person using another authentication method using the person image IMG_P. For example, the authentication unit 311 may authenticate the subject person using the subject person's iris.

[0049] To perform face authentication, the authentication unit 311 detects a face area FA in which the face of the subject is captured from the person image IMG_P, as shown in FIG. 6 illustrating an example of the person image IMG_P. Then, the authentication unit 311 may extract facial feature points of the subject included in the facial area FA. Then, the authentication unit 311 may calculate the similarity between the facial feature points of the subject included in the facial area FA and the facial feature points of the registered person. If the similarity between the facial feature points of the subject and the facial feature points of the registered person is higher than a predetermined authentication threshold, the authentication unit 311 may determine that the subject is the same as the registered person. If the similarity between the facial feature points of the subject and the facial feature points of the registered person is lower than a predetermined authentication threshold, the authentication unit 311 may determine that the subject is not the same as the registered person.

[0050] Referring again to FIG. 5, if the authentication in step S12 is not successful (i.e., it is determined that the subject is not the same as the registered person) (step S13: No), the entry / exit management unit 313 prohibits the subject from entering the restricted area (step S19).

[0051] On the other hand, if the authentication in step S12 is successful (i.e., the subject is determined to be the same as the registered person) (step S13: Yes), the biometric determination unit 312 then determines whether the subject determined to be the same as the registered person in step S12 is biometric (steps S14 to S16).

[0052] Specifically, the biometric determination unit 312 acquires authentication time ta (step S14). The authentication time ta indicates the time when one of the multiple person images IMG_P acquired in step S10, which was actually used to authenticate the subject in step S12, was captured. In other words, the authentication time ta indicates the time when one of the multiple person images IMG_P acquired in step S10, from which feature points determined to have a similarity with the facial feature points of a registered person higher than a predetermined authentication threshold, was captured.

[0053] Thereafter, the biometric determination unit 312 acquires, from the plurality of thermal images IMG_T acquired in step S11, a thermal image IMG_T captured at a time of interest tb determined based on the authentication time ta acquired in step S14 (step S15). In other words, the biometric determination unit 312 acquires a thermal image IMG_T captured at a time of interest tb, which is at least one of the plurality of times at which the plurality of thermal images IMG_T acquired in step S11 were captured (step S15). The plurality of thermal images IMG_T acquired in step S11 are stored in, for example, the storage device 32. In this case, the biometric determination unit 312 may acquire the thermal image IMG_T captured at the time of interest tb from the storage device 32.

[0054] In the third embodiment, the target time tb includes a closest time tb1 to the authentication time ta among the multiple times at which the multiple thermal images IMG_T obtained in step S11 were captured. Specific examples of the target time tb (particularly, the closest time tb1) determined based on the authentication time ta will be described below with reference to Fig. 7. Fig. 7 is a timing chart showing the relationship between the authentication time ta and the target time tb (particularly, the closest time tb1).

[0055] 7, the visible light camera 1 captures images of the subject at times t11, t12, t13, t14, and t15. The thermal camera 2 captures images of the subject at times t21, t22, t23, and t24. The visible light camera 1 and the thermal camera 2 do not necessarily capture images of the subject at synchronized times. In this case, the times t11, t12, t13, t14, and t15 at which the visible light camera 1 captures images of the subject are not necessarily synchronized with the times t21, t22, t23, and t24 at which the thermal camera 2 captures images of the subject.

[0056] Here, it is assumed that the authentication unit 311 authenticates the subject using a person image IMG_P generated by capturing an image of the subject at time t13. In this case, the authentication time ta is time t13. As a result, the time t23 closest to time t13 (i.e., the time with the smallest difference from time t13) is the closest time tb1. As a result, in the example shown in FIG. 7, the biometric determination unit 312 acquires a thermal image IMG_T captured at time t23, which is the closest time tb1.

[0057] 5, the biometrics determination unit 312 then uses the thermal image IMG_T acquired in step S15 to determine whether the subject appearing in the thermal image IMG_T is a living body (step S16). As described above, the visible light camera 1 and the thermal camera 2 are aligned so that the visible light camera 1 and the thermal camera 2 can capture images of the same subject. Therefore, there is a high possibility that the subject appearing in the person image IMG_P (i.e., the subject determined to be the same as the registered person in step S12) is captured in the thermal image IMG_T acquired in step S15. This is because the thermal image IMG_T acquired in step S15 is generated by the thermal camera 2 capturing an image of the subject at a time of interest tb (particularly, the closest time tb1) determined based on the authentication time ta. Therefore, the operation of determining whether the subject appearing in the thermal image IMG_T acquired in step S15 is a living body is equivalent to the operation of determining whether the subject determined to be the same as the registered person in step S12 is a living body.

[0058] In order to determine whether the subject is a living body, the living body determination unit 312 specifies, in the thermal image IMG_T, an area corresponding to the face area FA detected to authenticate the subject, as an attention area TA to be noted for determining whether the subject is a living body, as shown in Fig. 8. Specifically, as described above, the visible camera 1 and the thermal camera 2 are aligned so that the visible camera 1 and the thermal camera 2 can capture images of the same subject. In other words, the visible camera 1 and the thermal camera 2 are aligned so that the imaging range of the visible camera 1 and the imaging range of the thermal camera 2 at least partially overlap. In this case, a first area in the person image IMG_P and a second area in the thermal image IMG_T that captures the same scene as the first area correspond to each other. Therefore, the biometric determination unit 312 can use a projection transformation matrix based on the positional relationship between the visible camera 1 and the thermal camera 2 to identify an attention area TA in the thermal image IMG_T that corresponds to the facial area FA of the person image IMG_P (i.e., that is expected to contain the same scene as that captured in the facial area FA).

[0059] Thereafter, the living body determination unit 312 determines whether or not the subject is a living body based on the temperature distribution within the attention area TA. Here, since the attention area TA corresponds to the face area FA, there is a high possibility that the subject's face is captured in the attention area TA. Therefore, the operation of determining whether or not the subject is a living body based on the temperature distribution within the attention area TA is equivalent to the operation of determining whether or not the subject is a living body based on the body surface temperature distribution of the subject (in particular, the body surface temperature distribution of the face, which is an example of a target part of the subject that should be noted in order to determine whether or not the subject is a living body).

[0060] If it is determined in step S16 that the subject is not a living body (step S17: No), the entrance / exit management unit 313 prohibits the subject from entering the restricted area (step S19).

[0061] On the other hand, if the result of the determination in step S16 is that the subject is determined to be a living body (step S17: Yes), the entrance / exit management unit 313 permits the subject to enter the restricted area (step S18).

[0062] (3-4) Technical Effects As described above, in the third embodiment, the authentication device 3 determines whether or not the subject is a living body by using the thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at a time of interest tb (particularly, the closest time tb1) determined based on the authentication time ta. Therefore, compared to the authentication device of the comparative example that determines whether or not the subject is a living body by using the thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at an arbitrary time not taking the authentication time ta into consideration, the authentication device 3 can more accurately determine whether or not the subject is a living body.

[0063] Specifically, the authentication device of the comparative example may determine whether the subject is a living body using a thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at a time significantly different from the authentication time ta. However, in a thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at a time significantly different from the authentication time ta, the subject's face may not be captured in the attention area TA. This is because, if the thermal camera 2 captures the subject at a time significantly different from the authentication time ta, the positional relationship between the subject, the visible camera 1, and the thermal camera 2 at the authentication time ta may differ from the positional relationship between the subject, the visible camera 1, and the thermal camera 2 at the time the thermal camera 2 captured the subject. This is particularly noticeable when the subject is moving. In this case, the subject's face may not be properly captured in the attention area TA of the thermal image IMG_T identified from the face area FA of the person image IMG_P. For example, the subject's face may be captured at a position off-center of the attention area TA of the thermal image IMG_T. In this case, the authentication device of the comparative example may determine whether the subject is a living body or not based on the temperature distribution of the attention area TA in the thermal image IMG_T where the subject is not properly captured (i.e., a temperature distribution different from the body surface temperature distribution of the subject).As a result, the authentication device of the comparative example may have a poor accuracy in determining whether the subject is a living body or not.

[0064] However, in the third embodiment, the authentication device 3 determines whether the subject is a living body by using a thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at the closest time tb1 to the authentication time ta when the visible light camera 1 captured the image of the subject. In other words, the authentication device 3 does not determine whether the subject is a living body by using a thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at a time significantly different from the authentication time ta. As a result, the time when the visible light camera 1 captures an image of the subject to authenticate the subject (i.e., the authentication time) and the time when the thermal camera 2 captures an image of the subject to determine whether the subject is a living body (i.e., the closest time tb1) are close to each other. Therefore, the subject's face is likely to be properly captured in the attention area TA of the thermal image IMG_T identified from the face area FA of the person image IMG_P. Therefore, the authentication device 3 can properly determine whether the subject is a living body based on the temperature distribution of the attention area TA of the thermal image IMG_T where the subject is properly captured (i.e., the subject's body surface temperature distribution). As a result, there is a low possibility that the accuracy of determining whether or not a subject is a living body will deteriorate in the authentication device 3. In other words, the authentication device 3 can determine whether or not a subject is a living body with higher accuracy than the authentication device of the comparative example.

[0065] (3-5) Modifications Next, a modified example of the authentication device 3 in the third embodiment will be described. However, the authentication device 1000 in the first embodiment may also employ the same components as the modified example described below.

[0066] (3-5-1) First Modification In the above description, the time of interest tb is the closest time tb1 to the authentication time ta. In the first modified example, in addition to the closest time tb1, at least one previous or next time tb2, which is a time before or after the closest time tb1, is used as the time of interest tb. That is, in the first modified example, the time of interest tb may include, in addition to the closest time tb1, at least one previous or next time tb2, which is a time before or after the closest time tb1, among the multiple times at which the multiple thermal images IMG_T acquired in step S11 of FIG. 5 were captured. In this case, the biometric determination unit 312 acquires multiple thermal images IMG_T in step S15 of FIG. 5, including the thermal image IMG_T captured at the closest time tb1 and the thermal image IMG_T captured at the previous or next time tb2.

[0067] Note that the time before or after the closest time tb1 refers to at least one of a time after the closest time tb1 and a time before the closest time tb1. Furthermore, when both the closest time tb1 and the previous or next time tb2 are used as the target time tb, the closest time tb1 and at least one previous or next time tb2 constitute at least two consecutive times among the multiple times at which the multiple thermal images IMG_T acquired in step S11 were captured. In other words, the thermal image IMG_T captured at the closest time tb1 and at least two thermal images IMG_T captured at at least one previous or next time tb2 constitute at least two thermal images IMG_T that are temporally consecutive among the multiple thermal images IMG_T acquired in step S11 in FIG. 5.

[0068] A specific example of the time of interest tb (particularly the time before and after tb2) determined based on the authentication time ta will be described below with reference to Fig. 9. Fig. 9 is a timing chart showing the relationship between the authentication time ta and the time of interest tb.

[0069] In the example shown in Fig. 9, similar to the example shown in Fig. 7, the visible light camera 1 captures images of the subject at times t11, t12, t13, t14, and t15, respectively. The thermal camera 2 captures images of the subject at times t21, t22, t23, and t24, respectively.

[0070] Here, it is assumed that the authentication unit 311 authenticates the subject using a person image IMG_P generated by capturing an image of the subject at time t13. In this case, the authentication time ta is time t13. As a result, the time t23 closest to time t13 (i.e., the time with the smallest difference from time t13) becomes the closest time tb1. Furthermore, time t22 before time t23 may be used as the previous or next time tb2. Furthermore, time t24 after time t23 may be used as the previous or next time tb2.

[0071] 9, in step S15 of Fig. 5, the biometric determination unit 312 acquires a thermal image IMG_T captured at time t23, which is the closest time tb1. Furthermore, the biometric determination unit 312 acquires at least one of a thermal image IMG_T captured at time t22, which is the time tb2 before or after the time tb1, and a thermal image IMG_T captured at time t24, which is the time tb2 before or after the time tb1.

[0072] 5, the biometrics determination unit 312 may determine whether the subject appearing in the thermal image IMG_T is a living body by using at least one of the thermal images IMG_T acquired in step S15. Even when multiple thermal images IMG_T are acquired in step S15, it is highly likely that the subject appearing in the person image IMG_P (i.e., the subject determined to be the same as the registered person in step S12) is included in each of the thermal images IMG_T acquired in step S15, just as in the case where a single thermal image IMG_T is acquired in step S15. This is because the thermal image IMG_T acquired in step S15 is generated by the thermal camera 2 capturing an image of the subject at a time of interest tb (specifically, a time tb1 closest to the authentication time ta and a time tb2 before and after the authentication time ta) determined based on the authentication time ta. Therefore, the living body determination unit 312 can appropriately determine whether or not the subject is a living body by using at least one of the multiple thermal images IMG_T acquired in step S15.

[0073] However, as shown in FIG. 10 , which illustrates multiple thermal images IMG_T, not all of the multiple thermal images IMG_T necessarily capture the subject's face properly in the attention area TA. For example, in the example shown in FIG. 10 , the thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at time t22, which corresponds to time tb2 before and after the target time, captures the subject's face properly near the center of the attention area TA. Meanwhile, the thermal images IMG_T generated by the thermal camera 2 capturing an image of the subject at time t23, which corresponds to the closest time tb1, and at time t24, which corresponds to time tb2 before and after the target time, capture at least a portion of the subject's face outside the attention area TA. This is particularly noticeable when the subject is moving. In this case, the liveness determination unit 312 may select at least one thermal image IMG_T from the multiple thermal images IMG_T in which the subject's face is properly captured near the center of the attention area TA, and use the selected at least one thermal image IMG_T to determine whether the subject is a living body. Alternatively, the living body determination unit 312 may calculate the similarity between the body surface temperature distribution of the subject captured in the thermal image IMG_T and the registered body surface temperature distribution for each of the multiple thermal images IMG_T, and determine whether the subject is a living body or not using the statistical values ​​of the multiple calculated similarities. For example, the living body determination unit 312 may determine that the subject is a living body if the average, mode, median, or maximum of the multiple similarities is higher than a threshold.

[0074] As described above, in the first modified example, the authentication device 3 can determine whether or not the subject is a living body by using not only the thermal image IMG_T captured at the closest time tb1 but also the thermal images IMG_T captured before and after the time tb2. Therefore, in a situation where at least a part of the subject's face is captured in a position outside the attention area TA in the thermal image IMG_T captured at the closest time tb1, the authentication device 3 can determine whether or not the subject is a living body with higher accuracy.

[0075] (3-5-2) Second Modification In the second modified example, the authentication device 3 may adjust the position of the attention area TA, which is identified from the position of the face area FA of the person image IMG_P, within the thermal image IMG_T. The authentication operation in the second modified example will be described below with reference to Fig. 11. Fig. 11 is a flowchart showing the flow of the authentication operation in the second modified example. Note that the same step numbers are used for processes that have already been described, and detailed descriptions thereof will be omitted.

[0076] As shown in FIG. 11, in the second modified example, the authentication device 3 also performs the operations from step S10 to step S15.

[0077] Thereafter, the liveness determination unit 312 uses the thermal image IMG_T acquired in step S15 to determine whether the subject appearing in the thermal image IMG_T is a living body (step S16b). Specifically, as described above, the liveness determination unit 312 first identifies, in the thermal image IMG_T, an area corresponding to the face area FA detected to authenticate the subject, as an attention area TA to be focused on in order to determine whether the subject is a living body (step S161b). Thereafter, the liveness determination unit 312 adjusts the position of the attention area TA identified from the position of the face area FA within the thermal image IMG_T (step S162b). Thereafter, the liveness determination unit 312 determines whether the subject is a living body based on the temperature distribution within the attention area TA whose position has been adjusted (step S163b). Note that the processes of steps S161b and S163b may be the same as the operation of step S16 in FIG. 5 described above.

[0078] The liveness determination unit 312 may adjust the position of the attention region TA by moving the attention region TA within the thermal image IMG_T, as shown in Fig. 12. For example, the liveness determination unit 312 may adjust the position of the attention region TA in the vertical direction by moving the attention region TA along the vertical direction of the thermal image IMG_T. For example, in addition to or instead of moving the attention region TA along the vertical direction of the thermal image IMG_T, the liveness determination unit 312 may adjust the position of the attention region TA in the horizontal direction by moving the attention region TA along the horizontal direction of the thermal image IMG_T. Note that Fig. 12 shows an example in which the liveness determination unit 312 moves the attention region TA along the horizontal direction of the thermal image IMG_T.

[0079] The biometrics determination unit 312 may adjust the position of the attention area TA based on the thermal image IMG_T acquired in step S15. For example, the biometrics determination unit 312 may adjust the position of the attention area TA based on the temperature distribution indicated by the thermal image IMG_T acquired in step S15. Specifically, in the thermal image IMG_T, the temperature indicated by the image portion in which the subject is captured is usually different from the temperature indicated by the image portion in which the subject is not captured (e.g., the image portion in which the background of the subject is captured). For example, the temperature indicated by the image portion in which the subject is captured is higher than the temperature indicated by the image portion in which the subject is not captured. Therefore, it can be said that the temperature distribution indicated by the thermal image IMG_T indirectly indicates the position in which the subject is captured in the thermal image IMG_T. Therefore, the biometrics determination unit 312 may adjust the position of the attention area TA based on the thermal image IMG_T acquired in step S15 so that the attention area TA moves toward the position in the thermal image IMG_T in which the subject is captured.

[0080] As an example, FIG. 13(a) shows the temperature distribution in a pixel row including multiple pixels arranged horizontally among the multiple images constituting the thermal image IMG_T, and FIG. 13(b) shows the temperature distribution in a pixel row including multiple pixels arranged vertically among the multiple images constituting the thermal image IMG_T. As shown in FIGS. 13(a) and 13(b), the temperature indicated by the portion of the image in which the subject appears differs from the temperature indicated by the portion of the image in which the subject does not appear. Therefore, the position of the subject (e.g., the center of the face) can be estimated from the temperature distribution of the pixel row. For this reason, the biometric determination unit 312 may calculate the temperature distribution of the pixel row based on the thermal image IMG_T acquired in step S15, and may adjust the position of the attention region TA based on the temperature distribution of the pixel row so that the attention region TA moves toward the face of the subject within the thermal image IMG_T. The biometric determination unit 312 may adjust the position of the attention region TA so that the center of the attention region TA moves toward the center of the face of the subject within the thermal image IMG_T. The biometric determination unit 312 may adjust the position of the attention area TA so that the center of the subject's face and the center of the attention area TA coincide with each other in the thermal image IMG_T.

[0081] In this way, in the second modified example, since the position of the attention area TA is adjustable, the authentication device 3 can appropriately determine whether or not the subject is a living body based on the temperature distribution of the attention area TA in the thermal image IMG_T where the subject is properly captured (i.e., the body surface temperature distribution of the subject). The authentication device 3 can more accurately determine whether or not the subject is a living body.

[0082] (3-5-3) Third Modification As described in the second modified example, the temperature distribution shown in the thermal image IMG_T indirectly indicates the position where the subject is captured in the thermal image IMG_T. In this case, the authentication device 3 may determine whether the subject's face is properly captured in the attention area TA or whether at least a portion of the subject's face is captured outside the attention area TA in the thermal image IMG_T used to determine whether the subject is a living body. If it is determined that at least a portion of the subject's face is captured outside the attention area TA, the authentication device 3 may determine whether the subject is a living body using another thermal image IMG_T in which the subject's face is properly captured in the attention area TA. As an example, the authentication device 3 may determine whether the subject is a living body using another thermal image IMG_T in which the subject's face is captured at the center of the attention area TA or relatively close to the center.

[0083] As an example, in the first modified example described above, the authentication device 3 acquires a plurality of thermal images IMG_T corresponding to a plurality of target times tb, respectively. In this case, the authentication device 3 may determine whether the subject's face is properly captured in the target area TA in each of the plurality of thermal images IMG_T, or whether at least a portion of the subject's face is captured in a position outside the target area TA. In this case, the authentication device 3 may select one thermal image IMG_T from the plurality of thermal images IMG_T in which the subject's face is properly captured in the target area TA, and use the selected one thermal image IMG_T to determine whether the subject is a living body. 14, which shows a plurality of thermal images IMG_T corresponding to a plurality of times of interest tb, in the thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at time t22, the subject's face is properly captured in the area of ​​interest TA, while in the thermal images IMG_T generated by the thermal camera 2 capturing an image of the subject at time t23 and time t24, at least a part of the subject's face is captured in a position outside the area of ​​interest TA. In this case, based on the temperature distribution shown in the plurality of thermal images IMG_T, the authentication device 3 may select the thermal image IMG_T generated by the thermal camera 2 capturing an image of the subject at time t22 as one thermal image IMG_T in which the subject's face is properly captured in the area of ​​interest TA.

[0084] In this way, in the third modification, the authentication device 3 can determine whether or not the subject is a living body by using a single thermal image IMG_T in which the subject's face is properly captured in the attention area TA. The authentication device 3 can determine whether or not the subject is a living body with higher accuracy.

[0085] (3-5-4) Fourth Modification In the above description, the authentication device 3 that authenticates the subject appearing in the person image IMG_P determines whether the subject is a living body using the thermal image IMG_T. However, an arbitrary spoofing determination device that does not authenticate the subject appearing in the person image IMG_P may use the thermal image IMG_T to determine whether the subject appearing in the thermal image IMG_T is a living body, similar to the above-described authentication device 3. In other words, any spoofing determination device may determine whether a living body is appearing in the thermal image IMG_T. Even in this case, the arbitrary spoofing determination device can determine whether the subject is a living body with relatively high accuracy, similar to the above-described authentication device 3.

[0086] For example, in a facility where subjects whose body surface temperature is within the normal range are permitted to stay, but subjects whose body surface temperature is outside the normal range are prohibited from staying (e.g., subjects whose body surface temperature is outside the normal range are required to leave), a thermal camera 2 may be installed to measure the body surface temperature of subjects staying in the facility. Examples of such facilities include at least one of an office building, a public facility, a restaurant, and a hospital. In this case, the facility may be equipped with a stay management device that uses a thermal image IMG_T generated by the thermal camera 2 capturing an image of a subject attempting to enter the facility to determine whether the body surface temperature of the subject staying in the facility is normal and to request that a subject whose body surface temperature is outside the normal range leave the facility. Similar to the authentication device 3 described above, this stay management device may determine whether the subject captured in the thermal image IMG_T is a living body.

[0087] (4) Fourth embodiment Next, a fourth embodiment of the authentication device, engine generation device, authentication method, engine generation method, and recording medium will be described. Hereinafter, the authentication device, engine generation device, authentication method, engine generation method, and recording medium in the fourth embodiment will be described using an authentication system SYS4 to which the authentication device, engine generation device, authentication method, engine generation method, and recording medium in the fourth embodiment are applied.

[0088] (4-1) Configuration of the authentication system SYS4 First, the configuration of the authentication system SYS4 in the fourth embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram showing the configuration of the authentication system SYS4 in the fourth embodiment. Note that components that have already been described are given the same reference numerals, and detailed description thereof will be omitted.

[0089] 15, the authentication system SYS4 in the fourth embodiment differs from the authentication system SYS3 in the third embodiment in that the authentication system SYS4 further includes an engine generation device 4. Other features of the authentication system SYS4 may be the same as other features of the authentication system SYS3.

[0090] The engine generation device 4 is capable of performing an engine generation operation for generating a determination engine ENG for determining whether a subject is a living organism using a thermal image IMG_T. The determination engine ENG may be any engine as long as it is capable of determining whether a subject is a living organism using a thermal image IMG_T. For example, the determination engine ENG may be an engine that outputs a determination result as to whether a subject is a living organism based on at least a portion of the thermal image IMG_T (e.g., an image portion included in a region of interest TA of the thermal image IMG_T). For example, the determination engine ENG may be an engine that outputs a determination result as to whether a subject is a living organism when at least a portion of the thermal image IMG_T (e.g., an image portion included in a region of interest TA of the thermal image IMG_T) is input. For example, the determination engine ENG may be an engine that outputs a determination result as to whether a subject is a living organism based on feature quantities of at least a portion of the thermal image IMG_T (e.g., an image portion included in a region of interest TA of the thermal image IMG_T). For example, the judgment engine ENG may be an engine that outputs a judgment result as to whether or not the subject is a living body when the features of at least a portion of the thermal image IMG_T (for example, an image portion of the thermal image IMG_T included in the area of ​​interest TA) are input.

[0091] In the fourth embodiment, an example will be described in which the engine generation device 4 generates a determination engine ENG by performing machine learning using an image showing a body surface temperature distribution of a person similar to the thermal image IMG_T. In this case, the determination engine ENG is an engine that can be generated by machine learning (a so-called trainable learning model). An example of an engine that can be generated by machine learning is an engine that uses a neural network (for example, a learning model).

[0092] The engine generation device 4 may transmit the generated determination engine ENG to the authentication device 3 via the communication network NW. In this case, the authentication device 3 may determine whether the subject is a living body by using the thermal image IMG_T and the determination engine ENG.

[0093] (4-2) Configuration of the engine generator 4 Next, the configuration of the engine generation device 4 in the fourth embodiment will be described with reference to Fig. 16. Fig. 16 is a block diagram showing the configuration of the engine generation device 4 in the fourth embodiment.

[0094] 16, the engine generation device 4 includes a calculation device 41, a storage device 42, and a communication device 43. The engine generation device 4 may further include an input device 44 and an output device 45. However, the authentication device 3 may not include at least one of the input device 44 and the output device 45. The calculation device 41, the storage device 42, the communication device 43, the input device 44, and the output device 45 may be connected via a data bus 46.

[0095] The arithmetic device 41 includes, for example, at least one of a CPU, a GPU, and an FPGA. The arithmetic device 41 loads a computer program. For example, the arithmetic device 41 may load a computer program stored in the storage device 42. For example, the arithmetic device 41 may load a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the engine generation device 4. The arithmetic device 41 may acquire (i.e., download or load) the computer program from a device (not shown) located outside the engine generation device 4 via the communication device 43 (or another communication device). The arithmetic device 41 executes the loaded computer program. As a result, logical functional blocks for executing operations to be performed by the engine generation device 4 (e.g., the above-mentioned engine generation operations) are realized within the arithmetic device 41. In other words, the arithmetic device 41 can function as a controller for realizing logical functional blocks for executing operations (in other words, processing) to be performed by the engine generation device 4.

[0096] Fig. 16 shows an example of logical functional blocks realized in the arithmetic device 41 to execute the engine generation operation. As shown in Fig. 16, an image extraction unit 411, an image generation unit 412, and an engine generation unit 413 are realized in the arithmetic device 41. Note that the operations of the image extraction unit 411, the image generation unit 412, and the engine generation unit 413 will be described in detail later with reference to Fig. 17 etc., and therefore will not be described in detail here.

[0097] The storage device 42 can store desired data. For example, the storage device 42 may temporarily store a computer program executed by the arithmetic device 41. The storage device 42 may temporarily store data that the arithmetic device 41 temporarily uses when the arithmetic device 41 is executing a computer program. The storage device 42 may store data that the engine generation device 4 stores long-term. The storage device 42 may include at least one of a RAM, a ROM, a hard disk device, a magneto-optical disk device, an SSD, and a disk array device. In other words, the storage device 42 may include a non-temporary recording medium.

[0098] The communication device 43 can communicate with the visible camera 1, the thermal camera 2, and the authentication device 3 via the communication network NW. In the fourth embodiment, the communication device 43 transmits the generated determination engine ENG to the authentication device 3 via the communication network NW.

[0099] The input device 44 is a device that accepts information input to the engine generation device 4 from outside the engine generation device 4. For example, the input device 44 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the engine generation device 4. For example, the input device 44 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the engine generation device 4.

[0100] The output device 45 is a device that outputs information to the outside of the engine generation device 4. For example, the output device 45 may output information as an image. That is, the output device 45 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 45 may output information as sound. That is, the output device 45 may include an audio device (a so-called speaker) that can output sound. For example, the output device 45 may output information on paper. That is, the output device 45 may include a printing device (a so-called printer) that can print desired information on paper.

[0101] (4-3) Engine Generation Operation Performed by the Engine Generation Device 4 Next, the flow of the engine generation operation performed by the engine generation device 4 will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the flow of the engine generation operation performed by the engine generation device 4.

[0102] 17, the image extraction unit 411 extracts at least one extracted image IMG_E from the training data set 420 (step S41). The training data set 420 may be stored in, for example, the storage device 42 (see FIG. 16). Alternatively, the image extraction unit 411 may acquire the training data set 420 from a device external to the engine generation device 4 using the communication device 43.

[0103] FIG. 18 shows an example of the data structure of the training dataset 420. As shown in FIG. 18, the training dataset 420 includes a plurality of unit data 421. Each unit data 421 includes a sample image IMG_S, area-of-interest information 422, and a correct answer label 423. The sample image IMG_S is an image showing the body surface temperature distribution of a sample person. For example, an image generated by capturing an image of the sample person using the thermal camera 2 or a thermal camera other than the thermal camera 2 may be used as the sample image IMG_S. For example, an image simulating an image generated by capturing an image of the sample person using the thermal camera 2 or a thermal camera other than the thermal camera 2 may be used as the sample image IMG_S. In the fourth embodiment, an area of ​​interest TA is set in advance in the sample image IMG_S. The area of ​​interest TA set in the sample image IMG_S is an area that should be noted in order to determine whether the sample person is a living body. For example, the attention area TA may be an area in which a part of the sample person (for example, the face described above) that should be noted in order to determine whether the sample person is a living body is captured. Information about the attention area TA that is preset in the sample image IMG_S is included in the unit data 421 as attention area information 422. The correct answer label 423 indicates whether the sample person captured in the sample image IMG_S is a living body or not.

[0104] The learning data set 420 may include a plurality of unit data 421 each including a plurality of sample images IMG_S showing the body surface temperature distributions of a plurality of different sample persons. The learning data set 420 may also include a plurality of unit data 421 each including a plurality of sample images IMG_S showing the body surface temperature distributions of the same sample person.

[0105] The image extraction unit 411 may randomly extract at least one sample image IMG_S as an extracted image IMG_E from the training data set 420. In this case, the image extraction unit 411 may extract all of the multiple sample images IMG_S included in the training data set 420 as extracted images IMG_E. Alternatively, the image extraction unit 411 may extract some of the multiple sample images IMG_S included in the training data set 420 as extracted images IMG_E, while not extracting other parts of the multiple sample images IMG_S included in the training data set 420 as extracted images IMG_E.

[0106] Alternatively, the image extraction unit 411 may extract, as the extracted image IMG_E, at least one sample image IMG_S that satisfies predetermined extraction conditions from the training dataset 420. The extraction conditions may include imaging environment conditions determined based on the imaging environment in which at least one of the visible light camera 1 and the thermal camera 2 captures an image of the subject. In other words, the extraction conditions may include imaging environment conditions that reflect the actual imaging environment in which at least one of the visible light camera 1 and the thermal camera 2 captures an image of the subject. In this case, the image extraction unit 411 may extract, as the extracted image IMG_E, at least one sample image IMG_S that satisfies the imaging environment conditions from the training dataset 420. For example, the image extraction unit 411 may extract, as the extracted image IMG_E, a sample image IMG_S that has characteristics similar to those of a thermal image IMG_T generated by the thermal camera 2 in a predetermined imaging environment indicated by the imaging environment conditions from the training dataset 420. The image extraction unit 411 may extract a sample image IMG_S and an extracted image IMG_E from the learning dataset 420, which have characteristics similar to those of a thermal image IMG_T generated by a thermal camera 2 that captures an image of a subject under the imaging environment indicated by the imaging environment conditions.

[0107] The imaging environment may include the positional relationship between the visible camera 1 and the thermal camera 2. The imaging environment may include the positional relationship between the visible camera 1 and the subject. In particular, the imaging environment may include the positional relationship between the visible camera 1 and the subject at the time when the visible camera 1 captures an image of the subject. The positional relationship between the visible camera 1 and the subject may include the distance between the visible camera 1 and the subject. The positional relationship between the visible camera 1 and the subject may include the relationship between the direction in which the visible camera 1 is facing (e.g., the direction in which the optical axis of the optical system such as a lens provided in the visible camera 1 extends) and the direction in which the subject is facing (e.g., the direction in which the subject's face is facing, extending in front of the subject). The imaging environment may include the positional relationship between the thermal camera 2 and the subject. In particular, the imaging environment may include the positional relationship between the thermal camera 2 and the subject at the time when the thermal camera 2 captures an image of the subject. The positional relationship between the thermal camera 2 and the subject may include the distance between the thermal camera 2 and the subject. The positional relationship between the thermal camera 2 and the subject may include the relationship between the direction in which the thermal camera 2 is facing (for example, the direction in which the optical axis of the optical system, such as a lens, provided in the thermal camera 2 extends) and the direction in which the subject is facing. The imaging environment may include the optical characteristics of the visible light camera 1 (for example, the optical characteristics of the optical system, such as a lens, provided in the visible light camera 1). The imaging environment may include the optical characteristics of the thermal camera 2 (for example, the optical characteristics of the optical system, such as a lens, provided in the thermal camera 2).

[0108] As described above, the visible camera 1 and the thermal camera 2 may capture an image of a subject moving toward the visible camera 1 and the thermal camera 2, or may capture an image of a subject standing still in front of the visible camera 1 and the thermal camera 2. In this case, the imaging environment when the visible camera 1 and the thermal camera 2 capture an image of a moving subject is generally different from the imaging environment when the visible camera 1 and the thermal camera 2 capture an image of a stationary subject. For this reason, at least one of the conditions that the visible camera 1 and the thermal camera 2 capture an image of a moving subject and the conditions that the visible camera 1 and the thermal camera 2 capture an image of a stationary subject may be used as the imaging environment condition.

[0109] The state of the subject captured in the thermal image IMG_T changes depending on the imaging environment. For example, the state of the subject captured in a thermal image IMG_T generated by capturing an image of a moving subject generally differs from the state of the subject captured in a thermal image IMG_T generated by capturing an image of a stationary subject. Therefore, the operation of extracting at least one extraction image IMG_E that satisfies the imaging environment conditions may be considered equivalent to the operation of extracting, as the extraction image IMG_E, a sample image IMG_S in which a sample person is captured in a state similar to the state of the subject captured in the thermal image IMG_T generated under a predetermined imaging environment indicated by the imaging environment conditions.

[0110] As an example, when the visible camera 1 and the thermal camera 2 capture an image of a moving subject, the visible camera 1 and the thermal camera 2 are relatively likely to capture the image of the subject from an oblique direction, whereas when the visible camera 1 and the thermal camera 2 capture an image of a stationary subject, the visible camera 1 and the thermal camera 2 are relatively likely to capture the image of the subject from a frontal direction. In this case, the condition that the visible camera 1 and the thermal camera 2 capture the image of the subject from a frontal direction may be used as the imaging environment condition under which the visible camera 1 and the thermal camera 2 capture the image of the stationary subject. Similarly, the condition that the visible camera 1 and the thermal camera 2 capture the image of the subject from an oblique direction may be used as the imaging environment condition under which the visible camera 1 and the thermal camera 2 capture the image of a moving subject. When the imaging environment condition under which the visible camera 1 and the thermal camera 2 capture the image of the subject from a frontal direction is used, the image extraction unit 411 may extract, as the extracted image IMG_E, a sample image IMG_S in which a sample person facing forward is captured. When the imaging environment conditions are such that the visible camera 1 and the thermal camera 2 capture images of the subject from an oblique direction, the image extraction unit 411 may extract a sample image IMG_S that captures a sample person looking obliquely as the extracted image IMG_E.

[0111] 17 again, the image generation unit 412 then uses the extracted image IMG_E extracted in step S41 to generate a training image IMG_L that is actually used in machine learning (step S42). In particular, the image generation unit 412 generates a plurality of training images IMG_L (step S42). Specifically, the image generation unit 412 changes the positional relationship between the attention area TA set in the extracted image IMG_E and the face of the sample person (i.e., the attention area) that appears in the extracted image IMG_E, thereby generating a training image IMG_L that is the extracted image IMG_E in which the positional relationship between the attention area TA and the face of the sample person has been changed.

[0112] The image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person in one extracted image IMG_E in a plurality of different change modes. In this case, the image generation unit 412 can generate a plurality of training images IMG_L from one extracted image IMG_E, each of which has a different change mode for the positional relationship between the attention area TA and the face of the sample person. As a result, the image generation unit 412 can further increase the number of training images IMG_L used in machine learning. This is a significant advantage for machine learning, where the more data used as samples, the more efficient the learning becomes.

[0113] The image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person based on the imaging environment in which at least one of the above-mentioned visible camera 1 and thermal camera 2 images the subject. Specifically, as described above, the state of the subject appearing in the thermal image IMG_T changes depending on the imaging environment. In this case, the image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person so as to generate a learning image IMG_L in which the sample person appears in a state similar to the state of the subject appearing in the thermal image IMG_T generated in an actual imaging environment in which at least one of the above-mentioned visible camera 1 and thermal camera 2 images the subject.

[0114] For example, as described above, the visible camera 1 and the thermal camera 2 may capture an image of a subject moving toward the visible camera 1 and the thermal camera 2, respectively, or may capture an image of a subject standing still in front of the visible camera 1 and the thermal camera 2. In this case, the image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person so as to generate a learning image IMG_L in which the sample person appears in a state similar to that of the subject appearing in the thermal image IMG_T generated by capturing an image of the moving subject with the thermal camera 2. The image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person so as to generate a learning image IMG_L in which the sample person appears in a state similar to that of the subject appearing in the thermal image IMG_T generated by capturing an image of the stationary subject with the thermal camera 2.

[0115] As an example, a thermal image IMG_T generated by capturing an image of a moving subject is more likely to have a larger deviation between the center of the attention area TA and the center of the subject's face than a thermal image IMG_T generated by capturing an image of a stationary subject. Therefore, when the thermal camera 2 captures an image of a moving subject, the image generation unit 412 may change the positional relationship between the attention area TA and the sample person's face so as to generate a training image IMG_L in which the deviation between the center of the attention area TA and the center of the sample person's face is relatively large. On the other hand, when the thermal camera 2 captures an image of a stationary subject, the image generation unit 412 may change the positional relationship between the attention area TA and the sample person's face so as to generate a training image IMG_L in which the deviation between the center of the attention area TA and the center of the sample person's face is relatively small.

[0116] As another example, in a thermal image IMG_T generated by capturing an image of a moving subject, the subject's face is more likely to deviate in more directions from the center of the attention area TA than in a thermal image IMG_T generated by capturing an image of a stationary subject. Therefore, when the thermal camera 2 captures an image of a moving subject, the image generation unit 412 may change the positional relationship between the attention area TA and the sample person's face so as to generate multiple training images IMG_L in which the sample person's face deviates in relatively more directions from the center of the attention area TA. Specifically, the image generation unit 412 may generate multiple training images IMG_L in which the sample person's face deviates in four different directions (e.g., upward, downward, rightward, and leftward) from the center of the attention area TA. On the other hand, when the thermal camera 2 captures an image of a stationary subject, the image generation unit 412 may change the positional relationship between the attention area TA and the sample person's face so as to generate multiple training images IMG_L in which the sample person's face deviates relatively less from the center of the attention area TA. Specifically, the image generating unit 412 may generate a plurality of learning images IMG_L in which the face of the sample person is shifted in only one direction or two directions (for example, upward and downward) relative to the center of the attention area TA.

[0117] As another example, when the visible camera 1 and the thermal camera 2 capture an image of a moving subject, the visible camera 1 and the thermal camera 2 may capture the image of the subject from a position relatively far from the subject, whereas when the visible camera 1 and the thermal camera 2 capture an image of a stationary subject, the visible camera 1 and the thermal camera 2 may capture the image of the subject from a position relatively close to the subject. The greater the distance between the visible camera 1 and the thermal camera 2 and the subject, the smaller the subject's face appears in the thermal image IMG_T. Therefore, when the thermal camera 2 captures an image of a moving subject, the image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person so as to generate a training image IMG_L in which the sample person's face is relatively small. On the other hand, when the thermal camera 2 captures an image of a stationary subject, the image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person so as to generate a training image IMG_L in which the sample person's face is relatively large.

[0118] The image generation unit 412 may change the positional relationship between the attention region TA and the face of the sample person by changing the characteristics of the attention region TA within the extraction image IMG_E. The characteristics of the attention region TA may include the position of the attention region TA. In this case, as shown in FIG. 19(a), the image generation unit 412 may change the positional relationship between the attention region TA and the face of the sample person by changing the position of the attention region TA within the extraction image IMG_E. In other words, the image generation unit 412 may change the positional relationship between the attention region TA and the face of the sample person by moving the attention region TA within the extraction image IMG_E. Furthermore, the characteristics of the attention region TA may include the size of the attention region TA. In this case, as shown in FIG. 19(b), the image generation unit 412 may change the positional relationship between the attention region TA and the face of the sample person by changing the size of the attention region TA within the extraction image IMG_E. In other words, the image generation unit 412 may change the positional relationship between the attention region TA and the face of the sample person by enlarging or reducing the attention region TA within the extraction image IMG_E.

[0119] The image generation unit 412 may change the characteristics of the extraction image IMG_E in which the attention area TA is set, thereby changing the positional relationship between the attention area TA and the face of the sample person. The characteristics of the extraction image IMG_E may include the position of the extraction image IMG_E (e.g., the position relative to the attention area TA). In this case, as shown in FIG. 20(a), the image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person by changing the position of the extraction image IMG_E relative to the attention area TA. In other words, the image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person by moving (e.g., translating) the extraction image IMG_E relative to the attention area TA. Furthermore, the characteristics of the extraction image IMG_E may include the size of the extraction image IMG_E. In this case, as shown in FIG. 20(b), the image generation unit 412 may change the positional relationship between the attention area TA and the face of the sample person by changing the size of the extraction image IMG_E. That is, the image generating section 412 may change the positional relationship between the attention area TA and the face of the sample person by enlarging or reducing the extracted image IMG_E.

[0120] 17 again, the engine generation unit 413 then generates a determination engine ENG using the plurality of training images IMG_L generated in step S42 (step S43). That is, the engine generation unit 413 generates the determination engine ENG by performing machine learning using the plurality of training images IMG_L generated in step S42 (step S43). Specifically, the engine generation unit 413 inputs each of the plurality of training images IMG_L generated in step S42 to the determination engine ENG. As a result, the determination engine ENG outputs a determination result as to whether or not a sample person appearing in each training image IMG_L is a living body. Thereafter, the engine generation unit 413 updates the parameters of the determination engine ENG using a loss function based on the error between the determination result of the determination engine ENG and the correct label 423 corresponding to each training image IMG_L. Typically, the engine generation unit 413 updates the parameters of the determination engine ENG so that the error between the determination result of the determination engine ENG and the correct label 423 corresponding to each training image IMG_L becomes small (preferably, minimized). Alternatively, since the determination engine ENG performs so-called binary classification, the engine generation unit 413 may update the parameters of the determination engine ENG using an index value based on the confusion matrix (for example, at least one of accuracy, recall, specificity, precision, and F-measure). As a result, the determination engine ENG is generated.

[0121] (4-3) Technical Effects As described above, in the fourth embodiment, the engine generation device 4 generates a training image IMG_L from the extracted image IMG_E based on the imaging environment in which the thermal camera 2 captures the subject. In this case, the training image IMG_L reflects information about the imaging environment in which the thermal camera 2 captures the subject. Therefore, the engine generation device 4 can generate a determination engine ENG that reflects information about the imaging environment by performing machine learning using the training image IMG_L that reflects information about the imaging environment. For example, the engine generation device 4 can generate a determination engine ENG for determining whether the subject is a living body using a thermal image IMG_T generated by the thermal camera 2 capturing the subject in a specific imaging environment by performing machine learning using the training image IMG_L that reflects information about a specific imaging environment. As a result, by using the determination engine ENG that reflects information about the specific imaging environment, the authentication device 3 can determine whether the subject is a living body with high accuracy from the thermal image IMG_T generated by the thermal camera 2 capturing the subject in a specific imaging environment, compared to using a determination engine ENG that does not reflect information about the specific imaging environment. That is, the engine generation device 4 can generate a determination engine ENG that can determine with high accuracy whether or not a subject is a living body.

[0122] (4-4) Modification The engine generation device 4 may generate a plurality of different determination engines ENG, and the authentication device 3 may select one of the plurality of determination engines ENG and determine whether or not the subject is a living body using the selected one determination engine ENG. In this case, the authentication device 3 may change the determination engine ENG used to determine whether or not the subject is a living body during an authentication period in which the authentication operation is being performed.

[0123] As an example, as described above, at least one of the amount of deviation between the center of the attention area TA in the thermal image IMG_T and the center of the subject's face (hereinafter simply referred to as the "amount of deviation between the attention area TA and the face") and the direction of deviation between the center of the attention area TA in the thermal image IMG_T and the center of the subject's face (hereinafter simply referred to as the "direction of deviation between the attention area TA and the face") may change depending on the imaging environment. In this case, the engine generation device 4 may generate multiple types of training images IMG_L that differ in at least one of the amount of deviation between the attention area TA and the face and the direction of deviation between the attention area TA and the face, and generate multiple determination engines ENG using the multiple types of training images IMG_L. For example, the engine generation device 4 may (i) change the positional relationship between the attention region TA and the face in a first change manner to generate a first training image IMG_L in which the amount of deviation between the attention region TA and the face falls within a first range and the direction of deviation between the attention region TA and the face is a first direction, and (ii) change the positional relationship between the attention region TA and the face in a second change manner different from the first change manner to generate a second training image IMG_L in which the amount of deviation between the attention region TA and the face falls within a second range different from the first range and / or the direction of deviation between the attention region TA and the face is a second direction different from the first direction. Thereafter, the engine generation device 4 may generate a first determination engine ENG using the first training image IMG_L, and generate a second determination engine ENG using the second training image IMG_L.

[0124] 21, the engine generation device 4 may generate a training image IMG_L#1 in which the face is not shifted relative to the attention region TA, a training image IMG_L#2 in which the face is shifted upward relative to the attention region TA, a training image IMG_L#3 in which the face is shifted downward relative to the attention region TA, a training image IMG_L#4 in which the face is shifted left relative to the attention region TA, and a training image IMG_L#5 in which the face is shifted right relative to the attention region TA. Thereafter, the engine generation device 4 may generate a determination engine ENG#1 using the training image IMG_L#1, a determination engine ENG#2 using the training image IMG_L#2, a determination engine ENG#3 using the training image IMG_L#3, a determination engine ENG#4 using the training image IMG_L#4, and a determination engine ENG#5 using the training image IMG_L#5. Compared to the determination engines ENG#2 to ENG#5, the determination engine ENG#1 can more accurately determine whether a subject is a living organism using a thermal image IMG_T in which the face is not shifted relative to the attention area TA. Compared to the determination engines ENG#1 and ENG#3 to ENG#5, the determination engine ENG#2 can more accurately determine whether a subject is a living organism using a thermal image IMG_T in which the face is shifted upward relative to the attention area TA. Compared to the determination engines ENG#1 to ENG#2 and ENG#4 to ENG#5, the determination engine ENG#3 can more accurately determine whether a subject is a living organism using a thermal image IMG_T in which the face is shifted downward relative to the attention area TA. Compared to the determination engines ENG#1 to ENG#3 and ENG#5, the determination engine ENG#4 can more accurately determine whether a subject is a living organism using a thermal image IMG_T in which the face is shifted to the left relative to the attention area TA. Compared to the determination engines ENG#1 to ENG#4, the determination engine ENG#5 is able to more accurately determine whether or not a subject is a living body using a thermal image IMG_T in which the face is shifted to the right relative to the area of ​​interest TA.

[0125] When such a determination engine ENG is generated, the authentication device 3 may estimate at least one of the amount of deviation between the attention area TA and the face and the direction of deviation between the attention area TA and the face based on the imaging environment during the authentication period, and select one determination engine ENG from the plurality of determination engines ENG that corresponds to at least one of the estimated amount of deviation and the direction of deviation. That is, the authentication device 3 may select a determination engine ENG generated using a training image IMG_L that corresponds to at least one of the estimated amount of deviation and the direction of deviation. Then, the authentication device 3 may use the selected determination engine ENG to determine whether the subject is a living body. As a result, the authentication device 3 can determine whether the subject is a living body with higher accuracy than when the determination engine ENG to be used by the authentication device 3 is not selectable.

[0126] The imaging environment used to estimate at least one of the amount of deviation between the attention area TA and the face and the direction of deviation between the attention area TA and the face may include, for example, the positional relationship (typically, the distance between the visible camera 1 and the subject) between the visible camera 1 and the subject at the time when the visible camera 1 images the subject (i.e., at the above-mentioned authentication time ta). The imaging environment used to estimate at least one of the amount of deviation between the attention area TA and the face and the direction of deviation between the attention area TA and the face may include the positional relationship between the visible camera 1 and the thermal camera 2. In this case, the authentication device 3 may estimate at least one of the amount of deviation between the attention area TA and the face and the direction of deviation between the attention area TA and the face using information (e.g., at least one of a table, an arithmetic expression, a function, a graph, etc.) that defines the relationship between the positional relationship between the visible camera 1 and the subject and the positional relationship between the visible camera 1 and the thermal camera 2 and at least one of the amount of deviation between the attention area TA and the face and the direction of deviation between the attention area TA and the face. In addition, when the positional relationship between the visible camera 1 and the subject is used to estimate at least one of the amount of deviation between the attention area TA and the face and the direction of deviation between the attention area TA and the face, the authentication system SYS4 may be equipped with a measurement device for measuring the positional relationship between the visible camera 1 and the subject (typically, the distance between the visible camera 1 and the subject).

[0127] Note that the engine generating device 2000 in the second embodiment may also employ the same components as those in the above-described modified example.

[0128] (5) Supplementary notes The following additional notes are provided regarding the above-described embodiment. [Appendix 1] an authentication means for authenticating a target person using a human image generated by capturing an image of the target person with a visible light camera at a first time; a determination means for determining whether the subject is a living body or not, using a plurality of thermal images generated by the thermal camera capturing images of the subject at a second time closest to the first time and a third time before and after the second time among a plurality of times at which the thermal camera captured images of the subject; An authentication device comprising: [Appendix 2] The determination means, based on the person image, specifies an area of ​​interest to be noted in at least one thermal image of the plurality of thermal images in order to determine whether the subject is a living body, adjusts the position of the area of ​​interest in the at least one thermal image based on the at least one thermal image, and determines whether the subject is a living body based on the temperature distribution in the area of ​​interest whose position has been adjusted. 10. The authentication device of claim 1. [Appendix 3] The determination means identifies an area of ​​interest to be noted in each of the plurality of thermal images based on the person image in order to determine whether the subject is a living body, selects at least one thermal image from the plurality of thermal images in which a part of interest of the subject to be noted in order to determine whether the subject is a living body is captured in the area of ​​interest based on the plurality of thermal images, and determines whether the subject is a living body based on the at least one selected thermal image. 3. An authentication device according to claim 1 or 2. [Appendix 4] the determination means determines whether the subject is a living body using a determination engine capable of determining whether the subject is a living body from the plurality of thermal images; The determination engine is generated by a learning operation including: a first operation of extracting at least one sample image as an extracted image from a learning dataset including a plurality of sample images each showing a body surface temperature distribution of the sample person and having a region of interest set therein to be focused on in order to determine whether the sample person is a living body; a second operation of generating a learning image by changing the positional relationship between the region of interest set in the extracted image and a part of interest of the sample person to be focused on in order to determine whether the sample person is a living body, based on an imaging environment in which the visible light camera and the thermal camera image the subject; and a third operation of performing machine learning using the learning image. 4. An authentication device according to any one of claims 1 to 3. [Appendix 5] The second operation changes at least one of the position and size of the attention area in the extracted image and the position and size of the extracted image, thereby changing the positional relationship between the attention area and the attention site. 5. The authentication device of claim 4. [Appendix 6] The determination means selects one determination engine from a plurality of determination engines generated by a plurality of second actions each having a different change mode of the positional relationship based on the imaging environment, and determines whether the subject is a living body using the selected one determination engine. 6. An authentication device according to claim 4 or 5. [Appendix 7] The imaging environment includes a positional relationship between the subject and the visible light camera at the first time, and a positional relationship between the visible light camera and the thermal camera. 10. The authentication device of claim 6. [Appendix 8] An engine generation device that generates a determination engine for determining whether a subject is a living body using a thermal image generated by capturing an image of the subject with a thermal camera, an extraction means for extracting at least one sample image as an extracted image from a learning dataset including a plurality of sample images each showing a body surface temperature distribution of a sample person and each having a region of interest set thereon to be focused on in order to determine whether the sample person is a living body; an image generation means for generating a learning image by changing the positional relationship between the region of interest set in the extracted image and a part of interest of the sample person that should be focused on in order to determine whether the sample person is a living body, based on the imaging environment in which the thermal camera images the subject; and an engine generation means for generating the determination engine by performing machine learning using the training image; An engine generating device comprising: [Appendix 9] The image generating means changes the positional relationship between the region of interest and the site of interest by changing at least one of the position and size of the region of interest within the extracted image and the position and size of the extracted image. 9. The engine generator of claim 8. [Appendix 10] the image generation means generates a first learning image by changing a positional relationship between the attention region and the attention portion set in the extracted image in a first change manner, and generates a second learning image by changing a positional relationship between the attention region and the attention portion set in the extracted image in a second change manner different from the first change manner; The engine generation means generates a first determination engine by performing machine learning using the first training image, and generates a second determination engine by performing machine learning using the second training image. 10. The engine generator device of claim 8 or 9. [Appendix 11] authenticating the subject using a person image generated by capturing an image of the subject with a visible light camera at a first time; determining whether the subject is a living body or not using a plurality of thermal images generated by the thermal camera capturing images of the subject at a second time closest to the first time and a third time before and after the second time among a plurality of times at which the thermal camera captured images of the subject; Authentication methods, including: [Appendix 12] 1. An engine generation method for generating a determination engine for determining whether a subject is a living body using a thermal image generated by capturing an image of the subject with a thermal camera, comprising: Extracting at least one sample image as an extracted image from a learning dataset including a plurality of sample images each showing a body surface temperature distribution of the sample person and each having a region of interest set therein for determining whether the sample person is a living body; generating a learning image by changing the positional relationship between the region of interest set in the extracted image and a part of interest of the sample person that should be noted in order to determine whether the sample person is a living body, based on an imaging environment in which the thermal camera images the subject; generating the determination engine by performing machine learning using the training image; An engine generation method including: [Appendix 13] authenticating the subject using a person image generated by capturing an image of the subject with a visible light camera at a first time; determining whether the subject is a living body or not using a plurality of thermal images generated by the thermal camera capturing images of the subject at a second time closest to the first time and a third time before and after the second time among a plurality of times at which the thermal camera captured images of the subject; A recording medium on which a computer program is recorded that causes a computer to execute an authentication method including the steps of: [Appendix 14] 1. An engine generation method for generating a determination engine for determining whether a subject is a living body using a thermal image generated by capturing an image of the subject with a thermal camera, comprising: Extracting at least one sample image as an extracted image from a learning dataset including a plurality of sample images each showing a body surface temperature distribution of the sample person and each having a region of interest set therein for determining whether the sample person is a living body; generating a learning image by changing the positional relationship between the region of interest set in the extracted image and a part of interest of the sample person that should be noted in order to determine whether the sample person is a living body, based on an imaging environment in which the thermal camera images the subject; generating the determination engine by performing machine learning using the training image; A recording medium on which a computer program is recorded that causes a computer to execute an engine generation method including the above.

[0129] At least some of the constituent elements of each of the above-described embodiments can be appropriately combined with at least some of the other constituent elements of each of the above-described embodiments. Some of the constituent elements of each of the above-described embodiments may not be used. Furthermore, to the extent permitted by law, the disclosures of all documents (e.g., published patent applications) cited in this disclosure are incorporated by reference as part of the description of this disclosure.

[0130] This disclosure may be modified as appropriate within the scope of the claims and the technical idea that can be read from the entire specification. Authentication devices, engine generation devices, authentication methods, engine generation methods, computer programs, and recording media that incorporate such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0131] SYS3, SYS4 authentication systems 1. Visible light camera 2. Thermal camera 3 Authentication Device 31 Arithmetic unit 311 Authentication Department 312 Biometrics Unit 313 Entrance / Exit Control Department 32 Storage device 321 Registered Person DB 322 Registered Body Surface Temperature Distribution DB 4 Engine Generator 41 Arithmetic device 411 Image Extraction Unit 412 Image Generation Unit 413 Engine Generation Unit 42 Storage device 420 training dataset 421 Unit Data 422 Area of ​​Interest Information 423 Correct Label 1000 Authentication Device 1001 Authentication Department 1002 Judgment section 2000 Engine Generator 2001 Extraction part 2002 Image Generation Department 2003 Engine Generation Division IMG_P People image IMG_T Thermal image IMG_S Sample image IMG_E Extracted image IMG_L training image FA Face Area TA focus area ENG Judgment Engine

Claims

1. an authentication means for acquiring a person image generated by capturing an image of a target person with a visible camera, and detecting a face area including at least a part of the target person's face in the person image, and authenticating the target person; a time when the person image used for authentication by the authentication means is captured is defined as a first time; a second time is set to a time closest to the first time among a plurality of times at which the thermal camera captured an image of the subject; When a time consecutive to the second time among the plurality of times and a time before or after the second time is defined as a third time, a determination means for obtaining a plurality of thermal images generated by the thermal camera capturing images of the subject at the second time and the third time, identifying a region of interest that corresponds to the face region in the plurality of thermal images, and determining whether the subject is a living body based on a temperature distribution within the region of interest; An authentication device comprising:

2. The determination means adjusts the position of the region of interest within the plurality of thermal images, and determines whether the subject is a living body based on a temperature distribution within the region of interest whose position has been adjusted. The authentication device according to claim 1 .

3. the determination means determines whether the subject is a living body using a determination engine capable of determining whether the subject is a living body from the plurality of thermal images; The determination engine is generated by a learning operation including: a first operation of extracting at least one sample image as an extracted image from a learning dataset including a plurality of sample images each showing a body surface temperature distribution of the sample person and having a region of interest set therein to be focused on in order to determine whether the sample person is a living body; a second operation of generating a learning image by changing the positional relationship between the region of interest set in the extracted image and the face of the sample person to be focused on in order to determine whether the sample person is a living body, based on an imaging environment in which the visible light camera and the thermal camera image the subject; and a third operation of performing machine learning using the learning image.

3. The authentication device according to claim 1 or 2.

4. The second operation changes at least one of the position and size of the attention area in the extracted image and the position and size of the extracted image, thereby changing the positional relationship between the attention area and the face. The authentication device according to claim 3 .

5. The determination means selects one determination engine from a plurality of determination engines generated by a plurality of second actions each having a different change mode of the positional relationship based on the imaging environment, and determines whether the subject is a living body using the selected one determination engine. The authentication device according to claim 3 .

6. The imaging environment includes a positional relationship between the subject and the visible light camera at the first time, and a positional relationship between the visible light camera and the thermal camera. The authentication device according to claim 5 .

7. acquiring a person image generated by capturing an image of a target person with a visible camera, and detecting a face area including at least a part of the target person's face in the person image, thereby authenticating the target person; a time when the person image used for authenticating the subject is captured is defined as a first time; a second time is set to a time closest to the first time among a plurality of times at which the thermal camera captured an image of the subject; When a time consecutive to the second time among the plurality of times and a time before or after the second time is defined as a third time, acquiring a plurality of thermal images generated by the thermal camera capturing images of the subject at the second time and the third time, identifying a region of interest in the plurality of thermal images that corresponds to the face region, and determining whether the subject is a living body based on a temperature distribution within the region of interest; Authentication methods, including:

8. an authentication means for acquiring a person image generated by capturing an image of a target person with a visible camera, and detecting a face area including at least a part of the target person's face in the person image, and authenticating the target person; a time when the person image used for authentication by the authentication means is captured is defined as a first time; a second time is set to a time closest to the first time among a plurality of times at which the thermal camera captured an image of the subject; When a time consecutive to the second time among the plurality of times and a time before or after the second time is defined as a third time, a determination means for obtaining a plurality of thermal images generated by the thermal camera capturing images of the subject at the second time and the third time, identifying a region of interest that corresponds to the face region in the plurality of thermal images, and determining whether the subject is a living body based on a temperature distribution within the region of interest; A computer program for causing a computer to execute the above.

Images