Authentication device, authentication method, and program

Abstract

To provide a face authentication system configured to authenticate a person to be authenticated, using a face image of the person even when a part of the face of the person is covered with a mask.SOLUTION: In a personal authentication system 100, a face authentication apparatus 120 includes: an input unit which inputs image data to be subjected to authentication processing, to a first authentication engine which executes authentication processing based on first feature information; an image acquisition unit 150 which acquires image data of a captured image obtained by imaging an object to be authenticated; an authentication code acquisition unit 154 which acquires authentication information allocated to the authentication object; a complementary image extraction unit 156 which refers to a non-masked image storage unit and extracts image data of a non-masked image or part of the non-masked image which is associated with the authentication information acquired by an authentication information acquisition unit; and a synthetic image generation unit 160 which synthesizes the captured image with the non-masked image or the part of the non-masked image extracted by the complementary image extraction unit 156 to generate a synthetic image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an authentication device, an authentication method, and a program. [Background technology]

[0002] Patent Document 1 discloses a face recognition system that performs face recognition using a plurality of recognition engines. [Prior art document] [Patent documents] [Patent Document 1] Patent No. 6839313 Summary of the Invention [Means for solving the problem]

[0003] In a first aspect of the present invention, an authentication device is provided. The authentication device authenticates an authentication target based on, for example, (i) first feature information indicating an external feature of an object that is a part of the authentication target, and (ii) authentication information that is information different from the first feature information and that is assigned to the authentication target. The authentication device includes, for example, an input unit that inputs image data to be the target of authentication processing to a first authentication engine that executes authentication processing based on the first feature information. The authentication device includes, for example, a captured image acquisition unit that acquires image data of a captured image of an object to be authenticated. The authentication device includes, for example, an authentication information acquisition unit that acquires authentication information assigned to the authentication target. The authentication device includes, for example, an extraction unit that refers to, for example, (i) the authentication information assigned to each of one or more authentication targets, and (ii) an unobstructed image storage unit that stores image data of at least one of an unobstructed image and a part of an unobstructed image, which is an image of the external appearance of each of the one or more authentication targets, in which the object is not obstructed by a predetermined type of obstruction, in association with each other, and extracts image data of an unobstructed image or a part of an unobstructed image associated with the authentication information acquired by the authentication information acquisition unit. The authentication device includes, for example, a synthesis unit that generates a synthetic image by synthesizing the captured image with the unobstructed image or a part of the unobstructed image extracted by the extraction unit. In the authentication device, for example, when the authentication information acquisition unit acquires the authentication information, the input unit inputs image data of the synthetic image generated by the synthesis unit to the first authentication engine.

[0004] In the above authentication device, when the captured image acquisition unit acquires image data of the captured image and the authentication information acquisition unit has not acquired the authentication information, the input unit may input image data of the captured image or a portion of the captured image to the first authentication engine.

[0005] In the above authentication device, the authentication information may be written, displayed, or projected on an article attached to or held by the authentication target. In the above authentication device, the authentication information acquisition unit may acquire the authentication information assigned to the authentication target by analyzing image data of the captured image. In the above authentication device, the article may be an obstruction.

[0006] In the authentication device, the authentication information may be stored in a storage device of a communication terminal attached to or held by the authentication target. In the authentication device, the authentication information acquisition unit may acquire the authentication information assigned to the authentication target from the communication terminal.

[0007] In the above authentication device, the synthesis unit may generate a synthetic image by referring to a shielding object storage unit that stores authentication information assigned to the authentication target and information indicating a type of shielding object in association with each other.

[0008] The authentication device may include an exposed image detection unit that detects an image of an exposed part that is not covered by a shielding object from among images of the object included in the captured image. The authentication device may include a second authentication engine that is an authentication engine that authenticates the authentication object using an algorithm different from that of the first authentication engine and executes an authentication process based on second feature information indicating an external feature of a predetermined part of the object. In the authentication device, when image data of an image of the exposed part detected by the exposed image detection unit is input, the second authentication engine may execute an authentication process based on an external feature of the exposed part and output information indicating one or more candidates for the authentication object. In the authentication device, the first authentication engine may determine whether or not to authenticate the authentication object by determining whether or not the first feature information of the subject of the image data input by the input unit matches each of the first feature information of the one or more candidates.

[0009] The authentication device may include an unobstructed image acquisition unit that acquires image data of an unobstructed image of each of the one or more authentication targets. The authentication device may include a blocked image acquisition unit that acquires image data of a blocked image, which is an image of the appearance of each of the one or more authentication targets, in which a part of the target is blocked by a predetermined type of blocking object. The authentication device may include a blocked image detection unit that detects an image of a part corresponding to a blocked part in each of the blocked images of the one or more authentication targets, which is blocked by a blocking object, from the unobstructed images of each of the one or more authentication targets. The authentication device may include an unobstructed image management unit that stores at least one of image data of the unobstructed image acquired by the unobstructed image acquisition unit and the image detected by the blocked image detection unit in the unobstructed image storage unit in association with authentication information assigned to each of the one or more authentication targets.

[0010] The authentication device may include an authentication information detection unit that analyzes image data of the obstructed image and detects authentication information included in the obstructed image. In the authentication device, the unobstructed image management unit may store image data of at least one of the unobstructed image acquired by the unobstructed image acquisition unit and the image detected by the obstructed image detection unit in the unobstructed image storage unit in association with the authentication information detected by the authentication information detection unit. In the authentication device, the authentication target may be a person or an animal. The target object may be a face.

[0011] In a second aspect of the present invention, an assistance device is provided. The assistance device assists, for example, an authentication device that authenticates an authentication target. The assistance device includes, for example, an authentication information acquisition unit that acquires authentication information assigned to the authentication target. The assistance device includes, for example, an unobstructed image acquisition unit that acquires image data of an unobstructed image, which is an image of the appearance of an object that is a part of the authentication target, and in which the object is not obscured by a predetermined type of obscurant. The assistance device includes, for example, an obscured image acquisition unit that acquires image data of an obscured image, which is an image of the appearance of the object to be authenticated, and in which a part of the object is obscured by a predetermined type of obscurant. The assistance device includes, for example, an obscured image detection unit that detects, from the unobscured image, an image of a part that corresponds to an obscured part in the obscured image that is obscured by an obscurant. The assistance device includes, for example, an unobscured image management unit that causes a storage device to store, in association with each other, image data of the image detected by the obscured image detection unit and authentication information assigned to the authentication target.

[0012] In a third aspect of the present invention, there is provided an authentication method. The authentication method is, for example, an authentication method for authenticating an authentication target based on (i) first feature information indicating an external feature of an object that is a part of the authentication target, and (ii) authentication information that is information different from the first feature information and that is assigned to the authentication target. The authentication method includes, for example, an input step of inputting image data to be the target of authentication processing to a first authentication engine that executes authentication processing based on the first feature information. The authentication method includes, for example, a captured image acquisition step of acquiring image data of an image of an object to be authenticated. The authentication method includes, for example, an authentication information acquisition step of acquiring authentication information assigned to the authentication target. The authentication method includes, for example, an extraction step of extracting image data of an unobstructed image or a part of an unobstructed image associated with the authentication information acquired in the authentication information acquisition step, by referring to an unobstructed image storage device that stores, in association with each other, (i) authentication information assigned to each of one or more authentication targets, and (ii) image data of at least one of an unobstructed image and a part of an unobstructed image, which are images of the external appearance of each of the one or more authentication targets, in which the object is not obstructed by a predetermined type of obstruction. The authentication method includes, for example, a synthesis step of generating a synthetic image by synthesizing a captured image with an unobstructed image or a part of the unobstructed image extracted in the extraction step. In the authentication method, for example, the input step includes a step of inputting image data of the synthetic image generated in the synthesis step to a first authentication engine when authentication information is acquired in the authentication information acquisition step. Each step of the authentication method may be executed by a computer.

[0013] In a fourth aspect of the present invention, a support method is provided. The support method is, for example, a support method for supporting an authentication device that authenticates an authentication target. The support method includes, for example, an authentication information acquisition step of acquiring authentication information assigned to the authentication target. The support method includes, for example, an unobstructed image acquisition step of acquiring image data of an unobstructed image, which is an image of the appearance of an object that is a part of the authentication target, and in which the object is not obscured by a predetermined type of obscurant. The support method includes, for example, an obscured image acquisition step of acquiring image data of an obscured image, which is an image of the appearance of the object to be authenticated, and in which a part of the object is obscured by a predetermined type of obscurant. The support method includes, for example, an obscured image detection step of detecting, from the unobscured image, an image of a part corresponding to an obscured part in the obscured image that is obscured by the obscurant. The support method includes, for example, an unobscured image management step of storing, in a storage device, image data of the image detected in the obscured image detection step and authentication information assigned to the authentication target in association with each other. Each step of the support method may be executed by a computer.

[0014] In a fifth aspect of the present invention, a program is provided. The above program may be a program for causing a computer to function as an authentication device according to the above first embodiment. The above program may be a program for causing a computer to function as an assistance device according to the above second embodiment. The above program may be a program for causing a computer to execute an authentication method according to the above third embodiment. The above program may be a program for causing a computer to execute an assistance method according to the above fourth embodiment. A computer-readable medium storing the above program may be provided. The computer-readable medium may be a non-transitory computer-readable medium. The computer-readable medium may be a computer-readable recording medium.

[0015] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0016] [Figure 1] 1 shows an example of a system configuration of a personal authentication system 100. [Diagram 2] 2 shows another example of the system configuration of the personal authentication system 100. [Diagram 3] 2 shows an example of an internal configuration of a registration unit 130. [Figure 4] 2 shows an example of the internal configuration of the storage unit 140. [Diagram 5] 1 shows an example of a system configuration of a personal authentication system 500. [Figure 6] 1 illustrates an example of an authentication processing method. [Figure 7] 4 illustrates a schematic diagram of another example of an authentication processing method. [Figure 8] 8 shows an example of the internal configuration of a face authentication device 820. [Figure 9] 1 illustrates an example of a registration process. [Figure 10] An example of the system configuration of a computer 3000 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The present invention will be described below through the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numbers are used for the same or similar parts, and duplicate explanations may be omitted.

[0018] [Outline of the Personal Authentication System 100] 1 shows an example of a system configuration of a personal authentication system 100. In this embodiment, the details of the personal authentication system 100 will be described taking as an example a case where the personal authentication system 100 executes authentication processing of the person to be authenticated 20 using an image of the face 22 of the person to be authenticated 20.

[0019] In this embodiment, the personal authentication system 100 includes an imaging device 110 and a face authentication device 120. In this embodiment, the imaging device 110 and the face authentication device 120 can transmit and receive information to and from each other. In this embodiment, the face authentication device 120 includes a registration unit 130, a storage unit 140, an image acquisition unit 150, a face detection unit 152, an authentication code acquisition unit 154, a complement image extraction unit 156, a composite image generation unit 160, and a face authentication engine 170. In this embodiment, the storage unit 140 includes a complement information storage unit 142 and a registration information storage unit 144.

[0020] In this embodiment, each unit of the personal authentication system 100 (for example, at least one of the imaging device 110 and the face authentication device 120) can transmit and receive information to and from a communication terminal 102 used by the person to be authenticated 20 (sometimes referred to as the communication terminal 102 of the person to be authenticated 20) via the communication network 10. Furthermore, each unit of the personal authentication system 100 may transmit and receive information to and from the communication terminal 102 of the person to be authenticated 20 by short-distance wireless communication or wired communication.

[0021] Incidentally, a conventional face authentication system performs a discrimination process or identity verification process for the person to be authenticated 20 by comparing or collating a feature amount obtained by analyzing a facial image of the person to be authenticated 20 captured during authentication with a feature amount obtained by analyzing a facial image of the person to be authenticated 20 registered in advance. The discrimination process for the person to be authenticated 20 may be an example of an authentication process. The identity verification process may be an example of an authentication process. In the conventional face authentication system, the above feature amount is calculated based on the detection result of feature points such as the eyes, nose, and mouth. Therefore, when a part of the face 22 of the person to be authenticated 20 is covered with, for example, a mask 32, the face authentication system cannot detect the feature points of the part covered with the mask 32, and cannot authenticate the person to be authenticated 20.

[0022] As a method for realizing a face authentication system capable of authenticating the person to be authenticated 20 using an image of the face 22 of the person to be authenticated 20 even when a part of the face 22 of the person to be authenticated 20 is covered by a mask 32, for example, an authentication engine that performs face authentication using the feature amounts of the feature points of the upper half of the face may be incorporated into the face authentication system in addition to an authentication engine that performs face authentication using the feature amounts of the feature points of the entire face. For example, when the face authentication system determines that the person to be authenticated 20 is not wearing a mask 32, the face authentication system executes authentication processing of the person to be authenticated 20 using an authentication engine that performs face authentication using the feature amounts of the feature points of the entire face. On the other hand, when the face authentication system determines that the person to be authenticated 20 is wearing a mask 32, the face authentication system executes authentication processing of the person to be authenticated 20 using an authentication engine that performs face authentication using the feature amounts of the feature points of the upper half of the face.

[0023] However, in order to realize the above method, in addition to a normal authentication engine that performs face authentication using the feature values ​​of the feature points of the entire face, it is necessary to implement an authentication engine that performs face authentication using the feature values ​​of the feature points of the upper half of the face, which increases the development cost of the authentication system. In addition, the article that covers the face of the person to be authenticated 20 is not limited to the mask 32. For example, when the person to be authenticated 20 wears sunglasses, the upper half of the face of the person to be authenticated 20 is covered by the sunglasses. Therefore, when a part of the face 22 is covered by an arbitrary article (sometimes called an obscuring item), in order to accurately authenticate the person to be authenticated 20 using the feature values ​​of the feature points of the part of the face 22 that is not covered by the article, it is necessary to implement multiple authentication engines such as an authentication engine for a person wearing a mask and an authentication engine for a person wearing sunglasses in addition to a normal authentication engine that performs face authentication using the feature values ​​of the feature points of the entire face, which further increases the development cost of the authentication system.

[0024] When a part of the face 22 of the person to be authenticated 20 is covered by an article, the area of ​​the exposed part of the face 22 is smaller than when the face 22 of the person to be authenticated 20 is not covered by an article. As a result, the authentication accuracy of an authentication engine for a mask wearer, an authentication engine for a sunglasses wearer, etc. tends to be smaller than the authentication accuracy of an authentication engine that performs face authentication using the feature amounts of feature points of the entire face.

[0025] Therefore, even if the above method is adopted and multiple authentication engines such as an authentication engine for mask wearers and an authentication engine for sunglasses wearers are implemented, depending on how the thresholds for judgment in the authentication process are adjusted, multiple users registered in advance may be extracted as candidates for the person to be authenticated 20, making it impossible to identify the person to be authenticated 20. Furthermore, depending on how the thresholds are adjusted, the authentication system may not be able to authenticate the person to be authenticated 20, even if the person to be authenticated 20 is the same user registered in advance.

[0026] Therefore, the personal authentication system 100 according to this embodiment authenticates the person to be authenticated 20 based on (i) the feature amount of the face 22 of the person to be authenticated 20 and (ii) the authentication code assigned to the person to be authenticated 20. As a result, according to one embodiment of the personal authentication system 100, the authentication process of the person to be authenticated 20 can be executed using the same authentication engine in both cases where the face 22 of the person to be authenticated 20 is partially covered by an article and where the face 22 of the person to be authenticated 20 is not covered by an article. According to another embodiment of the personal authentication system 100, when the authentication process of the person to be authenticated 20 is executed using a plurality of authentication engines, it may be easy to adjust the threshold value for judgment in the authentication process. Furthermore, the amount of calculation in the authentication process may be reduced.

[0027] Moreover, according to the personal authentication system 100 of this embodiment, the user experience of the person to be authenticated 20 is greatly improved. For example, information devices such as PCs, tablets, and smartphones are configured to automatically lock the screen if no user operation is performed for a certain period of time. Depending on the settings of the information device, face authentication may be required when the user releases the lock and resumes use of the information device. For example, if the user is wearing a mask, the user must put on and take off the mask every time face authentication is performed, which causes stress. Therefore, for example, when the user uses the information device to perform work, the efficiency of the work may also decrease.

[0028] In contrast, according to the personal authentication system 100 of the present embodiment, a face authentication system capable of authenticating a user wearing a mask based on a facial image of the user can be easily constructed. As a result, the stress level of the user can be reduced and work efficiency can be improved.

[0029] [Example of embodiment] For example, according to one embodiment of the personal authentication system 100, when a part of the face 22 of the person to be authenticated 20 is covered by an arbitrary article, the personal authentication system 100 uses the authentication code to generate a composite image in which the face 22 of the person to be authenticated 20 is not covered by the arbitrary article. The personal authentication system 100 authenticates the person to be authenticated 20 using the composite image.

[0030] Specifically, first, a first database is prepared by performing a registration process for each user, in which (i) an authentication code assigned to each of one or more users and (ii) at least one of image data of an image of the appearance of the face of each of one or more users (sometimes simply referred to as a face image) in which the face of each user is not covered by an article (sometimes referred to as an unworn image) and image data of a part of the unworn image (sometimes referred to as a complementary image) are stored in association with each other. The first database may store (i) identification information (sometimes referred to as a user ID) for identifying each of one or more users, (ii) an authentication code assigned to each of one or more users, and (iii) image data of the unworn image and / or complementary image of each of one or more users in association with each other.

[0031] The complementary image may be an image of a portion of the unworn image where the user's face is covered by an article. The complementary image may be an image of the unworn image including a portion of the user's face covered by an article. The complementary image is generated, for example, based on an image in which the user's face is not covered by an article and an image in which the user's face is covered by an article.

[0032] If the article is a mask, the complementary image may be an image of a part of the user's face that is covered by the mask when the user wears the mask, or an image of the lower half of the user's face. If the article is eyewear, the complementary image may be an image of a part of the user's face that is covered by the eyewear when the user wears the eyewear, or an image of the upper half of the user's face.

[0033] The first database may store a plurality of unworn images relating to a single user. For example, the first database stores a single user ID, image data of an unworn image in a state where the face of a user identified by the user ID is not covered by a first type of article, and image data of an unworn image in a state where the face of the user is not covered by a second type of article, in association with each other. The second type of article may be an article of a different type from the first type of article. For example, the first type of article is a mask, and the second type of article is sunglasses.

[0034] Similarly, the first database may store a plurality of complementary images for a single user. For example, the first database stores a single user ID, image data of a complementary image of a portion of the face of the user identified by the user ID that is covered with a first type of item, and image data of a complementary image of a portion of the face of the user that is covered with a second type of item, in association with each other.

[0035] The first database may be a database different from the second database that stores (i) the user IDs of one or more users and (ii) images of the facial appearances of the one or more users in association with each other. The contents of the first database and the second database may be integrated to form a single database.

[0036] In the personal authentication system 100, the type of the article that shields the target of the authentication process may be determined in advance. The type of the target and the type of the article that shields the target may be associated in advance. For example, when the target is a face (that is, when the personal authentication system 100 executes face authentication process), various types of articles that can shield a part of the face or head may be adopted as the predetermined types of articles.

[0037] Examples of types of shielding items that can shield a part of the face or head include hats, bandanas, headphones, earmuffs, eyewear, masks, handkerchiefs, scarves, mufflers, folding fans, cloth, paper, etc. Examples of eyewear include glasses, sunglasses, goggles, eye patches, contact lenses, etc. Contact lenses may be intended to adjust vision, or may be intended to adjust the color, shape, or size of the pupil of the eye.

[0038] In one embodiment, the type of the shielding item is determined based on the general name of the shielding item. The general name may be a name that is more general or standard than the product name of the shielding item, and may be a simple name, a common name, a customary name, or a common name. Examples of the general name of the shielding item include a hat, a bandana, headphones, earmuffs, eyewear, a mask, a handkerchief, a scarf, a muffler, a folding fan, a folding fan, a cloth, and a paper.

[0039] In another embodiment, the type of shielding item is determined based on the general name of the shielding item and the specifications of the shielding item. Examples of the specifications of the shielding item include use, size, shape, color, pattern, material, etc. For example, a small size mask and a medium size mask can be treated as different types of shielding items. Also, a urethane mask and a nonwoven mask can be treated as different types of shielding items.

[0040] In still another embodiment, the type of the occluding item is determined based on the product name or model number of the occluding item. The type of the occluding item may be determined based on the manufacturer name and / or distributor name of the occluding item and the product name or model number of the occluding item. In this way, the personal authentication system 100 can, for example, refer to a database that stores the model number of the occluding item in association with the size and / or shape of the occluding item, and identify the size and / or shape of the occluding item from the model number of the occluding item.

[0041] In the personal authentication system 100, the first database may store image data of a complementary image in association with information indicating a type of article corresponding to the complementary image. For example, when the complementary image is an image of a part of a user's face that is covered by a mask when the user wears the mask (sometimes referred to as a complementary image corresponding to a mask), the first database stores image data of the complementary image in association with information indicating that the type of article covering the user's face is a mask.

[0042] As described above, the first database can store multiple complementary images related to a single user. According to the present embodiment, the first database stores, for each user, image data of each of the multiple complementary images in association with information indicating the type of article corresponding to each complementary image. This allows the personal authentication system 100 to easily extract a complementary image corresponding to the mask, for example, when it is determined that the person to be authenticated 20 is wearing a mask 32.

[0043] As described above, the information indicating the type of the article includes, for example, information indicating the product name or model number of the article. Furthermore, the personal authentication system 100 can, for example, refer to a database that stores the model number of the occluding item in association with the size and / or shape of the occluding item, and identify the size and / or shape of the occluding item from the model number of the occluding item. This allows the personal authentication system 100 to identify the position, size and / or range of the complement image in the image of the user's entire face, using the information indicating the type of the article.

[0044] When the personal authentication system 100 authenticates a specific person 20 to be authenticated, the personal authentication system 100 first acquires image data of an image (sometimes referred to as a captured image) of the face 22 of the person 20 to be authenticated. The personal authentication system 100 also acquires an identification code of the person 20 to be authenticated. For example, the personal authentication system 100 analyzes the captured image to detect an authentication code reflected in the captured image. The personal authentication system 100 may acquire the authentication code from an information processing terminal used by the person 20 to be authenticated via a communication line.

[0045] Next, the personal authentication system 100 refers to the first database and extracts image data of the unattached image or the complemented image associated with the identification code of the person to be authenticated 20. The personal authentication system 100 synthesizes the captured image and the extracted unattached image or the complemented image to generate an image of the person to be authenticated 20 not wearing the shielding item (sometimes referred to as a composite image).

[0046] For example, when the person to be authenticated 20 is wearing a mask 32, in the captured image, the upper half of the face 22 of the person to be authenticated 20 is exposed, and the lower half of the face 22 of the person to be authenticated 20 is covered by the mask 32. In this case, in the composite image, the image of the upper half of the face is derived from the captured image. Also, in the composite image, the image of the lower half of the face is derived from the extracted unmasked image or the complemented image.

[0047] As a result, for example, if the person to be authenticated 20 is user A and the authentication code acquired by the personal authentication system 100 matches the authentication code assigned to user A, a composite image of the face of user A without wearing the mask 32 is generated. On the other hand, even if the authentication code acquired by the personal authentication system 100 matches the authentication code assigned to user A, if the person to be authenticated 20 is not user A, a composite image is generated in which the upper half of the face of the person to be authenticated 20 and the lower half of the face of user A are combined. Note that even if the person to be authenticated 20 is user A, if the authentication code assigned to user A cannot be acquired, for example, a composite image is not generated.

[0048] After that, the personal authentication system 100 inputs the image data of the composite image to an authentication engine that compares the feature amounts of the feature points of the entire face. The authentication engine analyzes the composite image and performs authentication processing of the person to be authenticated 20. The authentication engine, for example, refers to a second database and determines whether or not there is a user whose overall facial features match those of the person to be authenticated 20 among multiple users registered in the second database.

[0049] According to this embodiment, an input image for an existing authentication engine is generated from a captured image of the face 22 of the person to be authenticated 20 in accordance with the specifications of the existing authentication engine. As a result, according to this embodiment, it is possible to execute authentication processing of the person to be authenticated 20 using the same authentication engine, for example, when the person to be authenticated 20 is wearing a mask 32 and when the person to be authenticated 20 is not wearing a mask 32. The above authentication engine may be an authentication engine that compares feature amounts of feature points of the entire face.

[0050] [Another Example of the Embodiment] For example, according to another embodiment of the personal authentication system 100, the personal authentication system 100 uses a plurality of authentication engines to perform authentication processing of the person to be authenticated 20. For example, the personal authentication system 100 performs authentication processing of the person to be authenticated 20 using (i) an authentication engine that authenticates the person to be authenticated 20 using image data of an image of the entire face, and (ii) an authentication engine that authenticates the person to be authenticated 20 using image data of an image of a portion of the face. The authentication engine that authenticates the person to be authenticated 20 using image data of an image of a portion of the face authenticates the person to be authenticated 20 using, for example, image data of an image of a portion of the face 22 of the person to be authenticated 20 that is not covered by an obstructing item (sometimes referred to as an exposed portion).

[0051] Furthermore, in this embodiment, the personal authentication system 100 authenticates the person to be authenticated 20 based on an authentication code assigned to the person to be authenticated 20. For example, at least one of the above-mentioned multiple authentication engines authenticates the person to be authenticated 20 based on the authentication code assigned to the person to be authenticated 20.

[0052] Specifically, first, the personal authentication system 100 acquires image data of the captured image of the person to be authenticated 20 and an identification code of the person to be authenticated 20 by a procedure similar to that described above. Next, the personal authentication system 100 determines whether or not a part of the face 22 of the person to be authenticated 20 is covered by an arbitrary article.

[0053] When it is determined that the face 22 of the person to be authenticated 20 is not partially covered by any object, the personal authentication system 100 inputs the image data of the captured image to an authentication engine that authenticates the person to be authenticated 20 using image data of the entire face image, among a plurality of authentication engines. Thus, authentication processing of the person to be authenticated 20 is executed.

[0054] On the other hand, when it is determined that a part of the face 22 of the person to be authenticated 20 is covered by an arbitrary article, the personal authentication system 100 inputs the image data of the captured image to an authentication engine among the multiple authentication engines, which authenticates the person to be authenticated 20 using image data of the image of the part of the face. The authentication engine executes authentication processing of the person to be authenticated 20 using the image of the exposed part of the face 22 of the person to be authenticated 20. The authentication engine, for example, refers to the second database and determines whether or not there is a user who matches the characteristics of the exposed part of the face 22 of the person to be authenticated 20 from among the multiple users registered in the second database.

[0055] When a single user is extracted as a user matching the characteristics of the exposed part of the face 22 of the person to be authenticated 20, the personal authentication system 100 outputs the result as the authentication result. On the other hand, when multiple users are extracted as users matching the characteristics of the exposed part of the face 22 of the person to be authenticated 20, the personal authentication system 100 continues the authentication process using an authentication engine that authenticates the person to be authenticated 20 using image data of the entire face image. Furthermore, when a user matching the characteristics of the exposed part of the face 22 of the person to be authenticated 20 is not extracted, the judgment criteria of the authentication engine may be adjusted so that multiple users are extracted.

[0056] When multiple users are extracted as users matching the characteristics of the exposed part of the face 22 of the person to be authenticated 20, an authentication engine that authenticates the person to be authenticated 20 using image data of an image of a part of the face outputs, for example, a user ID of a candidate for the person to be authenticated 20. Furthermore, the personal authentication system 100 uses an authentication code in a procedure similar to that of the above-described method to generate a composite image in which the face 22 of the person to be authenticated 20 is not occluded by the object.

[0057] After that, in the same procedure as the above-mentioned method, the personal authentication system 100 inputs the image data of the composite image to an authentication engine among a plurality of authentication engines, which uses image data of the whole face image to authenticate the person to be authenticated 20. The authentication engine analyzes the composite image to authenticate the person to be authenticated 20.

[0058] In this case, the authentication engine that authenticates the person to be authenticated 20 using image data of the image of the entire face may authenticate the person to be authenticated 20 by comparing the registered images of each of the multiple users identified by the user IDs output by the authentication engine that authenticates the person to be authenticated 20 using image data of the image of a part of the face with the above-mentioned composite image. This reduces the load on the authentication engine that authenticates the person to be authenticated 20 using image data of the image of the entire face compared to a case in which the authentication engine that authenticates the person to be authenticated 20 using image data of the image of the entire face compares the registered images of all users registered in the second database with the above-mentioned composite image. In addition, it becomes easier to develop an authentication engine that authenticates the person to be authenticated 20 using image data of the image of a part of the face. In addition, it may become easier to adjust the threshold value for judgment in the authentication process.

[0059] [Outline of each part related to the personal authentication system 100] In this embodiment, the communication network 10 may be a transmission path for wired communication, a transmission path for wireless communication, or a combination of a transmission path for wireless communication and a transmission path for wired communication. The communication network 10 includes, for example, at least one of a wireless packet communication network, the Internet, a P2P network, a dedicated line, a VPN, a power line communication line, a vehicle-to-vehicle communication line, and a road-to-vehicle communication line.

[0060] The communication network 10 may include (i) a mobile communication network such as a mobile phone network, or (ii) a wireless communication network such as a wireless MAN (e.g., WiMAX (registered trademark)), a wireless LAN (e.g., WiFi (registered trademark)), Bluetooth (registered trademark), Zigbee (registered trademark), NFC (Near Field Communication), etc. As described above, the wireless LAN, Bluetooth (registered trademark), Zigbee (registered trademark), and NFC may be examples of short-distance wireless communication methods.

[0061] In this embodiment, the mask 32 covers a part of the face 22 of the person to be authenticated 20. As a result, a part of the face 22 of the person to be authenticated 20 is occluded.

[0062] In this embodiment, the authentication code 34 is assigned to each of the one or more users. The authentication code 34 may be information different from identification information (sometimes referred to as a user ID) used by the personal authentication system 100 to identify each of the one or more users, may include the user ID, or may be the same as the user ID. The authentication code 34 may include identification information (sometimes referred to as an item ID) used to identify each of the occluding items, or may be the same as the item ID.

[0063] The authentication code 34 may be assigned to each of one or more shielding items. In this case, for example, in the personal authentication system 100, the user ID of a specific user is associated with the authentication code 34 assigned to the shielding item, so that the authentication code 34 assigned to the shielding item is also assigned to the specific user.

[0064] The authentication code 34 may (i) include a character string consisting of at least one of numbers, letters, and symbols, or (ii) be a graphic or image in which information including the character string is encoded. The authentication code 34 may be a graphic or image in which a code assigned to a user or an obscuring item and information indicating attributes of the user or obscuring item are encoded. Examples of user attributes include height, weight, and face shape. Examples of obscuring item attributes include product name, model number, and specifications. Examples of graphic or image in which information including a character string is encoded include a barcode and a two-dimensional code.

[0065] The authentication code 34 may be written, displayed or projected on an article worn or held by each user (e.g., the person to be authenticated 20). The above article may be an article that covers a part of the object of the authentication process by the personal authentication system 100 (e.g., the face 22 of the person to be authenticated 20). The above article does not have to be an article that covers a part of the object of the authentication process by the personal authentication system 100 (e.g., the face 22 of the person to be authenticated 20). When the authentication process by the personal authentication system 100 is face authentication process, examples of articles that do not cover the object include a mobile terminal of each user, a card-like or board-like object that each user can hold in their hand, and the like.

[0066] In one embodiment, the authentication code 34 is written on the surface of the mask 32. For example, the authentication code 34 is printed on the surface of the mask 32 in a manner that is sufficiently visible to normal human vision. The authentication code 34 may be printed on the surface of the mask 32 in a manner that is not visible to normal human vision. For example, the authentication code 34 is printed using special ink such as invisible ink, hidden ink, or IR transparent ink. The authentication code 34 may be expressed by fine dots. The authentication code 34 may be printed using a hidden printing technique or a copy prevention technique. In another embodiment, the authentication code 34 is projected on the surface of the mask 32. In yet another embodiment, the authentication code 34 is displayed or projected on a display device of the communication terminal 102 of the person to be authenticated 20.

[0067] The authentication code 34 may be written or projected on a part of the body of each user. For example, the authentication code 34 is printed or projected on the face or head of each user. For example, the authentication code 34 is printed or projected on the hand, arm, or chest of each user. In this way, for example, when the authentication process by the personal authentication system 100 is a facial authentication process, when the face or upper body of each user is imaged, the authentication code 34 can be imaged together with the face of each user. The authentication code 34 may be printed or projected on the foot of each user, or on the back of each user.

[0068] In this embodiment, the communication terminal 102 transmits and receives information to and from each part of the personal authentication system 100 via the communication network 10. The communication terminal 102 may transmit and receive information to and from each part of the personal authentication system 100 by a short-distance wireless communication method using electromagnetic waves or sound waves. The communication terminal 102 may transmit and receive information to and from each part of the personal authentication system 100 by a wired communication method.

[0069] Examples of the communication terminal 102 include a personal computer, a mobile terminal, etc. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA, a tablet, a notebook computer or laptop computer, a wearable computer, etc. The communication terminal 102 may be a card on which an IC chip, a memory, etc. are arranged.

[0070] In this embodiment, the imaging device 110 captures an image of the person to be authenticated 20 and generates a captured image 40 of the face 22 of the person to be authenticated 20. The imaging device 110 may output image data of the captured image 40 to the face authentication device 120.

[0071] In this embodiment, the face authentication device 120 executes authentication processing of the person to be authenticated 20. The face authentication device 120 executes authentication processing of the person to be authenticated 20 by using the captured image 40 output by the imaging device 110.

[0072] In this embodiment, the registration unit 130 registers a face image of each of the one or more users. The registration unit 130 may also register an authentication code 34 assigned to each of the one or more users. The registration unit 130 will be described in detail later.

[0073] In this embodiment, the storage unit 140 stores various types of information. In one embodiment, the storage unit 140 stores various types of information used in information processing of the personal authentication system 100 or the face authentication device 120. In another embodiment, the storage unit 140 stores various types of information generated by information processing of the personal authentication system 100 or the face authentication device 120. Details of the storage unit 140 will be described later.

[0074] In this embodiment, the complementary information storage unit 142 stores the first database described above. In one embodiment, the complementary information storage unit 142 stores (i) an authentication code 34 assigned to each of one or more users, and (ii) image data of at least one of a non-worn image and a complementary image 44 of each of the one or more users, in association with each other. In another embodiment, the complementary information storage unit 142 stores (i) a user ID for identifying each of one or more users, (ii) an authentication code 34 assigned to each of one or more users, and (iii) image data of at least one of a non-worn image and a complementary image 44 of each of the one or more users, in association with each other.

[0075] In this embodiment, for the purpose of simplifying the explanation, the details of each part of the face authentication device 120 will be explained by taking as an example a case where the complementary information storage unit 142 stores (i) an authentication code 34 assigned to each of one or more users and (ii) image data of the complementary image 44 of each of the one or more users in association with each other. The details of the complementary information storage unit 142 will be described later.

[0076] In this embodiment, the registration information storage unit 144 stores the second database described above. For example, the registration information storage unit 144 stores (i) the user ID of each of one or more users and (ii) the unworn image of each of one or more users in association with each other. The details of the registration information storage unit 144 will be described later.

[0077] In this embodiment, the image acquisition unit 150 acquires the captured image 40 output by the imaging device 110. The image acquisition unit 150 outputs the captured image 40 to the face detection unit 152 and the authentication code acquisition unit 154.

[0078] In this embodiment, the face detection unit 152 detects an image of the face 22 of the person to be authenticated 20 from the captured image 40 of the person to be authenticated 20 acquired by the image acquisition unit 150. The face detection unit 152 generates a face image 42 by, for example, cutting out the image of the face 22 detected from the captured image 40 and adjusting the size, color tone, etc. of the cut-out image. The face detection unit 152 outputs the face image 42 to the composite image generation unit 160.

[0079] The face detection unit 152 may input the image data of the face image 42 to the face recognition engine 170 as image data to be subjected to the authentication process. For example, when the image acquisition unit 150 acquires image data of the captured image 40 and the authentication code acquisition unit 154 does not acquire the authentication code 34, the face detection unit 152 inputs the image data of the face image 42 to the face recognition engine 170 as image data to be subjected to the authentication process.

[0080] As a result, for example, when the person to be authenticated 20 is not wearing a mask 32, an entire image of the face 22 of the person to be authenticated 20 captured by the imaging device 110 is input to the face authentication engine 170. As a result, authentication processing of the person to be authenticated 20 is executed.

[0081] In this embodiment, the authentication code acquisition unit 154 acquires the authentication code 34 assigned to the person to be authenticated 20. The authentication code 34 may be presented by the person to be authenticated 20 in various ways. The method of acquiring the authentication code 34 is not particularly limited, but in this embodiment, the authentication code acquisition unit 154 acquires the authentication code 34 assigned to the person to be authenticated 20 by analyzing image data of the captured image 40 in which the authentication code 34 appears. The authentication code acquisition unit 154 outputs the authentication code 34 assigned to the person to be authenticated 20 to, for example, the complementary image extraction unit 156. The authentication code acquisition unit 154 may output the authentication code 34 assigned to the person to be authenticated 20 to the composite image generation unit 160.

[0082] In this embodiment, the complementary image extraction unit 156 acquires the authentication code 34 assigned to the authentication subject 20 from the authentication code acquisition unit 154. The complementary image extraction unit 156 refers to the complementary information storage unit 142, and extracts image data of the complementary image 44 corresponding to the authentication code 34 acquired by the authentication code acquisition unit 154. The complementary image extraction unit 156 outputs the image data of the extracted complementary image 44 to the composite image generation unit 160.

[0083] At this time, the complementary image extraction unit 156 may, for example, refer to the complementary information storage unit 142 to determine whether or not an expiration date is set for the authentication code 34. If an expiration date is set for the authentication code 34, the complementary image extraction unit 156 determines whether or not the authentication code 34 is valid. If the authentication code 34 is valid, the complementary image extraction unit 156 executes the above-mentioned extraction operation. On the other hand, if the authentication code 34 is invalid, the complementary image extraction unit 156 may interrupt the above-mentioned extraction operation and output error information indicating that the authentication code 34 is invalid, instead of outputting the image data of the extracted complementary image 44 to the composite image generation unit 160.

[0084] In this embodiment, the composite image generating section 160 generates a composite image 50 by combining the captured image 40 with the complementary image 44 extracted by the complementary image extracting section 156. More specifically, the composite image generating section 160 generates the composite image 50 by combining the face image 42 included in the captured image 40 with the complementary image 44 extracted by the complementary image extracting section 156.

[0085] The composite image generating unit 160 may composite the face image 42 and the complement image 44 so that the contour of the face 22 of the person to be authenticated 20 becomes smooth at the joint between the face image 42 and the complement image 44. A known method may be used to composite two images into one image.

[0086] It is preferable that no obscuring item included in the face image 42 remains in the composite image 50. However, a part of the obscuring item included in the face image 42 may remain in the composite image 50. For example, the composite image generating unit 160 generates the composite image 50 such that the ratio of the area of ​​the portion resulting from the obscuring item included in the face image 42 to the entire area of ​​the composite image 50 is smaller than a predetermined value.

[0087] The composite image generating unit 160 may input the image data of the composite image 50 to the face recognition engine 170 as image data to be subjected to the authentication process. For example, when the authentication code acquiring unit 154 acquires the authentication code 34, the composite image generating unit 160 inputs the image data of the composite image 50 to the face recognition engine 170 as image data to be subjected to the authentication process.

[0088] As a result, for example, when the person to be authenticated 20 is wearing a mask 32, a composite image 50 of the entire face 22 of the person to be authenticated 20 is input to the face authentication engine 170. The face authentication engine 170 analyzes the input composite image 50 and performs authentication processing of the person to be authenticated 20. In this way, according to one embodiment of the personal authentication system 100, it is possible to perform authentication processing of the person to be authenticated 20 using the same authentication engine in both cases where a part of the face 22 of the person to be authenticated 20 is covered by an article and where the face 22 of the person to be authenticated 20 is not covered by an article.

[0089] In one embodiment, the composite image generating unit 160, for example, analyzes the face image 42 to extract the contour of the mask 32 and deletes the mask 32 from the face image 42. Next, the composite image generating unit 160 superimposes the complement image 44 on the face image 42, and then adjusts the position and size of the complement image 44 to generate the composite image 50.

[0090] In another embodiment, the composite image generating unit 160, for example, analyzes the face image 42 to extract the contour of the mask 32. Next, the composite image generating unit 160 superimposes the complementary image 44 on the face image 42, and then adjusts the position and size of the complementary image 44 to match the contour of the mask 32, thereby generating the composite image 50.

[0091] In still another embodiment, the composite image generating unit 160 generates the composite image 50 by referring to a database (not shown) that stores the authentication code 34 of the person to be authenticated 20 and information indicating the type of the article covering the face 22 of the person to be authenticated 20 in association with each other. According to this embodiment, the composite image generating unit 160 first refers to the database and acquires information indicating the type of the article that is associated with the authentication code 34 of the person to be authenticated 20. This allows the composite image generating unit 160 to determine that the type of the article covering the face 22 of the person to be authenticated 20 is a "mask" and that the lower half of the face 22 is covered by the article. Thereafter, the composite image generating unit 160 generates the composite image 50 by combining the face image 42 and the complement image 44 according to the same procedure as in the two embodiments described above.

[0092] As described above, the information indicating the type of item may include, for example, information indicating the product name or model number of the shielding item. When the above database stores the authentication code 34 and information indicating the product name or model number of the mask 32 in association with each other, the composite image generating unit 160 may, for example, refer to another database that stores the model number of the mask 32 in association with the size and / or shape of the mask 32, and obtain information indicating the size and / or shape of the mask 32 from the model number of the mask 32. The composite image generating unit 160 may determine the position, size, and / or range of the complementary image 44 in the face image 42 based on the information indicating the size and / or shape of the mask 32.

[0093] The composite image generating unit 160 may generate the composite image 50 such that the ratio of the area of ​​the portion derived from the facial image 42 to the entire area of ​​the face 22 of the person to be authenticated 20 in the composite image 50 is greater than a predetermined value. If the ratio of the area of ​​the portion derived from the facial image 42 to the entire area of ​​the face 22 of the person to be authenticated 20 in the composite image 50 is smaller than a predetermined value, the composite image generating unit 160 may not generate the composite image 50. In this case, instead of outputting the composite image 50, the composite image generating unit 160 may output error information indicating that generation of the composite image 50 has failed.

[0094] Similarly, the composite image generating unit 160 may generate the composite image 50 such that the ratio of the area of ​​the portion originating from the complement image 44 to the entire area of ​​the face 22 of the person to be authenticated 20 in the composite image 50 is smaller than a predetermined value. If the ratio of the area of ​​the portion originating from the complement image 44 to the entire area of ​​the face 22 of the person to be authenticated 20 in the composite image 50 is larger than a predetermined value, the composite image generating unit 160 may not generate the composite image 50. In this case, instead of outputting the composite image 50, the composite image generating unit 160 may output error information indicating that generation of the composite image 50 has failed.

[0095] In this embodiment, the composite image generating unit 160 may generate the composite image 50 by combining the captured image 40 with the multiple complementary images 44 extracted by the complementary image extracting unit 156. For example, if the complementary information storage unit 142 is configured to be able to store a single authentication code 34 and multiple complementary images 44 in association with each other, the complementary image extracting unit 156 extracts image data of the multiple complementary images 44 corresponding to the authentication code 34 acquired by the authentication code acquiring unit 154.

[0096] For example, when a user has multiple masks 32, a single authentication code 34 and multiple complementary images 44 corresponding to the multiple masks 32 are associated with each other through a registration process by the registration unit 130, and stored in the complementary information storage unit 142. When a user wears a mask 32 in multiple ways, a single authentication code 34 and multiple complementary images 44 corresponding to the multiple wearing ways may be associated with each other through a registration process by the registration unit 130, and stored in the complementary information storage unit 142. Examples of the wearing way of the mask 32 include a way in which the mask 32 completely covers the nose and mouth of the user, a way in which the mask 32 covers the mouth of the user but does not cover the nose, a way in which the mask 32 is worn on the user's chin, a way in which the mask 32 is hung over one of the user's ears, and the like.

[0097] When the complementary image extraction unit 156 extracts image data of a plurality of complementary images 44, the composite image generation unit 160 may generate a plurality of composite images 50 by combining the face image 42 with each of the plurality of complementary images 44. The composite image generation unit 160 may determine one of the plurality of composite images 50 as the composite image 50 to be output as an input to the face recognition engine 170. The composite image generation unit 160 determines the composite image 50 to be output as an input to the face recognition engine 170 based on, for example, the ratio of the area of ​​the portion derived from the face image 42 or the portion derived from the complementary image 44 to the entire area of ​​the face 22 of the person to be authenticated 20 in the composite image 50.

[0098] For example, the composite image generating unit 160 outputs image data of the composite image 50 having the smallest ratio of the total area of ​​the portions derived from the plurality of complement images 44 to the entire face area, among the plurality of composite images 50, as an input to the face recognition engine 170. The composite image generating unit 160 may output image data of the composite image 50 having the largest ratio of the area of ​​the portions derived from the face image 42 to the entire face area, among the plurality of composite images 50, as an input to the face recognition engine 170.

[0099] This allows the face recognition engine 170 to execute the authentication process for the authentication target person 20 by using a composite image in which the ratio of the area of ​​the portion derived from the face image 42 to the area of ​​the entire face in the composite image is large. As a result, the authentication accuracy is improved.

[0100] In this embodiment, the face recognition engine 170 executes the authentication process. For example, the face recognition engine 170 executes the authentication process of the person to be authenticated 20 based on a feature amount indicating the external features of the face 22 of the person to be authenticated 20 (sometimes referred to as the feature amount of the face 22 of the person to be authenticated 20). The face recognition engine 170 may be an authentication engine that executes the authentication process based on the feature amount of the input feature points of the entire face and the feature amount of the registered feature points of the entire face. The face recognition engine 170 may be an authentication engine that executes the authentication process based on the feature amount of the feature points arranged in an area larger than approximately half of the input face image and the feature amount of the feature points arranged in an area larger than approximately half of the registered face image.

[0101] [Specific configuration of each part of the personal authentication system 100] Each part of the personal authentication system 100 may be realized by hardware, software, or hardware and software. At least a part of each part of the personal authentication system 100 may be realized by a single server or by multiple servers. At least a part of each part of the personal authentication system 100 may be realized on a virtual machine or a cloud system. At least a part of each part of the personal authentication system 100 may be realized by a personal computer or a mobile terminal. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA, a tablet, a notebook computer or a laptop computer, and a wearable computer. Each part of the personal authentication system 100 may store information using a distributed ledger technology such as a blockchain or a distributed network.

[0102] When at least some of the components constituting the personal authentication system 100 are realized by software, the components realized by the software may be realized by starting software or a program that defines the operation of the components in an information processing device with a general configuration. The information processing device with the general configuration may include (i) a data processing device having a processor such as a CPU or GPU, a ROM, a RAM, a communication interface, etc., (ii) input devices such as a keyboard, a pointing device, a touch panel, a camera, a voice input device, a gesture input device, various sensors, a GPS receiver, etc., (iii) output devices such as a display device, a voice output device, a vibration device, etc., and (iv) storage devices such as a memory, an HDD, an SSD, etc. (including an external storage device).

[0103] In the information processing device of the above general configuration, the above data processing device or storage device may store the above software or program. The above software or program, when executed by a processor, causes the above information processing device to execute the operation defined by the software or program. The above software or program may be stored in a non-transitory computer-readable recording medium. The above software or program may be a program for causing a computer to function as the personal authentication system 100 or a part thereof. The above software or program may be a program for causing a computer to execute an information processing method in the personal authentication system 100 or a part thereof.

[0104] In one embodiment, the information processing method may be an authentication method for authenticating an object to be authenticated. For example, in the authentication method, the object to be authenticated is authenticated based on (i) first feature information indicating an external feature of an object that is a part of the object to be authenticated, and (ii) authentication information that is information different from the first feature information and that is assigned to the object to be authenticated. The authentication method includes, for example, an input step of inputting image data to be the object of the authentication process to a first authentication engine that executes the authentication process based on the first feature information. The authentication method includes, for example, a captured image acquisition step of acquiring image data of a captured image in which the object to be authenticated is captured. The authentication method includes, for example, an authentication information acquisition step of acquiring authentication information assigned to the object to be authenticated. The above authentication method includes, for example, an extraction step of extracting image data of the unobstructed image or a part of the unobstructed image associated with the authentication information acquired in the authentication information acquisition step by referring to an unobstructed image storage device that stores, in association with each other, (i) authentication information assigned to each of one or more authentication targets and (ii) an unobstructed image or at least one of image data of a part of the unobstructed image, which is an image of the appearance of each of the one or more authentication targets, in which the target is not obstructed by a predetermined type of obstruction. The above authentication method includes, for example, a synthesis step of generating a synthetic image by synthesizing a captured image and the unobstructed image or a part of the unobstructed image extracted in the extraction step. In the above authentication method, the input step includes, for example, a step of inputting image data of the synthetic image generated in the synthesis step to a first authentication engine when authentication information is acquired in the authentication information acquisition step.

[0105] In another embodiment, the information processing method may be a support method for supporting an authentication device that authenticates an authentication target. The support method includes, for example, an authentication information acquisition step of acquiring authentication information assigned to the authentication target. The support method includes, for example, an unobstructed image acquisition step of acquiring image data of an external image of an object that is a part of the authentication target, in which the object is not obscured by a predetermined type of obscurant. The support method includes, for example, an obscured image acquisition step of acquiring image data of an external image of an object that is a part of the authentication target, in which the object is not obscured by a predetermined type of obscurant. The support method includes, for example, an obscured image detection step of detecting, from the unobscured image, an image of a part that corresponds to an obscured part in the obscured image that is obscured by the obscurant. The support method includes, for example, an unobscured image management step of storing, in a storage device, image data of the image detected in the obscured image detection step and authentication information assigned to the authentication target in association with each other.

[0106] The person to be authenticated 20 may be an example of an authentication target. The face 22 may be an example of an object. The mask 32 may be an example of an obstruction. The mask 32 may be an example of a predetermined type of obstruction. The authentication code 34 may be an example of authentication information. The captured image 40 may be an example of a captured image. The face image 42 may be an example of a captured image. The complement image 44 may be an example of a portion of an unobstructed image. The composite image 50 may be an example of a composite image.

[0107] The personal authentication system 100 may be an example of an authentication device or an assistance device. The imaging device 110 may be an example of a captured image acquisition unit. The face authentication device 120 may be an example of an authentication device or an assistance device. The registration unit 130 may be an example of an unobstructed image acquisition unit, an obstructed image acquisition unit, an obstructed image detection unit, an unobstructed image management unit, an authentication information detection unit, or an assistance device. The storage unit 140 may be an example of an unobstructed image storage unit, an obstructing object storage unit, or a storage device. The complementary information storage unit 142 may be an example of an unobstructed image storage unit or a storage device. The image acquisition unit 150 may be an example of a captured image acquisition unit. The face detection unit 152 may be an example of an input unit. The authentication code acquisition unit 154 may be an example of an authentication information acquisition unit. The complementary image extraction unit 156 may be an example of an authentication information acquisition unit or an extraction unit. The composite image generation unit 160 may be an example of a synthesis unit or an input unit. The face recognition engine 170 may be an example of a first recognition engine.

[0108] The unworn image may be an example of an unobstructed image. The worn image may be an example of an obscured image. The feature amount of the face 22 of the person to be authenticated 20 may be an example of the first feature information. The first database may be an example of an unobstructed image storage unit. An authentication engine that compares the feature amounts of the feature points of the entire face may be an example of the first authentication engine. An authentication engine for a mask may be an example of the second authentication engine. An authentication engine for sunglasses may be an example of the second authentication engine. An object or obscuring item that obscures the user's face may be an example of an obscuring object. An image of the face 22 may be an example of an image of an object included in a captured image.

[0109] [An example of another embodiment] In the present embodiment, the details of one example of the personal authentication system 100 have been described by taking as an example a case in which the personal authentication system 100 executes the authentication process of the person to be authenticated 20 using an image of the face 22 of the person to be authenticated 20. However, the personal authentication system 100 is not limited to this embodiment. For example, the target of the authentication process of the personal authentication system 100 is not limited to a person.

[0110] In another embodiment, the target of the authentication process of the personal authentication system 100 may be an animal other than a human. Furthermore, the authentication process executed by the personal authentication system 100 is not limited to face recognition. In this case, the target of the authentication process of the personal authentication system 100 is not limited to a human or an animal. The target of the authentication process of the personal authentication system 100 may be an article.

[0111] In this embodiment, the details of one example of the personal authentication system 100 have been described by taking as an example a case in which the complementary information storage unit 142 stores one or more authentication codes 34 and image data of one or more complementary images 44 in association with each other, and the composite image generation unit 160 generates the composite image 50 by combining the face image 42 and the complementary image 44 corresponding to the authentication code 34 acquired by the authentication code acquisition unit 154. However, the personal authentication system 100 is not limited to this embodiment.

[0112] In another embodiment, the complementary information storage unit 142 stores one or more authentication codes 34 and image data of one or more non-worn images in association with each other. The composite image generation unit 160 generates a composite image 50 by combining the face image 42 and the non-worn image corresponding to the authentication code 34 acquired by the authentication code acquisition unit 154. A known method may be used to combine two face images into one face image.

[0113] In the present embodiment, the details of an example of the personal authentication system 100 have been described by taking as an example a case in which the composite image generating unit 160 generates the composite image 50 by combining the face image 42 with one or more complementary images 44 corresponding to a single occluding item. However, the personal authentication system 100 is not limited to the present embodiment.

[0114] In another embodiment, the composite image generator 160 generates the composite image 50 by combining the face image 42 with a plurality of complementary images 44 corresponding to a plurality of occluding items. The plurality of complementary images 44 may be complementary images associated with different authentication codes 34. The plurality of complementary images 44 may be complementary images associated with a single authentication code 34.

[0115] For example, the complementary information storage unit 142 stores the authentication code X and image data of a complementary image corresponding to the mask of the user A in association with each other. The complementary information storage unit 142 also stores the authentication code Y and image data of a complementary image corresponding to the contact lenses of the user A in association with each other. When the authentication code acquisition unit 154 acquires the authentication code X and the authentication code Y, the complementary image extraction unit 156 extracts a complementary image corresponding to the mask of the user A as the complementary image 44 corresponding to the authentication code X, and extracts a complementary image corresponding to the contact lenses of the user A as the complementary image 44 corresponding to the authentication code Y. The composite image generation unit 160 generates a composite image 50 including the entire face by combining the face image 42 of the authentication subject 20, the complementary image corresponding to the mask of the user A, and the complementary image corresponding to the contact lenses of the user A.

[0116] If the person to be authenticated 20 is user A, the feature amount determined from the composite image 50 matches the feature amount determined from the image of user A without contact lenses. This allows the personal authentication system 100 to authenticate the person to be authenticated 20 even if the person to be authenticated 20 is wearing a mask and contact lenses.

[0117] For example, the complementary information storage unit 142 stores a single authentication code X, image data of a complementary image corresponding to the mask of the user A, and image data of a complementary image corresponding to the contact lenses of the user A in association with each other. The complementary information storage unit 142 may store a single authentication code X and a user ID of the user A in association with each other, and may store a user ID of the user A, image data of a complementary image corresponding to the mask of the user A, and image data of a complementary image corresponding to the contact lenses of the user A in association with each other. When the authentication code acquisition unit 154 acquires the authentication code X, the complementary image extraction unit 156 extracts a complementary image corresponding to the mask of the user A and a complementary image corresponding to the contact lenses of the user A as the complementary image 44 corresponding to the authentication code X. The composite image generation unit 160 generates a composite image 50 by combining the face image 42, the complementary image corresponding to the mask of the user A, and the complementary image corresponding to the contact lenses of the user A.

[0118] If the person to be authenticated 20 is user A, the feature amount determined from the composite image 50 matches the feature amount determined from the image of user A without contact lenses. This allows the personal authentication system 100 to authenticate the person to be authenticated 20 even if the person to be authenticated 20 is wearing a mask and contact lenses.

[0119] When the composite image generating unit 160 generates the composite image 50 by combining the face image 42 and the multiple complementary images 44, the composite image generating unit 160 may generate multiple composite images 50. The composite image generating unit 160 may determine one of the multiple composite images 50 as the composite image 50 to be output as an input to the face recognition engine 170. The composite image generating unit 160 determines the composite image 50 to be output as an input to the face recognition engine 170 based on, for example, the ratio of the area of ​​the portion derived from the face image 42 or the portion derived from the complementary image 44 to the entire area of ​​the face 22 of the authentication target person 20 in the composite image 50.

[0120] For example, the composite image generating unit 160 outputs image data of the composite image 50 having the smallest ratio of the total area of ​​the portions derived from the plurality of complement images 44 to the entire face area, among the plurality of composite images 50, as an input to the face recognition engine 170. The composite image generating unit 160 may output image data of the composite image 50 having the largest ratio of the area of ​​the portions derived from the face image 42 to the entire face area, among the plurality of composite images 50, as an input to the face recognition engine 170.

[0121] This allows the face recognition engine 170 to execute the authentication process for the authentication target person 20 by using a composite image in which the ratio of the area of ​​the portion derived from the face image 42 to the area of ​​the entire face in the composite image is large. As a result, the authentication accuracy is improved.

[0122] When the composite image generating unit 160 generates the composite image 50 by combining the face image 42 and the multiple complementary images 44 associated with a single authentication code 34, the composite image generating unit 160 may determine whether to generate the composite image 50 based on a combination of the types of the multiple complementary images 44. For example, the storage unit 140 stores information indicating a combination of types of obscuring items for which the generation or output of the composite image 50 is prohibited. The composite image generating unit 160 may, for example, analyze the face image 42 to identify the types of the multiple obscuring items obscuring the face 22 of the person to be authenticated 20. The composite image generating unit 160 may identify the types of the multiple obscuring items based on the authentication code 34. The composite image generating unit 160 determines whether the identified combination of types of obscuring items corresponds to a combination of types of obscuring items for which the generation or output of the composite image 50 is prohibited.

[0123] The storage unit 140 may store information indicating a combination of the types of occluding items for which generation or output of the composite image 50 is permitted. In this case, the composite image generation unit 160 determines whether or not the identified combination of the types of occluding items corresponds to a combination of the types of occluding items for which generation or output of the composite image 50 is permitted.

[0124] For example, when image data of the composite image 50 generated using a complementary image corresponding to a mask and a complementary image corresponding to sunglasses is input to the face recognition engine 170, it becomes relatively difficult to prevent impersonation by a third party who obtains a mask and sunglasses with the authentication code 34. Therefore, for example, when the types of the multiple complementary images are a complementary image corresponding to a mask and a complementary image corresponding to sunglasses, the composite image generating unit 160 stops generating the composite image 50 or stops outputting the composite image 50. This makes it possible to suppress impersonation by a third party.

[0125] In this embodiment, the details of one example of the personal authentication system 100 have been described by taking as an example a case in which the authentication code 34 is printed on the surface of the mask 32 and the authentication code acquisition unit 154 acquires the authentication code 34 by analyzing the face image 42 including the image of the mask 32. However, the personal authentication system 100 is not limited to this embodiment.

[0126] In another embodiment, the authentication code 34 is displayed on the screen of the communication terminal 102 used by the person to be authenticated 20. The imaging device 110 captures an image of the person to be authenticated 20 such that the authentication code 34 displayed on the screen of the communication terminal 102 and the face 22 of the person to be authenticated 20 are included in the angle of view. In yet another embodiment, the authentication code 34 is written or projected on the body of the person to be authenticated 20 or on the surface of an article held by the person to be authenticated 20. The imaging device 110 captures an image of the person to be authenticated 20 such that the authentication code 34 and the face 22 of the person to be authenticated 20 are included in the angle of view.

[0127] Furthermore, in the personal authentication system 100, when multiple authentication codes are stored in association with the user ID of a single user, the multiple authentication codes may be read by the face authentication device 120 in different modes. For example, an authentication code for extracting a complementary image corresponding to a mask is written on the surface of the mask, and an authentication code for extracting a complementary image corresponding to contact lenses is displayed on the screen of the communication terminal 102 used by the person to be authenticated 20.

[0128] In still another embodiment, the authentication code 34 is stored in a storage device of the communication terminal 102 used by the authentication target 20. The authentication code acquisition unit 154 acquires the authentication code 34 from the communication terminal 102 via wired communication or wireless communication. The authentication code acquisition unit 154 may acquire the authentication code 34 from the communication terminal 102 via short-range wireless communication.

[0129] In this embodiment, the details of one example of the personal authentication system 100 have been described by taking as an example a case where the registration information storage unit 144 or the second database stores (i) the user IDs of one or more users and (ii) the unworn images of one or more users in association with each other. However, the personal authentication system 100 is not limited to this embodiment.

[0130] In another embodiment, the registration information storage unit 144 may store (i) the user ID of each of the one or more users and (ii) information indicating the feature amount extracted from the unworn image of each of the one or more users in association with each other. Similarly, the second database may store (i) the user ID of each of the one or more users and (ii) information indicating the feature amount extracted from the unworn image of each of the one or more users in association with each other.

[0131] In the present embodiment, the details of one example of the personal authentication system 100 have been described by taking as an example a case in which the personal authentication system 100 includes the imaging device 110 and the face authentication device 120. However, the personal authentication system 100 is not limited to the present embodiment.

[0132] In another embodiment, the communication terminal 102 used by the person to be authenticated 20 may have some or all of the functions of the personal authentication system 100. The communication terminal 102 used by the person to be authenticated 20 may execute the information processing in the above-described personal authentication system 100, or the communication terminal 102 and the face authentication device 120 may cooperate to execute the information processing in the above-described personal authentication system 100. For example, the camera of the communication terminal 102 used by the person to be authenticated 20 functions as the above-described imaging device 110. Also, the computer of the communication terminal 102 used by the person to be authenticated 20 may function as some or all of the above-described face authentication device 120.

[0133] FIG. 2 shows another example of the system configuration of the personal authentication system 100. The embodiment described with reference to FIG. 2 differs from the embodiment described with reference to FIG. 1 in that the authentication code 34 is not printed on the mask 32, but is stored in a storage device (not shown) of the communication terminal 102 worn or held by the person to be authenticated 20. This embodiment differs from the embodiment described with reference to FIG. 1 in that the authentication code acquisition unit 154 acquires the authentication code 34 not from the imaging device 110 but from the communication terminal 102. The embodiment described with reference to FIG. 2 may have the same configuration as the embodiment described with reference to FIG. 1, except for matters related to the above differences.

[0134] The method by which the authentication code acquisition unit 154 acquires the authentication code 34 from the communication terminal 102 is not particularly limited. In one embodiment, the authentication code acquisition unit 154 accesses the communication terminal 102 by a short-range wireless communication method using electromagnetic waves or sound waves, and acquires the authentication code 34 from the communication terminal 102. In another embodiment, the authentication code acquisition unit 154 accesses the communication terminal 102 by a wired communication method, and acquires the authentication code 34 from the communication terminal 102. In yet another embodiment, the authentication code acquisition unit 154 acquires the authentication code 34 from the communication terminal 102 by reading the authentication code 34 displayed or projected on a display device of the communication terminal 102.

[0135] Fig. 3 shows an example of the internal configuration of the registration unit 130. An example of a process for generating the complementary image 44 will be described with reference to Fig. 3. Also, an example of a process for constructing a database to be stored in the complementary information storage unit 142 and the authentication code acquisition unit 154 will be described with reference to Fig. 3.

[0136] In this embodiment, the registration unit 130 includes a login management unit 320, an authentication code issuing unit 332, and an authentication code detection unit 334. The registration unit 130 includes an unworn image acquisition unit 342, a worn image acquisition unit 344, an obscured portion detection unit 346, a complement image generation unit 348, a registration information management unit 352, and a complement information management unit 354.

[0137] In this embodiment, the login management unit 320 manages the login ID and password of each of one or more users. The login management unit 320 may control the login of a user who has accessed the face authentication device 120.

[0138] In this embodiment, the authentication code issuing unit 332 issues an authentication code 34 assigned to each of the one or more users. The authentication code issuing unit 332 may transmit the authentication code 34 assigned to each of the one or more users to the communication terminal 102 used by each of the one or more users. The authentication code issuing unit 332 may output the user ID of each of the one or more users and the authentication code 34 assigned to each of the one or more users to the complementary information management unit 354 in association with each other.

[0139] According to one embodiment, a single user is assigned multiple authentication codes 34. For example, a first authentication code is assigned to a mask used by user A, and a second authentication code is assigned to sunglasses used by user A. For example, in a registration process by the registration unit 130, the user ID of user A is associated with the first authentication code and the second authentication code, thereby assigning the first authentication code and the second authentication code to user A.

[0140] According to another embodiment, the same authentication code 34 is assigned to multiple users. For example, the same authentication code 34 is assigned to multiple users who belong to the same corporation, organization, or group. For example, the same authentication code 34 is printed on each of the five masks 32 used by each of family X consisting of five members A to E. In this case, the composite image generating unit 160 combines the face image 42 of member A with each of the complement images 44 of members A to E, and outputs the five generated composite images 50 to the face recognition engine 170. In addition, the face recognition engine 170 performs authentication processing of the authentication target person 20 by comparing the feature amount of each of the five composite images 50 generated by the composite image generating unit 160 with the feature amount of one or more unworn images registered in the registration information storage unit 144.

[0141] The authentication code issuing unit 332 may issue an authentication code 34 assigned to each of the one or more masks 32. The authentication code issuing unit 332 may associate identification information (sometimes referred to as an item ID) of each of the one or more masks 32 with the authentication code 34 assigned to each of the one or more masks 32, and transmit the associated information to the communication terminal 102 used by the manufacturer or seller of each of the one or more masks 32. The authentication code issuing unit 332 may associate the item ID of each of the one or more masks 32 with the authentication code 34 assigned to each of the one or more masks 32, and output the associated information to the complementary information management unit 354.

[0142] Multiple masks 32 may be assigned the same authentication code 34. In one embodiment, multiple masks 32 in the same lot are assigned the same authentication code 34. In another embodiment, multiple masks 32 intended for use by the same user are assigned the same authentication code 34.

[0143] In this embodiment, the authentication code detection unit 334 analyzes the image acquired by the wearing image acquisition unit 344 (sometimes referred to as a wearing image 64) and detects the authentication code 34 included in the wearing image 64. The authentication code detection unit 334 may output the detected authentication code 34 to the complementary information management unit 354.

[0144] In this embodiment, the non-worn image acquisition unit 342 acquires image data of the non-worn image 62 of each of one or more users. For example, the non-worn image acquisition unit 342 acquires an image of each user's face not covered by the mask 32 from the communication terminal 102 used by each user. The non-worn image acquisition unit 342 detects an image of each user's face from the above image. The non-worn image acquisition unit 342 generates the non-worn image 62 by cutting out the detected face image from the above image and adjusting the size, color tone, etc. of the cut out image. In this way, the non-worn image acquisition unit 342 can acquire the non-worn image 62.

[0145] In this embodiment, the wearing image acquisition unit 344 acquires image data of an image (sometimes referred to as a wearing image 64) that is an image of the appearance of the face of each of one or more users, in which a part of the face is covered by the mask 32. For example, the wearing image acquisition unit 344 acquires an image of the face of each user covered by the mask 32 from the communication terminal 102 used by each user. The wearing image acquisition unit 344 detects an image of the face of each user from the above image. The wearing image acquisition unit 344 generates the wearing image 64 by cutting out the detected face image from the above image and adjusting the size, color tone, etc. of the cut out image. In this way, the wearing image acquisition unit 344 can acquire the wearing image 64.

[0146] The wearing image acquiring unit 344 may acquire information indicating the type of the article (mask 32 in the above example) covering the face of each user from the communication terminal 102 used by each user. When the authentication code 34 is written on the mask 32, the wearing image acquiring unit 344 may acquire the authentication code 34 written on the mask 32 from the communication terminal 102 used by each user. For example, data of an input screen for inputting at least one of the information indicating the type of the article and the authentication code 34 written on the mask 32 may be transmitted to the communication terminal 102 of each user, and each user may be prompted to input the information.

[0147] In this embodiment, the masked portion detection unit 346 acquires image data of the unworn image 62 of each user from the unworn image acquisition unit 342. The masked portion detection unit 346 also acquires image data of the worn image 64 of each user from the worn image acquisition unit 344. The masked portion detection unit 346 detects an image of a portion corresponding to a portion masked by the mask 32 in the worn image 64 of each user (sometimes referred to as a masked portion) from the unworn image 62 of each user based on the unworn image 62 and worn image 64 of each user. The masked portion detection unit 346 outputs information indicating the detection result to the complement image generation unit 348.

[0148] In this embodiment, the complementary image generating unit 348 generates the complementary image 66 based on the detection result of the occluded portion detecting unit 346. For example, the complementary image generating unit 348 cuts out an image of a portion corresponding to the occluded portion of the worn image 64 of each user from the unworn image 62 of each user based on the detection result of the occluded portion detecting unit 346. The complementary image generating unit 348 generates the complementary image 66 of each user by adjusting the size, color tone, etc. of the cut out image. The complementary image generating unit 348 outputs image data of the generated complementary image 66 to the complementary information managing unit 354.

[0149] In this embodiment, the registration information management unit 352 associates the image data of the unworn image 62 of each user acquired by the unworn image acquisition unit 342 with the user ID of each user and stores the image data in the registration information storage unit 144. The unworn image 62 stored in the registration information storage unit 144 may be treated as a registered image of each user in the authentication process of the face recognition engine 170, for example.

[0150] In this embodiment, the complementary information management unit 354 stores at least one of the image data of the unworn image 62 of each user acquired by the unworn image acquisition unit 342 and the complementary image 66 of each user generated by the complementary image generation unit 348 in the complementary information storage unit 142 in association with the authentication code 34 assigned to each user. The complementary information management unit 354 may store at least one of the image data of the unworn image 62 of each user acquired by the unworn image acquisition unit 342 and the complementary image 66 of each user generated by the complementary image generation unit 348 in association with at least one of the authentication code 34 assigned to each user and the user ID of each user in the complementary information storage unit 142. The authentication code 34 assigned to each user may be the authentication code 34 issued by the authentication code issuing unit 332 or may be the authentication code 34 detected by the authentication code detection unit 334.

[0151] In this embodiment, the complementary information management unit 354 may use the complementary image 66 of the user who wishes to register to confirm whether or not the face authentication device 120 can authenticate the user before storing the image data of the complementary image 66 generated by the complementary image generation unit 348 in the complementary information storage unit 142. If it is confirmed that the user can be authenticated, the complementary information management unit 354 stores the image data of the complementary image 66 generated by the complementary image generation unit 348 in the complementary information storage unit 142. If it is not confirmed that the user can be authenticated, the complementary information management unit 354 may present a message to the user indicating that registration has failed. This can prompt the user to register again. Examples of the message presentation mode include a mode in which a message is displayed on the screen of the communication terminal 102, and a mode in which a message is output from an output device (not shown) of the face authentication device 120.

[0152] In one embodiment, the complementary information management unit 354 uses, for example, the composite image generation unit 160 to composite the worn image 64 and the complementary image 66 to generate the composite image 50. Next, the complementary information management unit 354 checks whether the face recognition engine 170 can authenticate the user who wishes to be registered by comparing the feature amount of the composite image 50 with the feature amount of the unworn image 62.

[0153] In another embodiment, the complementary information management unit 354 uses the imaging device 110 to capture an image of the user again. The complementary information management unit 354 uses the composite image generation unit 160 to composite the image captured by the imaging device 110 and the complementary image 66 to generate a composite image 50. Next, the complementary information management unit 354 checks whether the face recognition engine 170 can authenticate the user who wishes to be registered by comparing the feature amount of the composite image 50 with the feature amount of the unworn image 62.

[0154] The unworn image 62 may be an example of an unobstructed image. The worn image 64 may be an example of an obscured image. The complement image 66 may be an example of an image of a portion corresponding to an obscured portion. The complement image 66 may be an example of an image detected by an obscured image detection unit. The authentication code detection unit 334 may be an example of an authentication information detection unit. The unworn image acquisition unit 342 may be an example of an unobscured image acquisition unit. The worn image acquisition unit 344 may be an example of a obscured image acquisition unit. The obscured portion detection unit 346 may be an example of an obscured image detection unit.

[0155] 4 shows an example of an internal configuration of the storage section 140. In this embodiment, the storage section 140 includes a complementary information storage section 142, a registration information storage section 144, an item information storage section 442, and a setting information storage section 444.

[0156] In this embodiment, the complementary information storage unit 142 stores various information related to one or more complementary images. For each of one or more complementary images 66 related to one or more users, the complementary information storage unit 142 may store the user ID of each user, a complementary image ID for identifying each complementary image, an authentication code 34 associated with each complementary image by the complementary information management unit 354, information indicating the expiration date of the authentication code 34, identification information of a registered image corresponding to each complementary image (sometimes referred to as a registered image ID), and a URI of each complementary image in association with each other.

[0157] In this embodiment, the registration information storage unit 144 stores various information related to one or more registered images. For each of one or more registered images (i.e., unattached images 62) related to one or more users, the registration information storage unit 144 may store the user ID of each user, a registered image ID for identifying each registered image, information indicating the registration date and time of each registered image, information indicating the attachment state of the shielding item in each registered image, and identification information of a complementary image corresponding to each registered image, in association with each other.

[0158] In this embodiment, the item information storage unit 442 stores various information related to one or more shielding items. For each of the one or more shielding items, the item information storage unit 442 stores the authentication code 34 written on each shielding item, identification information for identifying each shielding item (sometimes referred to as an item ID), information indicating the type of each shielding item, information indicating detailed specifications of each shielding item, and information indicating the wearing position or wearing range of each shielding item when worn by a user, in association with each other.

[0159] In this embodiment, the setting information storage unit 444 stores various information related to the settings of the personal authentication system 100. For example, the setting information storage unit 444 stores information indicating a setting item and information indicating a setting content for each of one or more settings in association with each other.

[0160] 5 shows an example of a system configuration of a personal authentication system 500. The personal authentication system 500 differs from the personal authentication system 100 in that some functions of the face authentication device 120 are executed by the communication terminal 102, and other functions of the face authentication device 120 are executed by the management server 502. In this embodiment, the communication terminal 102 and the management server 502 can transmit and receive information to and from each other via the communication network 10. The personal authentication system 500 may have a similar configuration to the personal authentication system 100, except for matters related to the above differences.

[0161] According to this embodiment, the communication terminal 102 includes an imaging device 110, an image acquisition unit 150, a face detection unit 152, an authentication code acquisition unit 154, a complemented image extraction unit 156, a composite image generation unit 160, and a face authentication engine 170. The management server 502 includes registration units 130 and 140. It should be noted that the division of functions between the communication terminal 102 and the management server 502 is not limited to this embodiment.

[0162] 6 shows an example of an authentication processing method in the personal authentication system 100 or 500. According to this embodiment, first, in step 610 (step may be abbreviated as S), for example, the image acquisition unit 150 acquires image data of the captured image 40. Next, in S612, for example, the authentication code acquisition unit 154 analyzes the captured image 40 to detect a QR code (registered trademark) in which the authentication code 34 is embedded. Also, in S614, the complementary image extraction unit 156 refers to the complementary information storage unit 142 using the authentication code 34 indicated by the detected QR code (registered trademark) as a key, and acquires image data of the complementary image 44 associated with the authentication code 34.

[0163] Next, in S616, the composite image generating unit 160 generates a composite image 50 by combining the face image 42 included in the captured image 40 with the complemented image 44. In addition, in S618, the face recognition engine 170 executes face recognition processing using the composite image 50. Thereafter, the face recognition engine 170 outputs the result of the face recognition processing. This completes the authentication processing of the person to be authenticated 20.

[0164] Another example of the authentication processing method in the personal authentication system 100 or the personal authentication system 500 will be described with reference to Fig. 7, Fig. 8, and Fig. 9. Fig. 7 shows an outline of another example of the authentication processing method. Fig. 8 shows an outline of an example of the internal configuration of a face authentication device 820, which is another example of the face authentication device 120. Fig. 9 shows an outline of an example of information processing in the face authentication device 820.

[0165] Fig. 7 shows an outline of another example of the authentication processing method in the personal authentication system 100 or the personal authentication system 500. According to this embodiment, first, the step of S610 is executed in the same manner as in the authentication processing method shown in Fig. 6. Then, in S720, the steps of S612 to S616 are executed in the same manner as in the authentication processing method shown in Fig. 6, and a composite image 50 is generated.

[0166] Next, in S732, an exposed portion of the face 22 of the person to be authenticated 20 is detected from the face image 42. In addition, in S734, image data of the image of the exposed portion of the face 22 of the person to be authenticated 20 is input to an authentication engine different from the face authentication engine 170. Thereby, a first face authentication process is executed.

[0167] In this embodiment, steps S732 and S734 are executed in parallel with step S720. However, the order in which steps S720 and steps S732 and S734 are executed is not limited to this embodiment.

[0168] Next, in S740, the second face recognition process is executed by, for example, the face recognition engine 170, using one or more registered images determined to have a high degree of similarity in the first face recognition process in step S734 and the composite image 50 generated in step S720. Thereafter, the face recognition engine 170 outputs the result of the face recognition process. This ends the authentication process of the person to be authenticated 20.

[0169] According to this embodiment, after candidates for the person to be authenticated 20 are narrowed down in the first face authentication process, the registered image of the candidate is compared with the composite image 50 in the second face authentication process. As a result, it may be easier to adjust the threshold value for determination. Also, the amount of calculation in the authentication process may be reduced.

[0170] FIG. 8 illustrates an example of the internal configuration of a face authentication device 820. The face authentication device 820 is another example of the face authentication device 120, and is configured to be able to execute the authentication processing method described in relation to FIG. 7. In this embodiment, the face authentication device 820 differs from the face authentication device 120 in that it includes an exposed portion detection unit 822 and a partial authentication engine 824. The face authentication device 820 may include multiple types of partial authentication engines 824. The face authentication device 820 may have the same configuration as the face authentication device 120, except for matters related to the above differences.

[0171] In this embodiment, the exposed portion detection unit 822 detects an image of an exposed portion, which is a portion of the face 22 of the person to be authenticated 20 that is not covered by the mask 32, from the face image 42. For example, the exposed portion detection unit 822 detects an image of the upper half of the face 22 of the person to be authenticated 20 as the image of the exposed portion.

[0172] Moreover, the exposed portion detection unit 822 inputs image data of the detected image to the partial authentication engine 824. As a result, face authentication processing using a part of the face 22 is executed.

[0173] In this embodiment, the partial authentication engine 824 executes the first face authentication process described in relation to Fig. 7. The partial authentication engine 824 may be an authentication engine that authenticates an authentication target using an algorithm different from that of the face authentication engine 170. The partial authentication engine 824 may be an authentication engine that executes authentication processing based on the external appearance feature amount of a predetermined part of the face 22.

[0174] When image data of an image detected by the exposed portion detection unit 822 is input, the partial authentication engine 824 executes authentication processing based on the external characteristics of the exposed portion of the face 22. In this embodiment, the partial authentication engine 824 outputs, as a result of the authentication processing, information indicating one or more candidates that are relatively likely to be the authentication target person 20 to the face authentication engine 170.

[0175] In this embodiment, the face recognition engine 170 executes the authentication process for the person to be authenticated 20 by comparing the composite image 50 of the person to be authenticated 20 generated by the composite image generation unit 160 with each of the registered images of one or more candidates output by the partial recognition engine 824. For example, the face recognition engine 170 determines whether or not the subject of the composite image 50 of the person to be authenticated 20 generated by the composite image generation unit 160 matches the feature amount of each of the registered images of one or more candidates, thereby determining whether or not to authenticate the person to be authenticated 20.

[0176] The exposed portion detection section 822 may be an example of an exposed image detection section. The partial authentication engine 824 may be an example of a second authentication engine. The feature amount of the appearance of a predetermined part of the face 22 may be an example of second feature information.

[0177] Fig. 9 shows an example of a registration processing method in the registration unit 130. In this embodiment, the details of the registration processing method will be described using as an example a case where a user who wishes to register executes the registration processing using the face authentication device 120 described in relation to Fig. 1. Also, in this embodiment, the details of the registration processing method will be described using as an example a case where a QR code (registered trademark) with an authentication code 34 embedded therein is printed on the mask 32.

[0178] According to this embodiment, first, in S912, the imaging device 110 captures an image of a user not wearing a mask 32. The imaging device 110 captures an image of the user's face from a specific direction. In S914, the non-wearing image acquisition unit 342 acquires image data of an image of the user not wearing a mask 32 from the imaging device 110. As a result, a non-wearing image 62 is acquired.

[0179] In addition, in S922, the imaging device 110 captures an image of the user wearing the mask 32. The imaging device 110 captures an image of the user's face from a specific direction. In S924, the wearing image acquisition unit 344 acquires image data of an image of the user wearing the mask 32 from the imaging device 110. As a result, the wearing image 64 is acquired. Note that S912 and S914, and S922 and S924 may be executed in parallel or sequentially.

[0180] Next, in S932, the authentication code detection unit 334 analyzes the wearing image 64 to detect the authentication code 34. In addition, in S942, the complement image generation unit 348 generates a complement image 66. Note that S932 and S942 may be executed in parallel or sequentially.

[0181] Specifically, in S942, first, the occluded portion detection unit 346 compares the unworn image 62 and the worn image 64, and detects an image of a portion corresponding to "a portion occluded by the mask 32 in the worn image 64" from the unworn image 62. For example, the occluded portion detection unit 346 detects that the periphery of the user's mouth is occluded by the mask 32 in the worn image 64. Therefore, the occluded portion detection unit 346 detects an image of a portion corresponding to the periphery of the user's mouth in the unworn image 62. Thereafter, the complementary image generation unit 348 generates a complementary image 66 based on the detection result of the occluded portion detection unit 346.

[0182] Next, in S950, the complementary information management unit 354 executes a confirmation process. The complementary information management unit 354 confirms whether or not the user can be authenticated using the complementary image 66 generated by the complementary image generation unit 348. If it is confirmed that the user can be authenticated, in S960, the complementary information management unit 354 stores the complementary image 66 in the complementary information storage unit 142. The complementary information management unit 354 stores, for example, the user ID and the complementary image 66 in the complementary information storage unit 142 in association with each other.

[0183] Furthermore, in S960, the registration information management unit 352 may store the unattached image 62 in the complementary information storage unit 142. The registration information management unit 352 may, for example, associate the user ID with the unattached image 62 and store them in the registration information storage unit 144. Note that, in S950, if the complementary information management unit 354 is unable to confirm that the user can be authenticated using the complementary image 66, the complementary information management unit 354 may present the user with a message indicating that registration has failed.

[0184] 10 shows an example of a computer 3000 in which aspects of the present invention may be embodied in whole or in part. At least a part of the personal authentication system 100 may be realized by the computer 3000. For example, at least a part of the imaging device 110 may be realized by the computer 3000. For example, at least a part of the face authentication device 120 may be realized by the computer 3000. Also, at least a part of the communication terminal 102 may be realized by the computer 3000.

[0185] A program installed on the computer 3000 may cause the computer 3000 to function as or perform operations associated with an apparatus according to an embodiment of the present invention or one or more "parts" of the apparatus, and / or to perform a process or steps of the process according to an embodiment of the present invention. Such a program may be executed by the CPU 3012 to cause the computer 3000 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0186] The computer 3000 according to this embodiment includes a CPU 3012, a RAM 3014, a GPU 3016, and a display device 3018, which are connected to each other by a host controller 3010. The computer 3000 also includes input / output units such as a communication interface 3022, a hard disk drive 3024, a DVD-ROM drive 3026, and an IC card drive, which are connected to the host controller 3010 via an input / output controller 3020. The computer also includes legacy input / output units such as a ROM 3030 and a keyboard 3042, which are connected to the input / output controller 3020 via an input / output chip 3040.

[0187] The CPU 3012 operates according to a program stored in the ROM 3030 and the RAM 3014, thereby controlling each unit. The GPU 3016 acquires image data generated by the CPU 3012 into a frame buffer or the like provided in the RAM 3014 or into the GPU 3016 itself, and causes the image data to be displayed on the display device 3018.

[0188] The communications interface 3022 communicates with other electronic devices via a network. The hard disk drive 3024 stores programs and data used by the CPU 3012 in the computer 3000. The DVD-ROM drive 3026 reads programs or data from the DVD-ROM 3001 and provides the programs or data to the hard disk drive 3024 via the RAM 3014. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0189] The ROM 3030 stores therein a boot program or the like executed by the computer 3000 upon activation, and / or a program that depends on the hardware of the computer 3000. The input / output chip 3040 may also connect various input / output units to the input / output controller 3020 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0190] The programs are provided by a computer-readable storage medium such as a DVD-ROM 3001 or an IC card. The programs are read from the computer-readable storage medium, installed in the hard disk drive 3024, the RAM 3014, or the ROM 3030, which are also examples of computer-readable storage media, and executed by the CPU 3012. Information processing described in these programs is read by the computer 3000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing an operation or processing of information according to the use of the computer 3000.

[0191] For example, when communication is performed between computer 3000 and an external device, CPU 3012 may execute a communication program loaded in RAM 3014 and instruct communication interface 3022 to perform communication processing based on the processing described in the communication program. Under the control of CPU 3012, communication interface 3022 reads transmission data stored in a transmission buffer area provided in RAM 3014, hard disk drive 3024, DVD-ROM 3001, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0192] Furthermore, CPU 3012 may cause all or a necessary portion of a file or database stored in an external recording medium such as hard disk drive 3024, DVD-ROM drive 3026 (DVD-ROM 3001), an IC card, etc. to be read into RAM 3014, and perform various types of processing on the data on RAM 3014. CPU 3012 may then write back the processed data to the external recording medium.

[0193] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 3012 may perform various types of processing on the data read from the RAM 3014, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 3014. The CPU 3012 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 3012 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0194] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 3000 or in the vicinity of the computer 3000. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the above-described programs to the computer 3000 via the network.

[0195] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiments. In addition, the matters described for a specific embodiment can be applied to other embodiments to the extent that they are not technically inconsistent. It is clear from the claims that such modified or improved forms can also be included in the technical scope of the present invention.

[0196] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0197] 10 communication network, 20 person to be authenticated, 22 face, 32 mask, 34 authentication code, 40 captured image, 42 face image, 44 complement image, 50 composite image, 62 unworn image, 64 worn image, 66 complement image, 100 personal authentication system, 102 communication terminal, 110 imaging device, 120 face authentication device, 130 registration unit, 140 storage unit, 142 complement information storage unit, 144 registration information storage unit, 150 image acquisition unit, 152 face detection unit, 154 authentication code acquisition unit, 156 complement image extraction unit, 160 composite image generation unit, 170 face authentication engine, 320 login management unit, 332 authentication code issuing unit, 334 authentication code detection unit, 342 unworn image acquisition unit, 344 worn image acquisition unit, 346 covered portion detection unit, 348 Complementary image generating unit, 352 registration information management unit, 354 complementary information management unit, 442 item information storage unit, 444 setting information storage unit, 500 personal authentication system, 502 management server, 820 face authentication device, 822 exposed part detection unit, 824 partial authentication engine, 3000 computer, 3001 DVD-ROM, 3010 host controller, 3012 CPU, 3014 RAM, 3016 GPU, 3018 display device, 3020 input / output controller, 3022 communication interface, 3024 hard disk drive, 3026 DVD-ROM drive, 3030 ROM, 3040 input / output chip, 3042 keyboard

Claims

1. 1. An authentication device that authenticates an authentication target based on (i) first feature information indicating an external feature of an object that is a part of the authentication target, and (ii) authentication information that is information different from the first feature information and that is assigned to the authentication target, comprising: an input unit that inputs image data to be subjected to the authentication process to a first authentication engine that executes the authentication process based on the first characteristic information; a captured image acquisition unit that acquires image data of a captured image of the object to be authenticated; an authentication information acquisition unit that acquires the authentication information assigned to the authentication target; an extracting unit that refers to (i) the authentication information assigned to each of one or more authentication targets, and (ii) an unobstructed image storing unit that stores an unobstructed image in which the target is not obstructed by a predetermined type of obstruction and / or image data of a portion of the unobstructed image in association with the authentication information acquired by the authentication information acquiring unit, and extracts image data of the unobstructed image or a portion of the unobstructed image associated with the authentication information acquired by the authentication information acquiring unit; a synthesis unit that synthesizes the captured image with the unoccluded image or a part of the unoccluded image extracted by the extraction unit to generate a synthetic image; Equipped with the input unit inputs image data of the composite image generated by the composition unit to the first authentication engine when the authentication information acquisition unit acquires the authentication information. Authentication device.

2. the input unit inputs image data of the captured image or a part of the captured image to the first authentication engine when the captured image acquisition unit acquires image data of the captured image and the authentication information acquisition unit has not acquired the authentication information. The authentication device according to claim 1 .

3. The authentication information is written, displayed or projected onto an article attached to or held by the authentication target, The authentication information acquisition unit acquires the authentication information assigned to the authentication target by analyzing image data of the captured image.

3. The authentication device according to claim 1 or 2.

4. the authentication information is stored in a storage device of a communication terminal attached to or held by the authentication target, The authentication information acquisition unit acquires the authentication information assigned to the authentication target from the communication terminal. The authentication device according to claim 1 or 2.

5. The synthesis unit generates the synthesis image by referring to a shielding object storage unit that stores authentication information assigned to the authentication target and information indicating the type of the shielding object in association with each other. The authentication device according to any one of claims 1 to 4.

6. an exposed image detection unit that detects an image of an exposed portion of the object that is not covered by the covering object from an image of the object included in the captured image; a second authentication engine that authenticates the authentication object using an algorithm different from that of the first authentication engine and executes authentication processing based on second feature information that indicates an external feature of a predetermined part of the object; Further equipped with the second authentication engine, when image data of the image of the exposed portion detected by the exposed image detection unit is input, performs an authentication process based on an external characteristic of the exposed portion, and outputs information indicating one or more candidates for the authentication target; the first authentication engine determines whether or not the first feature information of the subject of the image data input by the input unit matches the first feature information of each of the one or more candidates, thereby determining whether or not to authenticate the authentication target. The authentication device according to any one of claims 1 to 5.

7. an unobstructed image acquisition unit that acquires image data of the unobstructed image of each of the one or more authentication targets; an obstructed image acquisition unit that acquires image data of an external appearance of each of the one or more authentication targets, the obstructed image being an image of an external appearance of each of the one or more authentication targets, in which a part of the target is obstructed by the obstructing object of the predetermined type; an obscured image detection unit that detects, from the unobstructed image of each of the one or more authentication targets, an image of a portion corresponding to a masked portion that is masked by the masking object in the masked image of each of the one or more authentication targets; an unobstructed image management unit that stores, in the unobstructed image storage unit, image data of at least one of the unobstructed image acquired by the unobstructed image acquisition unit and the image detected by the obscured image detection unit in association with the authentication information assigned to each of the one or more authentication targets; Further comprising: The authentication device according to any one of claims 1 to 6.

8. An authentication information detection unit that analyzes image data of the masked image and detects the authentication information included in the masked image, the unobstructed image management unit stores, in the unobstructed image storage unit, at least one of image data of the unobstructed image acquired by the unobstructed image acquisition unit and the image detected by the obscured image detection unit in association with the authentication information detected by the authentication information detection unit. The authentication device according to claim 7.

9. A program for causing a computer to function as the authentication device according to any one of claims 1 to 8.

10. A program for causing a computer to function as an assistance device that assists an authentication device that authenticates an authentication target, comprising: The support device includes: an authentication information acquisition unit that acquires authentication information assigned to the authentication target; an unobstructed image acquisition unit that acquires image data of an external image of an object that is a part of the authentication target, the image data being an unobstructed image in which the object is not obstructed by a predetermined type of obstruction; an obstructed image acquisition unit that acquires image data of an external appearance of the object to be authenticated, the image data being an obstructed image in which a part of the object is obstructed by the obstructing object of the predetermined type; an obscured image detection unit that detects, from the unobstructed image, an image of a portion of the obscured image that corresponds to a portion obscured by the obscuring object; an unobstructed image management unit that stores image data of the image detected by the obscured image detection unit and the authentication information assigned to the authentication target in a storage device in association with each other; Equipped with program.

11. 1. An authentication method for authenticating an authentication target based on (i) first feature information indicating an external feature of an object that is a part of the authentication target, and (ii) authentication information that is information different from the first feature information and that is assigned to the authentication target, comprising: an input step of inputting image data to be a target of the authentication process into a first authentication engine that executes an authentication process based on the first characteristic information; a captured image acquisition step of acquiring image data of a captured image of the object to be authenticated; acquiring the authentication information assigned to the authentication target; an extraction step of extracting image data of the unobstructed image or a part of the unobstructed image associated with the authentication information acquired in the authentication information acquisition step, by referring to (i) the authentication information assigned to each of one or more authentication targets, and (ii) an unobstructed image storage device that stores, in association with each other, an image of the appearance of each of the one or more authentication targets, an unobstructed image in which the target is not obstructed by a predetermined type of obstruction and / or image data of a part of the unobstructed image; a synthesis step of synthesizing the captured image and the unoccluded image or a part of the unoccluded image extracted in the extraction step to generate a synthetic image; having the input step includes a step of inputting image data of the composite image generated in the synthesis step into the first authentication engine when the authentication information is acquired in the authentication information acquisition step. Authentication method.

12. 1. A method for supporting an authentication device that authenticates an authentication target, comprising: acquiring authentication information assigned to the authentication target; an unobstructed image acquisition step of acquiring image data of an external image of an object that is a part of the authentication object, the image data being an unobstructed image in which the object is not obscured by a predetermined type of obstruction; a blocked image acquisition step of acquiring image data of an external image of the object to be authenticated, the image data being a blocked image in which a part of the object is blocked by the predetermined type of blocking object; a masked image detection step of detecting, from the unmasked image, an image of a portion of the masked image that corresponds to a masked portion of the masked image that is masked by the masking object; an unobstructed image management step of storing image data of the image detected in the obscured image detection step and the authentication information assigned to the authentication target in a storage device in association with each other; A method of support having the above.

Images