Information processing systems, facial recognition systems, and programs

Abstract

When a user's face cannot be extracted from a captured image, estimating the user using related information about the user improves the accuracy of user estimation compared to when the related information is not updated. [Solution] Camera 10 is positioned so that any user attempting to use any of the facilities 4, 6, or 8 is within its shooting range. The information processing device 20 has a specific level calculation unit 2064 that calculates a specific level for each candidate of a user by referring to related information registered in association with each facility user if the user's face recognition is unsuccessful using the image captured by camera 10, and an inquiry unit 2066 that queries the candidates if none of the candidates' specific levels meet the standard value, and identifies the user by referring to the query results. Related information update unit 208 updates the user's related information with information extracted from the user's captured image.

Description

Technical Field

[0001] The present invention relates to an information processing system, a face authentication system, and a program.

Background Art

[0002] Recently, as information used for user authentication, not only knowledge information such as a combination of a user ID and a password memorized by a user but also biometric information such as a face and a fingerprint may be used. For example, in face authentication, user authentication is performed by comparing a captured image of a user's face by a camera installed at a predetermined position with a pre-registered face image.

[0003] In Patent Document 1, a system has been proposed in which related information is also captured together with the face of the person to be authenticated, and authentication is performed using the related information when the face is not visible. The related information is an image of an object carried by the user such as clothing and a bag, or a person around the user, which has been captured together with the face image in the past. The related information is detected together when the detection of the face image from the captured image is successful, and is stored in association with the face image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The related information is to be used when a face image that can be used for face authentication cannot be detected from the captured image. However, if the related information is not generated with sufficient accuracy to be used for user authentication based on the related information, an incorrect authentication result will be obtained.

[0006] The present invention aims to improve the accuracy of user estimation when, in cases where a user's face image cannot be extracted from a captured image, the user is estimated using related information associated with the user, compared to when the related information is not updated. [Means for solving the problem]

[0007] The information processing system according to the present invention comprises a processor, wherein, if the facial image of a user intending to use the equipment cannot be extracted from the captured image, the processor extracts information related to the user from the captured image, obtains related information related to a user already registered as an eligible user of the equipment, selects candidates for the actual user in the photograph by referring to the degree of agreement between the information extracted from the captured image and the related information, calculates an identification level for each candidate, which is an indicator of whether the person in the photograph has been correctly identified from the related information, and if none of the calculated identification levels reach a predetermined threshold, the processor queries the candidates, estimates the user in the photograph by referring to the query results from the candidates, and updates the related information related to the estimated user with the information extracted from the captured image.

[0008] Furthermore, the processor is characterized in that, if the user has specified information in the query results from the candidate, it sets that information as related information associated with that user.

[0009] Furthermore, the processor is characterized by querying all of the candidates.

[0010] Furthermore, the processor is characterized by determining a method for querying the candidates in accordance with predetermined rules.

[0011] Furthermore, the processor is characterized in that it determines the order in which to query the candidates according to a specific level calculated for each candidate.

[0012] Furthermore, the processor is characterized by querying all candidates at once.

[0013] Furthermore, the processor is characterized in that if any of the users included in the candidates are using any of the multiple pieces of equipment managed by the information processing system, the processor will remove that user from the list of candidates.

[0014] Furthermore, the processor is characterized in that it calculates the specific level by referring to the degree of overlap in the settings of related information associated with the registered user.

[0015] The facial recognition system according to the present invention comprises the information processing system described above and a number of shooting means less than the number of devices that require facial recognition when use is initiated, and is characterized by performing facial recognition using a user's facial image extracted from a captured image acquired from any of the shooting means.

[0016] The program according to the present invention enables a computer to perform the following functions: extract information related to a user from a captured image if the facial image of a user attempting to use the equipment cannot be extracted from the captured image; acquire related information related to a user already registered as an eligible user of the equipment; select candidates for the actual user who was photographed by referring to the degree of agreement between the information extracted from the captured image and the related information; calculate an identification level for each candidate, which is an indicator of whether the person actually photographed can be correctly identified from the related information of the user; if none of the calculated identification levels reach a predetermined threshold, query the candidates; estimate the photographed user by referring to the query results from the candidates; and update related information related to the estimated user using the information extracted from the captured image. [Effects of the Invention]

[0017] According to the invention described in claim 1, when a user's face image cannot be extracted from a captured image, estimating the captured user using related information associated with the user can improve the accuracy of user estimation compared to when the related information is not updated.

[0018] According to the invention described in claim 2, the information specified by the user can be reflected in the related information.

[0019] According to the invention described in claim 3, without being limited to the user who has actually been photographed, the information specified by each of all candidates can be reflected in the related information of the user.

[0020] According to the invention described in claim 4, an inquiry can be made in a desired manner.

[0021] According to the invention described in claim 5, inquiries can be made in descending order of a specific level.

[0022] According to the invention described in claim 6, the photographed user can be estimated earlier.

[0023] According to the invention described in claim 7, users who cannot be candidates because they are using other equipment can be excluded from the inquiry targets.

[0024] According to the invention described in claim 8, a user who matches the information extracted from the photographed image and has a lower degree of overlap in the setting content of the related information with other users can be estimated as the photographed user with a high probability.

[0025] According to the invention described in claim 9, when the face image of the user cannot be extracted from the photographed image, when the photographed user is estimated using the related information related to the user, the estimation accuracy of the user can be improved compared to the case where the related information is not updated.

[0026] According to the invention described in claim 10, when the face image of the user cannot be extracted from the photographed image, when the photographed user is estimated using the related information related to the user, the estimation accuracy of the user can be improved compared to the case where the related information is not updated. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] [Figure 1] This is a schematic floor plan showing the interior of a room containing equipment where user authentication is performed by the information processing system in this embodiment. [Figure 2] This is a hardware configuration diagram of the computer forming the information processing device in this embodiment. [Figure 3] This figure shows an example of the block configuration of the information processing device in this embodiment. [Figure 4] This figure shows an example of the data structure of authentication information in this embodiment. [Figure 5] This figure shows an example of the data structure of related information in this embodiment. [Figure 6] This figure shows an example of the data structure of equipment information in this embodiment. [Figure 7] This figure shows an example of the data structure of camera information in this embodiment. [Figure 8A] This is a flowchart showing the user authentication process in this embodiment. [Figure 8B] This is a flowchart following Figure 8A. [Modes for carrying out the invention]

[0028] Hereinafter, preferred embodiments of the present invention will be described based on the drawings.

[0029] Figure 1 is a schematic plan view showing the interior of Room 2, where a facial recognition system utilizing the information processing system according to the present invention is installed. Users must successfully perform facial recognition in order to begin using the equipment in Room 2. The equipment where facial recognition is performed is shown in Figure 1 as a large conference room 4a and two small conference rooms 4b and 4c. When there is no need to distinguish between the large conference room 4a and the small conference rooms 4b and 4c, they are collectively referred to as "Conference Room 4". In addition, Room 2 is equipped with two multifunction printers 6 and eight personal computers (PCs) 8. Each PC 8 is placed on a desk. Furthermore, multiple cameras 10a to 10d are installed in Room 2 as means of capturing images. When there is no need to distinguish between the cameras 10a to 10d, they are collectively referred to as "Camera 10".

[0030] In this embodiment, for security reasons, user 12 can only begin using equipment 4, 6, and 8 after successfully authenticating themselves. In this embodiment, multi-factor authentication is performed, and the multiple elements used to achieve multi-factor authentication are knowledge information consisting of a user ID and password pair, and biometric information from a facial image. In this embodiment, it is assumed that a facial image is used as biometric information when achieving multi-factor authentication, but it is not necessary to limit it to only a facial image, and authentication may be performed in combination with other biometric information.

[0031] In this embodiment, in order to perform facial recognition, camera 10 is installed in a position where it can capture users who wish to use each of the devices 4, 6, and 8. In Figure 1, the shooting ranges 14a to 14d of each camera 10a to 10d are shown by dashed lines. When there is no need to distinguish between the shooting ranges 14a to 14d, they are collectively referred to as "shooting range 14".

[0032] As illustrated in Figure 1, in this embodiment, the cameras 10 are not installed in correspondence to each of the 4, 6, and 8 pieces of equipment. If cameras 10 were prepared in the same number as the 4, 6, and 8 pieces of equipment and installed in correspondence to each piece of equipment 4, 6, and 8, it might be possible to capture the user's face from the front. This would make it possible to improve the accuracy of facial recognition. However, preparing as many cameras 10 as there are pieces of equipment 4, 6, and 8 might incur enormous costs.

[0033] Therefore, in this embodiment, fewer cameras 10 than the total number of equipment 4, 6, and 8 are installed in Room 2. When using any of the equipment 4, 6, or 8 installed in Room 2, it is assumed that the user will basically stand still in front of the equipment (e.g., Conference Room 4 or Multifunction Printer 6) or sit down in front of the equipment (e.g., PC 8) when starting to use it. In this embodiment, it is sufficient to install each camera 10 in Room 2 so that the user in this posture can be captured within the shooting range 14 of at least one camera 10, and the number of cameras 10 to be installed, mounting positions, field of view, performance, etc. can be determined as appropriate.

[0034] The information processing device 20 is an example of an information processing system. The information processing device 20 is directly or indirectly connected to the camera 10 by wired (not shown) or wireless means, and acquires the captured images generated by the camera 10 taking pictures. The camera 10 may be installed as a component of the information processing system, or a camera or part of a camera already installed in another system, such as an access control system, may be used.

[0035] Figure 2 is a hardware configuration diagram of the computer forming the information processing device 20 in this embodiment. The information processing device 20 in this embodiment can be realized with a conventional general-purpose hardware configuration such as a PC. Specifically, the information processing device 20 has a CPU 21, ROM 22, RAM 23, a hard disk drive (HDD) 24 as a storage means, a user interface (UI) 25 including input means such as a mouse and keyboard and display means such as a display, and a network interface (IF) 26 provided as a communication means. The network interface 26 receives captured image data output by the camera 10.

[0036] Figure 3 shows an example of the block configuration of the information processing device 20 in this embodiment. The information processing device 20 includes a captured image acquisition unit 202, an image processing unit 204, an authentication processing unit 206, a related information update unit 208, an authentication related information storage unit 222, an equipment information storage unit 224, and a camera information storage unit 226. Components not used in the description of this embodiment are omitted from the figure.

[0037] The image acquisition unit 202 acquires images captured by the camera 10. The image processing unit 204 performs image processing on the acquired images using the following components.

[0038] The image processing unit 204 includes an authentication activity detection unit 2042, a face image extraction unit 2044, and a related information extraction unit 2046. The authentication activity detection unit 2042 detects a predetermined authentication activity by the user by analyzing the captured image. The "predetermined authentication activity" will be described later. The face image extraction unit 2044 extracts the user's face image by analyzing the captured image. Existing technologies may be used for face image extraction. The related information extraction unit 2046 extracts related information related to the user by analyzing the captured image. The "related information" will be described later.

[0039] The authentication processing unit 206 performs user authentication processing based on images captured by the camera 10. In this embodiment, the user is authenticated using the user's facial image and, if necessary, related user information. The authentication processing unit 206 includes a facial recognition unit 2062, a specific level calculation unit 2064, an inquiry unit 2066, and a notification unit 2068.

[0040] The facial recognition unit 2062 performs facial recognition. Specifically, it compares the facial image extracted by the facial image extraction unit 2044 with the facial image obtained by referring to the authentication information. If a matching facial image is found, it identifies the user associated with that facial image and authenticates that user.

[0041] The identification level calculation unit 2064 calculates the identification level of a user if facial recognition is unsuccessful. The "identification level" is an index value calculated based on relevant information of a candidate user who is presumed to have been photographed by the camera 10, and indicates whether the user has been correctly identified as the person who was actually photographed. A higher identification level value indicates that the user has been correctly identified, meaning that although facial recognition was unsuccessful, the user who was actually photographed has been identified based on the relevant information.

[0042] The inquiry unit 2066, if it cannot determine from the relevant information that the user who was actually photographed has been correctly identified, inquires with a designated user to confirm whether they are the user who was actually photographed. The notification unit 2068 notifies the equipment that the user photographed by camera 10 intends to use of the authentication result.

[0043] The related information update unit 208 updates the previously configured related information with any new related information it acquires. If new related information is acquired for an item that is not currently configured, it will be registered as a new item.

[0044] Figure 4 shows an example of the data structure of authentication information set and registered in the authentication-related information storage unit 222 in this embodiment.

[0045] Authentication information is information about a user who can use any of the equipment installed in Room 2 shown in Figure 1, and is used to authenticate that user. Authentication information is set by associating the user ID, which is the user's identification information, with a password, the user's username, and face image storage location information indicating where the face image used for facial authentication of the user is stored. Alternatively, the face image data itself may be directly linked instead of the face image storage location information. In this embodiment, authentication information is used by combining knowledge information consisting of a user ID and password pair with biometric information in the form of a face image. If authentication information other than the above is used, it will be managed using this authentication information. Authentication information is set and registered by an administrator or other person before user authentication is performed.

[0046] Figure 5 shows an example of the data structure of related information set and registered in the authentication-related information storage unit 222 in this embodiment. Related information is information other than authentication information related to a user who can use any of the equipment installed in room 2 shown in Figure 1. Basically, related information is set for each user for whom authentication information has been set. Related information is managed by associating it with the user ID. In this embodiment, related information is classified and managed into personal attribute information, equipment usage information, and image extraction information. Of course, it is not necessary to limit it to such classifications or types of information. Personal attribute information is set to the physical characteristics of the user. Figure 5 shows an example in which height and build are set, but it may also include features that can be extracted from captured images, such as walking style and hand movements. In this embodiment, the equipment is exemplified as conference room 4, multifunction printer 6, and PC 8, but the equipment usage information is set to the usage status of these pieces of equipment. For example, user ID "u001" "Taro Yamada" is configured to primarily use multifunction printer A with equipment ID "F0001" between 10:00 and 12:00 every day, and not use it or hardly use it at other times. Personal attribute information and equipment usage information may be set and registered by individual users or administrators. Equipment usage information may also be set automatically by referring to contracts, usage history, etc.

[0047] Image extraction information is information extracted from images captured by any of the cameras 10, and is information items related to the user. In the image extraction information example shown in Figure 5, the characteristics of the clothing worn today are set. For example, "Koichi Suzuki" with user ID "u002" is set to be wearing a black jacket on his upper body and green pants on his lower body. Image extraction information uses clothing as an example, but other items worn (bags, rings, accessories, etc.) may also be registered. Basically, image extraction information is set by extracting information about the user as related information from the same captured image referenced when the user successfully performs facial recognition. By extracting related information from the same captured image from which the facial image was extracted, it is possible to correctly link the user with the related information of that user. As described above, image extraction information is information that can be extracted by analyzing an image. However, the items to be set in image extraction information do not necessarily have to be extracted from the captured image; the user or administrator may set and register any items that can be extracted from an image, such as the color of the clothes.

[0048] In this embodiment, authentication information and related information are shown separately, but they may be managed in an integrated manner.

[0049] Figure 6 shows an example of the data structure of equipment information set and registered in the equipment information storage unit 224 in this embodiment. The equipment information is information about the equipment shown in Figure 1, which can only be used after the user has successfully authenticated. The equipment information is set by associating the equipment ID, which is the identification information of the equipment, with the equipment name, the identification level, and the list of available users, i.e., a list of users who are permitted to use the equipment. The "identification level" used in this embodiment is, as described above, an index value calculated based on the relevant information of a candidate user who is presumed to have been photographed by the camera 10, and is an index value that indicates whether the person actually photographed can be correctly identified from the relevant information of that user. The identification level set in the equipment information is a reference value that serves as a threshold and is compared with the calculated identification level. The higher the value of the identification level, the more it means that the user actually photographed can be correctly identified even if facial recognition is not possible. However, no matter how high the calculated identification level is, if it is not equal to or higher than the reference value of the identification level set for the equipment, it will not be determined that the user has been correctly identified. Therefore, the higher the level of security required by the equipment, the higher it is preferable to set the identification level for the equipment. The list of available users is a list of users who can use the equipment, and each user's user ID is assigned to it. Equipment information is pre-configured and registered by the administrator, etc., and updated as needed.

[0050] Figure 7 shows an example of the data structure of camera information to be set and registered in the camera information storage unit 226 in this embodiment. The camera information is set by associating the camera ID, which is the identification information of the camera 10, with the equipment ID of the equipment located within the shooting range of the camera, the shooting equipment coordinate information, that is, the coordinate information of each piece of equipment within the shooting range, and the accuracy information of the camera 10. For convenience, the camera ID is shown as being associated with the symbols of each camera 10 shown in Figure 1. Figure 1 illustrates the shooting range 14 for each camera 10, but a two-dimensional coordinate system is virtually set in room 2 according to a predetermined standard. For example, a two-dimensional coordinate system is set in room 2 with the lower left corner of room 2 as the origin, and the installation positions of equipment 4, 6, and 8 (for example, the center point of each piece of equipment) are represented by two-dimensional coordinate data within a predetermined memory width. If necessary, it may be represented in three dimensions. The shooting equipment coordinate information contains the coordinate data of each piece of equipment located within the shooting range of the camera 10. Therefore, by analyzing the image captured by the camera 10, it is possible to determine which piece of equipment the captured user is trying to use.

[0051] The accuracy information is an index value that represents the accuracy of whether a user captured by the camera 10 can be correctly authenticated, depending on the performance and size of the camera 10's shooting range. If the size of the camera 10's shooting range is set to 100, the accuracy can be set according to the proportion of the size of the captured user that is included in that shooting range. It can be inferred that the larger the user is captured by the camera 10, the higher the accuracy of user recognition. In other words, it means that the reliability of the captured image is higher. For example, in the case of camera 10d with camera ID "c004", the accuracy for camera 10d is 40% if the size of the user occupies 25% or less of the camera 10d's shooting range, 60% if the size of the user is greater than 25% but 50% or less, and 80% if the size of the user is greater than 50%.

[0052] The image extraction information included in the related information is intended to be extracted from captured images, so the accuracy set in the accuracy information can be said to be an indicator of the reliability of the related information extracted from the captured images. In Figure 7, the number of divisions, range, and accuracy value set in the accuracy information are set by the administrator, etc. Since the accuracy information depends on the performance and shooting range of each camera 10, it is also possible to set it automatically.

[0053] Next, the user authentication process in this embodiment will be explained using the flowcharts shown in Figures 8A and 8B.

[0054] As mentioned above, the various pieces of information shown in Figures 4-7 must be set and registered before the user authentication process is performed. However, when facial recognition is successful using a user's face image captured by any of the cameras 10, related information extracted from the same captured image is set and registered. In this embodiment, although none of the equipment is within the shooting range, a camera 10a is installed to photograph entrants from the front. In other words, if camera 10a is installed to photograph the entire body of an entering user from the front, it is basically possible to capture the user's face image, and related information can also be extracted from the captured image including the face image. Therefore, the image extraction information can be registered at least at the time the user enters the room.

[0055] The image acquisition unit 202 continuously acquires images from multiple cameras 10 (step S101). After the user enters room 2 shown in Figure 1, when using any of the equipment, they stand or sit in front of the equipment. Since camera 10 is constantly recording, it is able to capture the user standing in front of the equipment.

[0056] The authentication action detection unit 2042 in the image processing unit 204 detects a user and whether the user has performed a predetermined action by analyzing the captured image acquired from the camera 10. A predetermined action is, for example, raising a hand, remaining still for a few seconds, or sitting down. These predetermined actions are predetermined and communicated to the user in room 2 in advance. The authentication action detection unit 2042 repeats the above process until it detects the desired action from the captured image (N in step S102). Note that existing technologies may be used for user detection and extraction of predetermined actions.

[0057] A user who wishes to begin using the equipment must perform a prescribed action indicating their intention to begin using the equipment. The user may then choose any of the prescribed actions described above.

[0058] When a desired action is detected from the captured image (Y in step S102), the authentication action detection unit 2042 refers to the camera information to identify the camera 10 that detected the user and the equipment the user intends to use (step S103). Since the camera ID of the transmitting camera 10 is associated with the captured image when it is transmitted, the camera 10 that detected the user can be identified, and by comparing the position where the user was detected in the captured image with the camera equipment coordinate information of said camera 10, the equipment the user intends to use can be identified.

[0059] In the following explanation, we will focus on the images detected by the user and the equipment the user intends to use. Unless otherwise specified, when we simply refer to "images," we mean the images detected by the user. Similarly, when we simply refer to "equipment," we mean the equipment the user intends to use. Likewise, when we simply refer to "user," we mean the user who intends to use the equipment in question.

[0060] Next, the face image extraction unit 2044 extracts a face image by analyzing the captured image (step S104). Since the camera 10 may not be able to detect the face if it is photographing the user from behind or elsewhere, strictly speaking, it will attempt to extract a face image. Similarly, the related information extraction unit 2046 extracts related information by analyzing the captured image (step S105). As with the face image, there may be cases where predetermined related information cannot be extracted from the captured image, so strictly speaking, it will attempt to extract related information. Note that existing technologies may be used for the method of extracting face images and related information from the captured image.

[0061] Next, if the face image extraction unit 2044 has successfully extracted a face image (Y in step S106), the face authentication unit 2062 in the authentication processing unit 206 performs face authentication by comparing the user's face image extracted from the captured image with the user's face image obtained based on the authentication information (step S107). If face authentication is successful (Y in step S108), the user intending to use the equipment has been identified. The processing in this case, i.e., the processing from step S114 onwards, will be described later.

[0062] On the other hand, if facial recognition is unsuccessful (N in step S108), it means that the user attempting to use the equipment could not be identified through facial recognition. In this embodiment, a determination is made based on the user identification level, which is one of the features of this embodiment. To this end, the identification level calculation unit 2064 calculates the identification level as follows (step S109). The identification level is also calculated if the facial image extraction unit 2044 fails to extract a facial image (N in step S106).

[0063] In this embodiment, a specific level is calculated using the following formula. Of course, this formula is just one example, and different variables may be added, or weights may be set appropriately for the variables. Specific level = Image analysis accuracy × Relevance rate of related information ···(1)

[0064] In this embodiment, if facial recognition is unsuccessful, or if the user cannot be identified from the facial image even if facial recognition is successful, the user is identified using related information. Therefore, in this case, it is assumed that the related information has been extracted in step S105. Here, we will explain in detail using the data setting examples of various information shown in Figures 4 to 7.

[0065] First, let's assume that information indicating the user's clothing is "patterned + blue" is extracted from the captured image. The specific level calculation unit 2064 reads and obtains relevant information for each user from the authentication-related information storage unit 222. In this embodiment, the system selects candidates for the user who was actually photographed by referring to the degree of agreement between the information extracted from the captured image and the relevant information shown in Figure 5. Following the example settings for authentication information exemplified in Figure 5, it can be seen that the users whose relevant information matches are "Taro Yamada" with user ID "u001" and "Takashi Kimura" with user ID "u004". In other words, by referring to the relevant information, it can be inferred that the user is either "Taro Yamada" or "Takashi Kimura", so they are selected as candidates for the user who was actually photographed. Here, let's assume that the user is extracted from the image captured by camera 10c, and from the camera information of camera 10c, it is known that the equipment used by the user is multifunction printer 6 (referred to as "multifunction printer B" in this explanation), with equipment ID "F002" and equipment name "multifunction printer B".

[0066] In the example above, for convenience, users who perfectly match "Gara + Blue" extracted from the captured image are selected as candidates. However, if no users with a perfect match exist, or if various related information is set, candidates may be selected based on the degree of matching. In other words, users whose degree of matching exceeds a predetermined threshold are selected as candidates.

[0067] Here, we calculate the identification level of "Takashi Kimura" among the two candidates. To do this, the identification level calculation unit 2064 calculates the area occupied by "Takashi Kimura" within the image area of ​​the image captured by camera 10c. For example, if "Takashi Kimura" occupies 10% of the image area, the accuracy is 80% when referring to the camera information of camera 10c. Also, in this example, the related information for "Takashi Kimura" overlaps with the related information for "Taro Yamada" (i.e., there is one other person with the same related information, so a total of 2 people), so the overlap rate of related information for "Takashi Kimura" is (1-1 / 2)=50%. Therefore, the identification level for "Takashi Kimura" can be calculated as 80% × 50% = 40%. This means that the validity of the inference that the photographed user is "Takashi Kimura," or in other words, the confidence level that it is safe to identify the photographed user as "Takashi Kimura," corresponds to 40%. In this embodiment, this validity or confidence level corresponds to the aforementioned "identification level." The same specific level is calculated for other candidates, in this example, "Taro Yamada," but in this case, the specific level is the same as that of "Takashi Kimura."

[0068] As another example, suppose the related information that the user's clothing is "black + green" is extracted from the captured image. In this case, referring to the related information matches the related information of one user. Referring to the authentication information, it is found that the user is "Koichi Suzuki" with user ID "u002". In other words, it can be inferred that the user who was photographed is "Koichi Suzuki", so he is selected as a candidate for the user who was actually photographed. Here, it is assumed that the user is extracted from the image captured by camera 10c, and from the camera information of camera 10c, it is determined that the equipment the user is using is multifunction printer B.

[0069] If the image of "Koichi Suzuki" occupies 45% of the area of ​​the captured image, then referring to the camera information of camera 10c, the accuracy is 80%. Furthermore, "Koichi Suzuki" does not overlap with the related information of other people, and is unique to him (i.e., there are 0 people with the same related information, totaling 1 person), so the overlap rate of related information for "Koichi Suzuki" is (1-0 / 1)=100%. Therefore, the identification level of "Koichi Suzuki" can be calculated as 80% × 100% = 80%. This means that the validity of the inference that the photographed user is "Koichi Suzuki," or in other words, the confidence level that it is acceptable to identify the photographed user as "Koichi Suzuki," corresponds to 80%. In this embodiment, this index of validity or confidence corresponds to the aforementioned "identification level."

[0070] The specific level calculation unit 2064 calculates the specific level of each candidate user whose image was captured as described above.

[0071] The aforementioned formula (1) for calculating the specific level reflects only "today's attire" from the image extraction information among the related information, and calculates the specific level by referring to the degree of overlap of "today's attire." However, other items may also be reflected in the formula. In other words, items included in personal attribute information and equipment usage information may be incorporated into the formula for calculating the specific level. For example, the height estimated from the captured image may be compared with the height included in the personal attribute information of the related information, and the degree of agreement obtained by a predetermined calculation may be included in the formula. Also, if the user's entire body is captured in the captured image, the user's height can be obtained more accurately, so the weight coefficient value set for height may be made larger than that of other items. Furthermore, the date and time of capture of the captured image may be referred to, and if the date and time of capture corresponds to the settings in the equipment usage information, a relatively larger weight coefficient may be set compared to other items. If it does not correspond, the captured user may be excluded from the candidates.

[0072] Next, the authentication processing unit 206 compares the calculated specific level with a pre-set reference value. This reference value is a threshold used to determine whether the captured user can be definitively identified as the actual user. Here, the specific level set on the equipment, in this example, multifunction printer B, is used as the reference value. If the specific level is equal to or greater than the reference value, the user with that specific level is identified as the user who will actually use the equipment. On the other hand, if the specific level is less than the reference value, the user with that specific level is not identified as the user who will actually use the equipment. Note that this means it is uncertain to identify the user in this way, and does not mean that the user with that specific level is not the user who will actually use the equipment.

[0073] According to the equipment information shown in Figure 6, a specific level of 50% is set for multifunction printer B. In this embodiment, this specific level of 50% is used as the baseline value, so "Takashi Kimura," whose specific level is 40% as exemplified above, is not identified as a user who will actually use the equipment. The other candidates may or may not be identified as users who will actually use the equipment. However, in this case, the specific level of another candidate, "Taro Yamada," is calculated to be 40%, so none of the candidates meet the baseline value. On the other hand, "Koichi Suzuki," whose specific level is 80%, is identified as a user who will actually use the equipment.

[0074] Here, if the candidate's identification level is above the standard value, as in the example of "Koichi Suzuki" (Y in step S110), then, similar to when facial recognition is successful, the user who intends to use the equipment has been identified (step S111). The processing in this case, that is, the processing from step S114 onwards, will be described later.

[0075] On the other hand, if, as in the example of "Takashi Kimura," the specific level does not meet the standard value (N in step S110), and if "Taro Yamada," another candidate for user, also does not meet the standard value for the specific level, that is, if the specific levels of all candidate users do not meet the standard value (N in step S110), the inquiry unit 2066 attempts to identify the user who intends to use the equipment by executing the following inquiry process (step S112).

[0076] To contact users, you first need to select potential users to contact. Then, if you need to contact multiple users, you need to decide how to contact them and in what order. The method of contacting users can be determined according to established rules, for example, as follows:

[0077] First, it is necessary to select the users to contact. In short, the users to be selected are those who were actually photographed, based on the degree of match between the relevant information extracted from the photographed image and the actual user. Following the example above, it would be acceptable to select all the users whose relevant information matches the information extracted from the photographed image (in the example above, "Taro Yamada" and "Takashi Kimura").

[0078] However, it is not always possible to correctly extract information from the captured image to match with the related information shown in Figure 5. Depending on the size and angle at which the user is displayed, it may not be possible to correctly extract the information to be compared with the related information. Even if extraction is possible, depending on the angle at which the user is photographed, it may not be possible to correctly capture, for example, the pattern or color of the clothing. Following the above example, it may not be possible to extract information to compare with "pattern + blue" from the captured image. Therefore, instead of limiting selection to cases where the information extracted from the captured image and the related information are a perfect match, it may be possible to select based on the degree of match with the related information extracted from the captured image. For example, the candidates for inquiry may be broadened to include users whose clothing for today registered in the related information is "pattern + purple" or users whose upper body is "patterned". In this case, the inquiry unit 2066 will calculate the degree of match for each user according to a predetermined rule and select users whose calculated degree of match is above a predetermined threshold as inquiry targets.

[0079] Furthermore, since the users who can use the equipment are set according to the list of available users in the equipment information shown in Figure 6, these set users may be selected as candidates for inquiries. Alternatively, all users registered in the authentication information shown in Figure 4 may be selected as candidates for inquiries.

[0080] When multiple users are selected as contacts, it is necessary to determine the order in which to contact them. Simply put, all users can be contacted simultaneously and in parallel, similar to a broadcast. Alternatively, the order in which they are registered in the authentication information can be used. Even when contacting all users, those listed on the equipment information's available user list are more likely to be relevant, so they should be contacted first. Furthermore, the order can be determined based on a specific level calculated for each candidate user. Candidates whose information matches the information extracted from the captured images are more likely to be users actually intending to use the equipment, so candidates with a higher calculated specific level can be contacted first.

[0081] Furthermore, if you manage user information for any of the equipment currently in use, you may exclude those users from the list of contacts. This is because users who are already using one piece of equipment are unlikely to start using another. This makes inquiry processing more efficient as it avoids unnecessary inquiries.

[0082] You can also set a timeout period for when a response will be received, and if no reply is received by the deadline, you can then forward the inquiry to the next contact point.

[0083] Furthermore, for users who are the recipients of inquiries, it is also possible to send emails or messages to their mobile devices (smartphones, etc.) by referring to the user's personal information (not shown) which includes the individual email address of the person making the inquiry.

[0084] The inquiry unit 2066 makes an inquiry as described above and estimates the user who intends to use the equipment by referring to the result of the inquiry (step S113). If there are no malicious users, normally a positive response should be sent from only one inquiry recipient, that is, a response declaring that the user intending to use the equipment is themselves. Therefore, the user intending to use the equipment can be identified as the user estimated here.

[0085] Incidentally, when making an inquiry, the user will be asked which equipment (for example, multifunction printer B) they intend to use, and it is expected that they will respond with either yes or no as described above. When making this inquiry, the user may be asked to provide additional relevant information. For example, the user may be asked about their current clothing, or whether there have been any changes in the equipment usage status. For example, it may be revealed that the user was wearing a white jacket when entering the room, but took it off after entering, and is now wearing a brown hoodie. If the user adds information to the inquiry result in response to such an inquiry, the related information update unit 208 updates the related information set in the authentication related information storage unit 222 with the specified information.

[0086] Since the system attempts to identify a user by detecting authentication activity performed by a user using the equipment, even if facial recognition is successful, even if the identification level exceeds the threshold, even if the above query is initiated, one user will have been identified at this point.

[0087] Next, in step S105, it is checked whether the relevant information has been extracted. If the face image has not been extracted in step S106, the relevant information should have been extracted, as the user is identified using the relevant information. Also, if the face image has been extracted in step S106, it is assumed that the relevant information has also been extracted. Of course, it is not guaranteed that the entire body of the user will be photographed, so there is no guarantee that the relevant information will be extracted. If the relevant information has not been extracted in this way (N in step S114), the process is terminated. If the relevant information has been extracted (Y in step S114), the relevant information update unit 208 updates the relevant information extracted in step S105, especially the image extraction information (step S116). Alternatively, the relevant information update unit 208 may update the image extraction information at this point as relevant information with information specified by the user in response to an inquiry.

[0088] In this embodiment, we have used as an example a case where the user specifies the relevant information in response to an inquiry, but the user may also specify the relevant information proactively without responding to an inquiry. In this case, the relevant information update unit 208 receives the relevant information specified by the user and updates the relevant information set in the authentication relevant information storage unit 222.

[0089] According to this embodiment, since the relevant information can be updated to the latest state, the reliability of the relevant information can be improved. Consequently, even if facial recognition fails, the accuracy of user estimation using the relevant information can be improved.

[0090] As described above, once a user intending to use the equipment is identified, the authentication processing unit 206 performs the normal authentication process. That is, the authentication processing unit 206 performs user authentication using knowledge information, and if this authentication is successful, it refers to the equipment information and, if the identified user is a user who can use the equipment, it permits the user to start using it. In this case, the notification unit 2068 notifies that user authentication has been successful. Upon receiving this notification, the equipment unlocks the door or allows the user to log in, thereby enabling the user to start using the equipment. On the other hand, if the identified user is not a user who can use the equipment, upon receiving this notification, the equipment keeps the door locked or does not allow the user to log in. In other words, it does not allow the user to use the equipment.

[0091] As explained above, according to this embodiment, even if facial recognition is unsuccessful, the user attempting to use the equipment can be identified by referring to related information.

[0092] Although this embodiment has been described using the example of applying the information processing system to an authentication system that performs facial recognition using facial images extracted from captured images, it can be applied to any system that has the function of identifying a user from facial images extracted from captured images, not necessarily an authentication system.

[0093] In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In that case, the processor is configured to work in cooperation with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0094] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), a dedicated circuit for executing a specific process such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, these components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0095] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a set of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located on physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0096] This invention can also be applied to programs and program products.

[0097] (Note) (((1))) Equipped with a processor, The aforementioned processor, If the facial image of the user attempting to use the equipment cannot be extracted from the captured image, information related to the user will be extracted from the captured image. The relevant information relating to users who have been registered as users of the aforementioned equipment is obtained. Based on the degree of agreement between the information extracted from the aforementioned captured image and the aforementioned related information, candidates for the actual user who was photographed are selected. For each of the aforementioned candidates, an identification level is calculated based on the relevant information, which serves as an indicator of whether the person in the photograph can be correctly identified. If none of the calculated specific levels reach the predetermined threshold, the candidate will be contacted. Based on the inquiry results from the aforementioned candidates, the user who was photographed is estimated. The information extracted from the aforementioned captured images is used to update the relevant information related to the estimated user. An information processing system characterized by the following: (((2))) The information processing system (((1))) is characterized in that, if the user specifies information in the query result from the candidate, the processor sets the information to related information associated with the user. (((3))) The information processing system according to (((2))), characterized in that the processor queries all of the candidates. (((4))) The information processing system according to (((1))), characterized in that the processor determines a method for querying the candidate in accordance with predetermined rules. (((5))) The information processing system according to (((4))), characterized in that the processor determines the order in which to query the candidates according to a specific level calculated individually for each candidate. (((6))) The information processing system according to (((4))), characterized in that the processor queries all of the candidates at once. (((7))) The information processing system according to (((4))), characterized in that the processor removes any user included in the candidates from the list of candidates if that user is using any of the multiple pieces of equipment managed by the information processing system. (((8))) The information processing system according to any one of (((1))) to (((7))), characterized in that the processor calculates the specific level by referring to the degree of overlap of the settings of related information associated with the registered user. (((9))) An information processing system described in any one of (((1))) to (((8))), A number of shooting methods less than the number of devices requiring facial recognition when starting use, It has, Facial recognition is performed using a user's facial image extracted from a captured image obtained from one of the aforementioned shooting means. A facial recognition system characterized by the following features. (((10))) On the computer, If the facial image of a user attempting to use the equipment cannot be extracted from the captured image, a function will be provided to extract information related to the user from that image. A function to retrieve relevant information related to users who are registered as users of the aforementioned equipment. A function that selects candidates for the actual user in the photograph by referring to the degree of agreement between the information extracted from the aforementioned captured image and the aforementioned related information. A function to calculate an identification level for each of the aforementioned candidates, which serves as an indicator of whether the person in the photograph can be correctly identified from the user's related information. If none of the calculated specific levels reach a predetermined threshold, a function is provided to contact the candidate. A function to estimate the user who was photographed, based on the inquiry results from the aforementioned candidates. A function to update relevant information related to the estimated user using information extracted from the aforementioned captured image. A program to achieve this.

[0098] According to the invention described in (((1))), when a user's face image cannot be extracted from a captured image, estimating the captured user using related information associated with the user can improve the accuracy of user estimation compared to when the related information is not updated. According to the invention described in (((2))), information specified by the user can be reflected in related information. According to the invention described in (((3))), information specified by each candidate, not limited to the user who was actually photographed, can be reflected in the user's related information. According to the invention described in (((4))), inquiries can be made in a desired manner. According to the invention described in (((5))), queries can be made in descending order of specific levels. According to the invention described in ((6)), the user who has been photographed can be estimated more quickly. According to the invention described in (((7))), users who cannot be candidates because they are using other equipment can be excluded from the inquiry. According to the invention described in ((8)), it is possible to estimate with a high probability that the user whose information matches the information extracted from the captured image and whose related information settings have less overlap with those of other users is the user who was photographed. According to the invention described in (((9))), when a user's face image cannot be extracted from a captured image, estimating the captured user using related information associated with the user can improve the accuracy of user estimation compared to when the related information is not updated. According to the invention described in (((10))), when a user's face image cannot be extracted from a captured image, estimating the captured user using related information associated with the user can improve the accuracy of user estimation compared to when the related information is not updated. [Explanation of Symbols]

[0099] 2 Rooms, 4a Large conference room, 4b, 4c Small conference rooms, 6 Multifunction printer, 10a~10d Camera, 12 User, 14a~14d Shooting range, 20 Information processing device, 21 CPU, 22 ROM, 23 RAM, 24 Hard disk drive (HDD), 25 User interface (UI), 26 Network interface (IF), 202 Image acquisition unit, 204 Image processing unit, 206 Authentication processing unit, 208 Related information update unit, 222 Authentication related information storage unit, 224 Recent information storage unit, 226 Camera information storage unit, 2042 Authentication activity detection unit, 2044 Face image extraction unit, 2046 Related information extraction unit, 2062 Face recognition unit, 2064 Specific level calculation unit, 2066 Inquiry unit, 2068 Notification unit.

Claims

1. Equipped with a processor, The aforementioned processor, If the facial image of the user attempting to use the equipment cannot be extracted from the captured image, information related to the user will be extracted from the captured image. The relevant information relating to users who have been registered as users of the aforementioned equipment is obtained. Based on the degree of agreement between the information extracted from the aforementioned captured image and the aforementioned related information, candidates for the actual user who was photographed are selected. For each of the aforementioned candidates, an identification level is calculated based on the relevant information, which serves as an indicator of whether the person in the photograph can be correctly identified. If none of the calculated specific levels reach the predetermined threshold, the candidate will be contacted. Based on the inquiry results from the aforementioned candidates, the user who was photographed is estimated. The information extracted from the aforementioned captured images is used to update the relevant information related to the estimated user. An information processing system characterized by the following:

2. The information processing system according to claim 1, characterized in that, if the user specifies information in the query result from the candidate, the processor sets the information to relevant information related to the user.

3. The information processing system according to claim 2, characterized in that the processor queries all of the candidates.

4. The information processing system according to claim 1, characterized in that the processor determines a method for querying the candidate in accordance with predetermined rules.

5. The information processing system according to claim 4, characterized in that the processor determines the order in which to query the candidates according to a specific level calculated individually for each candidate.

6. The information processing system according to claim 4, characterized in that the processor queries all candidates at once.

7. The information processing system according to claim 4, wherein the processor removes any user included in the candidates from the list of candidates if that user is using any of the multiple pieces of equipment managed by the information processing system.

8. The information processing system according to claim 1, characterized in that the processor calculates the specific level by referring to the degree of overlap in the settings of related information associated with the registered user.

9. An information processing system according to any one of claims 1 to 8, A number of shooting methods less than the number of devices requiring facial recognition when starting use, It has, Facial recognition is performed using a user's facial image extracted from a captured image obtained from one of the aforementioned shooting means. A facial recognition system characterized by the following features.

10. On the computer, If the facial image of a user attempting to use the equipment cannot be extracted from the captured image, a function will be provided to extract information related to the user from that image. A function to retrieve relevant information related to users who are registered as users of the aforementioned equipment. A function that selects candidates for the actual user in the photograph by referring to the degree of agreement between the information extracted from the aforementioned captured image and the aforementioned related information. A function to calculate an identification level for each of the aforementioned candidates, which serves as an indicator of whether the person in the photograph can be correctly identified from the user's related information. If none of the calculated specific levels reach a predetermined threshold, a function is provided to contact the candidate. A function to estimate the user who was photographed, based on the inquiry results from the aforementioned candidates. A function to update relevant information related to the estimated user using information extracted from the aforementioned captured image. A program to achieve this.

Images