Information processing device, control method for information processing device, and program

Abstract

To improve the accuracy of facial recognition in a more suitable manner. [Solution] The feature extraction unit 303 extracts feature quantities from the face image obtained from the target image. The score calculation unit 305 compares the extracted first feature quantity with the second feature quantity held in the storage unit 304 to calculate a matching score. The authentication unit 306 determines that authentication is successful if the matching score exceeds a threshold. The additional registration / update determination unit 309 determines whether at least two of the following criteria are met: the shooting intent criterion, the score criterion, and the text criterion. If the additional registration / update unit 310 determines that two or more of the above criteria are met, it performs additional registration of the matching face image and matching feature quantity, or updates the registered face image and registered feature quantity held in the storage unit 304 using the matching face image and matching feature quantity.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] Face authentication is a technique that compares feature amounts extracted from two face images and determines whether the subjects of the two face images are the same person based on the result. Face authentication is often used for security purposes, but in recent years it has also been used for autofocus control of digital cameras and is also called a focus assist function. This focus assist function is particularly needed in scenes where it is difficult to focus on a desired subject, such as professional sports and athletic meets. However, in such scenes, variations in a person's expression and face orientation are large, and it may be difficult for the camera to correctly recognize the desired subject. In response to such problems, various techniques for improving recognition performance by updating or additionally registering registered feature amounts have been studied. Patent Document 1 discloses a technique for updating a registered feature amount if a collation score between a registered feature amount and a collation feature amount is equal to or greater than a threshold value. At this time, in order to prevent incorrect updating with the feature amount of another person, the threshold value for feature update is set to a higher value than the threshold value for person determination. Patent Document 2 discloses a technique for counting the number of successful authentication times in time-series images and, if this number is equal to or greater than a predetermined value, additionally registering an image that is similar to a successful authentication image by a predetermined value or more among failed authentication images. Patent Document 3 discloses a technique for updating a registered feature amount with that feature amount when the collation score is within a predetermined range.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

[0004] On the other hand, the technology disclosed in Patent Document 1 updates registered features using features from faces with high matching scores, i.e., faces that are closer to the registered face and have favorable conditions. Therefore, it tends not to update features from faces with unfavorable conditions, such as profiles, which are required in sports scenes. Furthermore, the technologies disclosed in Patent Documents 1 and 2 require successful authentication as a condition for adding or updating face images or features. Therefore, in situations where the subject is moving rapidly, such as in sports scenes, authentication may not be successful, and as a result, the function may not work. In addition, although the technology disclosed in Patent Document 3 allows feature updates even without successful authentication, there is a possibility that the features of another person may be mistakenly reflected.

[0005] In view of the above-mentioned problems, the present invention aims to improve the accuracy of facial recognition in a more suitable manner. [Means for solving the problem]

[0006] The information processing apparatus according to the present invention is characterized by comprising: acquisition means for acquiring a face image from a target image; extraction means for extracting feature quantities from the face image; holding means for holding the face image and the feature quantities extracted from the face image; calculation means for calculating a matching score by comparing a first feature quantity extracted by the extraction means and a second feature quantity held in the holding means; authentication means for determining that authentication is successful when the matching score exceeds a threshold; determination means for determining whether at least two of the following criteria are met: a shooting intention criterion related to determining the user's shooting intention, a score criterion related to determining the matching score, and a context criterion related to determining the shooting conditions; and, if the determination means determines that the two or more criteria are met, additional update means for adding the first face image acquired by the acquisition means and the first feature quantities to the holding means, or updating the second face image and the second feature quantities held in the holding means based on the first face image and the first feature quantities. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the accuracy of facial recognition in a more suitable manner. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows an example of a general configuration of an information processing device. [Figure 2] This diagram shows an example of the hardware configuration of an information processing device. [Figure 3] This diagram shows an example of the functional configuration of an information processing device. [Figure 4] This is a flowchart illustrating an example of processing performed by an information processing device. [Figure 5] This diagram shows an example of the display unit. [Figure 6] This diagram shows an example of the display unit. [Figure 7] This diagram shows an example of the functional configuration of an information processing device. [Figure 8]It is a flowchart showing an example of the processing of an information processing apparatus. [Figure 9] It is a diagram showing an example of the display on the display unit. [Figure 10] It is a flowchart showing an example of the processing of an information processing apparatus. [Figure 11] It is a diagram showing an example of the display on the display unit.

Embodiments for Carrying Out the Invention

[0009] Embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. In each figure, the same or corresponding parts are denoted by the same reference numerals. Redundant descriptions of such parts will be appropriately simplified or omitted.

[0010] <First Embodiment> As a first embodiment of the present disclosure, an example of a case where the technology according to the present disclosure is applied to improve the tracking performance when a user tracks a subject that is a registered tracking target in a sports scene with intense movement using a camera will be described. Specifically, in this embodiment, an example of adding registration of a subject for which authentication fails by determination based on a shooting intention criterion and a score criterion (to be described in detail later) and improving the tracking performance will be described.

[0011] FIG. 1 shows, as an example of the schematic configuration of the information processing apparatus according to this embodiment, an example of the configuration of the imaging apparatus when the information processing apparatus is realized as a so-called imaging apparatus. FIG. 1(a) is a front view of the information processing apparatus (imaging apparatus) according to this embodiment. Further, FIG. 1(b) is a rear view of the information processing apparatus. As shown in FIG. 1, the information processing apparatus 100 includes a shutter button 101, an imaging unit 102, a display unit 103, an operation dial 104, and a speaker 105.

[0012] The shutter button 101 is a button for receiving an instruction related to imaging of a still image or a moving image (hereinafter, these are collectively also simply referred to as an image), and in addition to a normal press (full press), a half-press state is also detected. For example, when a still image is taken by a user (photographer), when a half-press state is detected, control for focusing on the subject is applied, and when a full-press state is detected, in addition to focusing on the subject, control for imaging a still image is applied. In the present disclosure, for the sake of convenience, when simply described as "pressed" with respect to the shutter button 101, it is assumed that either "full press" or "half press" can be applied as long as there are no particular restrictions. The imaging unit 102 corresponds to an imaging unit composed of a lens and a sensor (imaging element), receives light information with the sensor through the lens, A / D-converts a signal corresponding to the reception result, and acquires digital data of a still image or a moving image.

[0013] The display unit 103 is an output interface for presenting the target information to the user by displaying the target information as an image based on the digital data, and is realized by a so-called display or the like. For example, the display unit 103 can display the image acquired by the imaging unit 102 as it is, display predetermined related information superimposed on the image, or display character information such as a menu. The operation dial 104 is a button unit that serves as an input interface for receiving operations such as selection instructions from the user. The operation dial 104 can be used, for example, to switch the content displayed on the display unit 103 or to give a selection instruction. The speaker 105 is a unit for outputting sounds such as voices and electronic sounds. The speaker 105 can be used, for example, to output an effect sound in accordance with an operation of the shutter button 101 or the operation dial 104, to output an effect sound in accordance with internal processing, or to output a voice during reproduction of a moving image.

[0014] The position, type, and number of buttons, imaging units, and display units mounted on the information processing device 100 are not limited to the examples described above, as shown in Figure 1. For example, an infrared sensor, a mode dial, etc., may also be provided.

[0015] (Hardware configuration) Referring to Figure 2, an example of the hardware configuration of an information processing device 100 according to one embodiment of the present disclosure will be described. As shown in Figure 2, the information processing device 100 has a CPU (Central Processing Unit) 201, RAM (Random Access Memory) 202, and ROM (Read Only Memory) 203. The information processing device 100 also has a digital signal processing unit 204, an encoder unit 205, a memory control unit 206, a communication control unit 207, an image processing unit 208, an imaging control unit 209, an operation control unit 210, a display control unit 211, and an audio output control unit 212. Each of the above-described components of the information processing device 100 is connected to communicate via a system bus 213. The imaging control unit 209 is connected to the imaging unit 102. The operation control unit 210 is connected to the shutter button 101 and the operation dial 104. The display control unit 211 is connected to the display unit 103. The audio output control unit 212 is connected to the speaker 105.

[0016] The CPU 201 comprehensively controls the operation of the entire information processing unit 100. RAM202 functions as the main memory and work area of ​​CPU201. ROM203 is a memory area that stores control programs and other data executed by the CPU201. The digital signal processing unit 204 performs various processes on the digital signal received via the system bus 213, such as white balance processing, gamma processing, and noise reduction processing. The encoder unit 205 processes the digital image data to convert it into a file format compliant with standards such as JPEG or MPEG. The memory control unit 206 is an interface for connecting to various media (e.g., hard disks, memory cards, CF cards, SD cards, and USB memory sticks).

[0017] The communication control unit 207 is an interface for connecting to external devices (e.g., PCs and smartphones) via wireless LAN or short-range wireless communication, and for sending and receiving data with such devices. The communication control unit 207 may also be used as an input source to the operation control unit 210 or an output destination to the display control unit 211 if the external device is capable of providing input / output functions to the user.

[0018] The image processing unit 208 performs face recognition processing using images acquired by the imaging unit 102 and obtained via the imaging control unit 209, images output from the digital signal processing unit 204, or images acquired from the media via the memory control unit 206 or the communication control unit 207. The image control unit 209 controls the operation of the imaging unit 102, such as focusing, opening and closing the shutter, and adjusting the aperture, based on instructions from the CPU 201. The image control unit 209 also acquires an image from the imaging unit 102 according to the imaging result.

[0019] The operation control unit 210 receives instructions from the user via the shutter button 101 or the operation dial 104 and transmits the information indicated by the instruction to the CPU 201. The display control unit 211 controls the display of images and characters shown on the display unit 103. The display unit 103 may also have a touchscreen function, in which case user instructions using the touchscreen function can be treated as input to the operation control unit 210. The audio output control unit 212 controls the sound, such as voice and sound effects, output to the speaker 105.

[0020] It should be noted that the above-described configuration is merely an example and does not limit the hardware configuration of the information processing device 100. For example, other components other than those exemplified above may be included.

[0021] (Functional Configuration) Referring to Figure 3, an example of the functional configuration of the information processing device 100 according to this embodiment will be described. As shown in Figure 3, the information processing device 100 includes an image acquisition unit 301, a face image acquisition unit 302, a feature extraction unit 303, a storage unit 304, a score calculation unit 305, and an authentication unit 306. The information processing device 100 also includes a first determination unit 307, a second determination unit 308, an additional registration / update determination unit 309, and an additional registration / update unit 310.

[0022] The image acquisition unit 301 acquires images (still images and moving images) captured by the imaging unit 102 via the imaging control unit 209. The face image acquisition unit 302 acquires a face image for comparison from the image acquired by the image acquisition unit 301. Hereafter, this face image will also be referred to as the "comparison face image." The comparison face image is an example of the first face image. The feature extraction unit 303 extracts matching features from the matching face image acquired by the face image acquisition unit 302. Hereafter, these features will also be referred to as "matching features." The matching features are an example of the first feature. The storage unit 304 stores multiple arbitrary face images and arbitrary feature quantities. In this embodiment, the storage unit 304 pre-stores one or more of the face images to be tracked as "registered face images" and the feature quantities extracted from the registered face images as "registered feature quantities". The registered face images are prepared by the user after one or more faces are extracted by the face image acquisition unit 302 from an image containing the target person. The registered feature quantities are feature quantities extracted from the registered face images by the feature extraction unit 303. The registered face image corresponds to an example of a second face image. The registered feature quantities correspond to an example of a second feature quantity.

[0023] The score calculation unit 305 calculates a matching score by comparing the matching feature quantity with the registered feature quantity stored in the storage unit 304. The authentication unit 306 determines whether the matching score exceeds a predetermined threshold (hereinafter referred to as the "authentication threshold"), and if it determines that the threshold is exceeded, it determines that authentication is successful. In addition, if the authentication unit 306 determines that authentication is successful, it may instruct the imaging control unit 209 to focus on the subject. The first determination unit 307 determines whether the shooting intent criteria regarding the user's shooting intent are met. The shooting intent criteria will be described in detail separately later. The second determination unit 308 determines whether or not the scoring criteria for the matching score are met. The scoring criteria for the matching score will be described in detail separately.

[0024] The additional registration / update determination unit 309 determines whether both the shooting intent criteria set by the first determination unit 307 and the score criteria set by the second determination unit 308 are met. The additional registration / update unit 310, if the additional registration / update determination unit 309 determines that the above conditions are met, uses the matched face image and matched feature to perform additional registration or update of the face image and feature. On the other hand, if the additional registration / update unit 310 determines that the above conditions are not met, it stops using the newly registered face image and feature based on the period since the last additional registration or update.

[0025] (process) Referring to Figures 4 to 6, an example of the processing of the information processing device 100 according to this embodiment will be described below. Here, we will first explain the processing of the information processing device 100 assuming that there is only one subject in the captured image. Figure 4 is a flowchart showing an example of the processing of the information processing device 100 according to this embodiment. The series of processes shown in Figure 4 is started, for example, when the information processing device 100, which has been registered by the user as a target to be tracked, is activated and transitions to a state in which imaging can be performed. Figure 5 shows an example of the display of the display unit 103 according to this embodiment when only one subject with a visible face is pictured.

[0026] In S401, the image acquisition unit 301 acquires the image input via the imaging unit 102. In S402, the face image acquisition unit 302 performs normalization processing on the image acquired in S401 by detecting facial organ points and extracting the face region based on the detected information to obtain a matched face image. Any known method may be applied as the normalization processing here. For example, a method may be used in which a face frame is detected on the image and the face region is extracted based on the detected information to obtain a matched face image.

[0027] In S403, the feature extraction unit 303 extracts matching features from the matching face image acquired in S402. Here, the feature extraction unit 303 takes the normalized face image as input and extracts features using a machine learning method that has been pre-trained so that features that are close to the same person are extracted, and features that are far apart from the same person are extracted. In this embodiment, a machine learning method called a neural network (in other words, a machine learning model) is used. However, the above is merely an example, and other machine learning methods may be used as means of feature extraction. As specific examples, linear dimensionality reduction methods such as principal component analysis (PCA) and linear discriminant analysis (LDA), nonlinear dimensionality reduction methods such as kernel PCA, and clustering methods such as k-mean may be used, and the method is not limited to a specific form.

[0028] In S404, the score calculation unit 305 calculates a matching score by comparing the matching features extracted in S403 with the registered features stored in the storage unit 304. Specifically, the matching score is calculated based on the relational expression shown below as (Equation 1).

[0029]

number

[0030] In the above (Equation 1), s is the matching score, f S is the matching feature, f EThese are registered features,<x,y> Let ||x|| represent the dot product of x and y, and ||x|| represent the length of x (L2 norm). As described above, in this embodiment, the matching score is the value obtained by taking the dot product of the normalized matching feature and the registered feature, i.e., the cosine similarity. However, this is merely an example, and distance-based indicators such as Euclidean distance or Manhattan distance, or indicators such as Pearson's correlation coefficient may also be applied. In other words, the means applied are not particularly limited as long as they are means that calculate the similarity between features. If the storage unit 304 holds multiple registered features, the score calculation unit 305 performs a comparison with each of these multiple registered features. In this case, multiple matching scores will be calculated.

[0031] In S405, the authentication unit 306 performs authentication processing based on the matching score calculated in S404. In this embodiment, the authentication unit 306 determines that authentication is successful if the matching score exceeds the authentication threshold, and determines that authentication is unsuccessful if the matching score is below the authentication threshold. If the authentication unit 306 determines that authentication is successful, it may instruct the imaging control unit 209 to focus on the target subject.

[0032] In S406, the first determination unit 307 determines whether the shooting intent criterion regarding the user's intention to take a picture is met. In this embodiment, the first determination unit 307 defines the shooting intent criterion as "the shutter button 101 is pressed, and the matching face image is present within a predetermined range in the image at a predetermined frequency or higher," and determines whether this condition is met. Here, the predetermined frequency refers to the number of a certain set of images within a certain defined range of images when the information processing device 100 acquires images at a predetermined frame rate. As a specific example, a frequency such as three images in five consecutive images corresponds to the predetermined frequency described above. The first determination unit 307 according to this embodiment is equipped with a state for counting the objects related to the specification of this frequency and determines whether or not the conditions for the frequency exemplified above are met.

[0033] Furthermore, the first determination unit 307 according to this embodiment displays the predetermined range as a frame of interest 501 on the display unit 103. In the example shown in Figure 5, the frame of interest 501 is displayed as a square in the center of the display unit 103. However, the example shown in Figure 5 is merely an example, and the frame of interest 501 may be any shape and size as long as it fits within the range of the display unit 103. For example, the frame of interest 501 may be a rectangle with the same width as the display unit 103 and half the height, located in the center of the display unit 103, or it may be a perfect circle with a diameter half the height of the display unit 103, located at the right edge of the display unit 103. Furthermore, the interest frame 501 is assumed to have a predetermined size, and this size may be changed to any value according to user instructions. However, this setting is merely an example, and this embodiment is not limited to this setting. Furthermore, in the above explanation, the condition for the shooting intent criterion was defined as "the shutter button 101 is pressed, and the matching face image is present within a predetermined range in the image at a predetermined frequency or higher." However, this condition is merely an example, and for example, the condition could be "the shutter button 101 is pressed," or "the matching face image is present within a predetermined range in the image at a predetermined frequency or higher." However, this setting is merely an example, and this embodiment does not limit itself to this condition.

[0034] In S407, the second determination unit 308 determines whether the scoring criteria for the matching score are met. In this embodiment, the second determination unit 308 determines whether the scoring criteria are met if "the matching score with the registered feature initially registered by the user is less than the authentication threshold and greater than or equal to the tolerance threshold at a predetermined frequency or more." The tolerance threshold applied here is a value set to prevent frequent misapplication of features, and is pre-set as a value smaller than the authentication threshold. If the above conditions are met, the second determination unit 308 displays a face frame 502 surrounding the subject's face on the display unit 103. The "predetermined frequency" that is the condition for determination in S406 and the "predetermined frequency" in S407 may be the same or different. For example, the determination criterion in S406 may be "a frequency of three or more out of five images," and the determination criterion in S407 may be "a frequency of two or more out of three images." Alternatively, frequency may not be used for the determination in S407, and instead, the condition may be "the state in which the matching score is less than the authentication threshold and greater than or equal to the second determination threshold at least once." Furthermore, this embodiment does not limit these conditions, and various forms of rules for conditions that reflect the shooting intent criterion and the matching score criterion can be applied.

[0035] In S408, the additional registration / update determination unit 309 determines whether the shooting intent criteria established in S406 and the score criteria established in S407 are met simultaneously. If the additional registration / update determination unit 309 determines that both the shooting intent criterion and the score criterion are met simultaneously, it proceeds to S409. In this case, the additional registration / update determination unit 309 changes the color of the face frame 502 and displays the matched face image in a separate frame on the display unit 103, as indicated by the reference numeral 503 in Figure 5. On the other hand, if the additional registration / update determination unit 309 determines that the shooting intent criterion and the score criterion are not met simultaneously, it proceeds to S412.

[0036] In S409, the additional registration / update determination unit 309 refers to the storage unit 304 and determines whether or not it holds additional registered face images and registered feature quantities. However, if the process in S409 is the first process in a series of processes shown in Figure 4 that are repeatedly executed, the storage unit 304 will only hold the face images and feature quantities of pre-registered subjects, and will not hold any additional registered face images or registered feature quantities. If the additional registration / update determination unit 309 determines in S409 that it does not have any additional registered face images or registered feature quantities, it proceeds to S410. On the other hand, if the additional registration / update determination unit 309 determines in S409 that it has additional registered face images and registered feature quantities, it proceeds to S411.

[0037] In S410, the additional registration / update unit 310 causes the matching face image and matching feature quantities to be stored in the storage unit 304 as additional registered face images and registered feature quantities. After the completion of the process in S410, the additional registration / update unit 310 proceeds to S401. As a result, the processes from S401 onwards are executed again.

[0038] In S411, the additional registration / update unit 310 updates the additional registered face images held by the storage unit 304 using the matched face image. In this process, updating means replacing an existing additional registered face image with the matched face image, or registering an additional face image by associating it with the matched face image. The additional registration / update unit 310 also updates the additional registered feature quantities held by the storage unit 304 using the matched feature quantities, the matched score, and the camera parameters at the time of imaging. The updating of the additional registered feature quantities is performed, for example, based on the relational expression shown below as (Equation 2).

[0039]

number

[0040] In the above (Equation 2), f AE ' represents the additional registered features after the update, f S is the matching feature, f AE Let be the additional registered features before the update, α be a parameter determined based on the matching score, and β be a parameter determined based on the camera parameters. For example, α is determined by a function that increases monotonically with respect to the input matching score, taking a large value when the matching score is large. Similarly, β is determined by a function or table that takes a large value when the camera is in focus. In this way, by incorporating the matching score and camera parameters into the update formula, it becomes possible to reduce the contribution of features obtained from extremely abnormal images, such as those that are completely out of focus, i.e., matching features unsuitable for registration, to the update process. Conversely, it is expected that matching features obtained from images that failed to authenticate but had a very high registration matching score, indicating a high degree of identity accuracy, will contribute more to the update process. After the completion of processing in S411, the additional registration / update unit 310 proceeds to processing in S401. As a result, processing from S401 onwards is executed again.

[0041] In S412, the additional registration / update determination unit 309 determines the period since the last additional registration or update and determines whether that period is greater than or equal to a certain amount. If the additional registration / update determination unit 309 determines in S412 that the period since the last additional registration or update has exceeded a certain amount, it proceeds to S413. On the other hand, if the additional registration / update determination unit 309 determines in S412 that the period since the last additional registration or update is not longer than a certain amount, it proceeds to S401. In this case, the processing from S401 onwards will be executed again.

[0042] In S413, the additional registration / update unit 310 stops using the additional registered face images and registered features held in the storage unit 304 (for example, discarding the additional registered face images and registered features). The storage unit 304 temporarily retains the face images and features that have stopped being used as temporarily stored face images and temporarily stored features, but discards them after a certain period of time. The use of these temporarily stored face images and temporarily stored features will be described in detail separately in a third embodiment. After the completion of processing in S413, the additional registration / update unit 310 proceeds to processing in S401. As a result, processing from S401 onwards is executed again. The above is an example of how the information processing device 100 processes images when only one subject with a visible face is pictured.

[0043] Next, as an example of the processing performed by the information processing device 100, we will explain the processing when the captured image contains multiple people. Figure 6 shows an example of the display of the display unit 103 according to this embodiment when there are multiple subjects with visible faces. Specifically, Figure 6(a) corresponds to an example of the display when the conditions set by the additional registration / update determination unit 309 are not met. Figure 6(b) corresponds to an example of the display when the conditions set by the additional registration / update determination unit 309 are met. In the example explained with reference to Figure 4, in S408, the additional registration / update determination unit 309 determined whether the shooting intent criterion established in S406 and the score criterion established in S407 were simultaneously met. If multiple subjects with visible faces are in the photograph, this determination process is performed for each of those subjects. Specifically, the series of processes from S402 to S407 are performed for all of the multiple subjects whose faces are visible, and the information obtained from the results is used to perform conditional judgments for all of the multiple subjects. At this time, if the subjects subject to conditional judgment are close together, the additional registration / update judgment unit 309 may display an enlarged frame 601 on the display unit 103 in a separate frame that enlarges to include the faces of each of those subjects, thereby encouraging the user to reduce the chance of misinterpretation. Furthermore, Figure 6(a) shows an example in which multiple face frames are displayed by the second determination unit 308, indicated by reference numerals 602 and 603. However, it is not necessarily required to display face frames for multiple people. For example, a control may be applied that displays a face frame for the subject with the highest registration matching score among the subjects targeted for face frame display, and informs the user that the face image and feature quantities of this subject will be used for additional registration or updating.

[0044] In S408, if there are multiple subjects that satisfy the determination conditions, the additional registration / update determination unit 309 determines one subject to be added to the registration or updated. As a specific example, the additional registration / update determination unit 309 may select the subject with the highest matching score among the subjects that satisfy the conditions. Note that this selection method is merely an example, and control may be applied to target subjects whose faces are closer to the center of the image, or subjects whose faces are most prominently displayed. Furthermore, if there are multiple matching scores for a single subject, the target may be determined using the average of those multiple matching scores. Of course, these are merely examples and do not limit this embodiment. After the target has been determined, the additional registration / update determination unit 309 performs subsequent processing using the matched face image and matched features of the selected subject. At this time, the additional registration / update determination unit 309 may control the system so that only the face frame 602 of the selected subject is displayed with a changed color, while the face frames 603 of other subjects are hidden. The additional registration / update determination unit 309 may also store the matched face images and matched features of subjects that were not selected for additional registration or update as temporarily stored face images and temporarily stored features in the storage unit 304. These temporarily stored face images and temporarily stored features are discarded after a certain period of time has elapsed. The above is an example of how the information processing device 100 processes images when there are multiple subjects whose faces are visible.

[0045] <Modified form of the first embodiment> In the embodiment described above, an example of a configuration and process in which the first determination unit 307, the second determination unit 308, and the additional registration / update determination unit 309 display interest frames 501, face frames 502, etc., on the display unit 103 was explained. By applying such a configuration and process, it is expected that, for example, the status of subject tracking and additional registration can be intuitively presented to the user. On the other hand, it is not always necessary to display this information, and it may be kept as internal processing.

[0046] Furthermore, while an example was described in which the additional registration / update determination unit 309 changes the color of the face frame 502 when the conditions are met, control may be applied such that the face frame 502 is hidden until the conditions are met, and then displayed from the point when the conditions are met. Also, while an example was described in which the matching face image 503 is displayed on the display unit 103 when the conditions are met, it is not necessarily required that the matching face image 503 be displayed. In addition, while the example shown in Figure 5 shows an example in which only one matching face image 503 is displayed, for example, if the conditions are met consecutively, other matching face images that next meet the conditions may be displayed in a manner adjacent to the matching face image 503. These also apply when there are multiple subjects with visible faces. Furthermore, a setting menu may be provided so that the user can selectively switch between these display modes.

[0047] In the embodiments described above, an example was explained in which additional registration is performed using a single matching face image and a single matching feature, but this is not necessarily required. For example, if the predetermined frequency in S407 is set to two out of three images, those two matching face images may be adopted as the additional registration face images. In this case, the additional registration feature may be an aggregated feature obtained by averaging the two matching feature quantities or by weighting the matching score.

[0048] The above-described embodiment explains an example in which the authentication unit 306 instructs the image processing unit 209 to focus on the target subject when authentication is successful, but this does not limit the actions to those that occur when authentication is successful. For example, instructions to control the entire information processing device 100 may be given, such as instructions for auto white balance or instructions for changing the shutter speed. Also, when authentication fails, instructions may be given to the entire information processing device 100 so that the user can recognize that authentication has failed. As a specific example, the audio output control unit 212 may be instructed to output a sound effect to indicate authentication failure.

[0049] (Effects and Benefits) With the above configuration, even if authentication fails, it becomes possible to register or update facial images and features with high confidence using features extracted from the facial images of the subject that is believed to be the target of tracking. This is expected to improve subject tracking performance, for example, in fast-paced sports scenes.

[0050] <Second Embodiment> In a second embodiment of this disclosure, an example of a technique for improving tracking performance when a user tracks a subject that is believed to be the target of tracking with a camera in specific situations such as musicals or concerts is described. Specifically, in this embodiment, an example of a technique for improving tracking performance is described by adding and registering subjects that fail to be authenticated based on a determination of shooting intent criteria and context criteria (details of which will be described separately). In this embodiment, the explanation will mainly focus on the parts that differ from the first embodiment described above, and detailed explanations of parts that are substantially the same as in the first embodiment will be omitted.

[0051] For example, in special situations such as musicals, even if the target to be tracked is pre-registered, the clothing, makeup, lighting, and angle of view may differ significantly, potentially leading to authentication failure. In the first embodiment described above, we explained an example of a technology that enables the addition / update of facial images and features with high reliability by combining a shooting intent criterion and a score criterion, even in cases where authentication fails. However, attempting to address cases where there is a large visual discrepancy between the registered target to be tracked and the actual subject may reduce reliability. This is because, in order to address such cases, for example, the second judgment threshold that constitutes the score criterion would have to be significantly lowered, increasing the likelihood of additional registration or updates being made by the wrong subject. In light of these circumstances, this embodiment uses contextual criteria related to shooting conditions as an alternative to score criteria. Furthermore, unlike the first embodiment, this embodiment does not pre-register subjects, but identifies and registers subjects after shooting has started. By applying this control, it becomes possible to track and focus on the target subject with high accuracy even in situations where the appearance at the time of imaging differs significantly from what was expected beforehand.

[0052] (Functional Configuration) Referring to Figure 7, an example of the functional configuration of the information processing device 100 according to this embodiment will be described. As shown in Figure 7, the information processing device 100 includes an image acquisition unit 301, a face image acquisition unit 302, a feature extraction unit 303, a storage unit 304, a score calculation unit 305, an authentication unit 306, a first determination unit 307, a third determination unit 701, and an additional registration / update determination unit 309. The holding unit 304 holds multiple arbitrary face images and arbitrary feature quantities. In this embodiment, the holding unit 304 does not hold registered face images or registered feature quantities in its initial state. The score calculation unit 305 compares the matching feature with the registered feature and calculates a matching score when the storage unit 304 stores the registered feature. The third determination unit 701 determines whether or not the contextual criteria (details of which will be described later) regarding the shooting conditions are met. The additional registration / update determination unit 309 determines whether the shooting intent criterion set by the first determination unit 307 and the context criterion set by the third determination criterion 701 are met.

[0053] (process) An example of the processing of the information processing device 100 according to this embodiment will be described with reference to Figures 8 and 9. Figure 8 is a flowchart showing an example of the processing of the information processing device 100 according to this embodiment. The series of processes shown in Figure 8 is started, for example, when the information processing device 100 is activated after the user has specified desired context information and transitions to a state in which imaging can be performed. Here, the desired contextual information is information that indicates when, where, and under what circumstances the subject to be tracked will be tracked. For example, this could include information such as "the subject to be tracked is a person standing in a specific position on stage during a musical that starts at □:00 on △th of ◯th month." Figure 9 shows an example of the display of the display unit 103 according to this embodiment.

[0054] In S404, if the storage unit 304 has registered features, the score calculation unit 305 compares the matching features extracted in S403 with the registered features and calculates a matching score. In S406, the first determination unit 307 determines whether the shooting intent criterion regarding the user's intention to take a picture is met. In this embodiment, the first determination unit 307 uses "the shutter button 101 is pressed" as a condition for the shooting intent criterion and determines whether this condition is met.

[0055] In S801, the third determination unit 701 determines whether the contextual criteria regarding the shooting conditions are met. In this embodiment, the third determination unit 701 uses the condition that "the contextual information regarding the appearance, spatial position, and date and time of the matching face image specified by the user matches the current context regarding them" as a contextual criterion, and determines whether the said condition is met. Furthermore, the third determination unit 701 displays a face frame 901 on the display unit 103 that surrounds the face of a subject that satisfies the context criteria. The third determination unit 701 may also display auxiliary information 902 on the display unit 103 that indicates to the user how well the context criteria are currently met.

[0056] In S802, the additional registration / update determination unit 309 determines whether the shooting intent criterion established in S406 and the context criterion established in S801 are met simultaneously. If the additional registration / update determination unit 309 determines in S802 that both the shooting intent criterion and the context criterion are met simultaneously, it proceeds to S409. In this case, the additional registration / update determination unit 309 changes the color of the face frame 901 and displays the matched face image in a separate frame on the display unit 103, as indicated by the reference numeral 903 in Figure 9. On the other hand, if the additional registration / update determination unit 309 determines in S802 that the shooting intent criterion and the context criterion are not met simultaneously, it proceeds to S412.

[0057] (Effects and Benefits) With the above configuration, even if the appearance of the target being tracked differs significantly from the user's initial expectations, it becomes possible to register or update facial images and features with high reliability. This is expected to improve subject tracking performance, even in special situations such as musicals.

[0058] <Third Embodiment> In the other embodiments described above, an example was explained in which, if no additional registration or update is performed for a certain period of time, the use of the newly registered face images and features is stopped, and they are temporarily stored as temporarily stored face images and features, which are then discarded after a certain period of time. In the other embodiments described above, an example was also explained in which, if there are multiple subjects, the matching face images and matching features of subjects that were not subject to additional registration or update are used as temporarily stored face images and features. This embodiment describes an example of a technique for improving tracking performance by updating registered face images and registered features using these temporarily stored face images and features. In this embodiment, the explanation will mainly focus on the parts that differ from the first and second embodiments described above, and detailed explanations of parts that are substantially the same as the first and second embodiments will be omitted.

[0059] (process) An example of the processing of the information processing device 100 according to this embodiment will be described with reference to Figures 10 and 11. Figure 10 is a flowchart showing an example of the processing of the information processing device 100 according to this embodiment. The series of processes shown in Figure 10 are started, for example, when an operation related to updating a registered face image and registered feature quantities is received from a user. Figure 11 is a diagram showing an example of the display of the display unit 103 according to this embodiment.

[0060] In S1001, the holding unit 304 determines whether it holds at least one pair of temporarily stored face images and temporarily stored feature quantities. If the storage unit 304 determines that it has a pair of temporarily stored face images and temporarily stored feature quantities, it proceeds to process S1002. On the other hand, if the storage unit 304 determines that it does not have a pair of temporarily stored face images and temporarily stored feature quantities, it proceeds to S1003.

[0061] In S1002, the holding unit 304 displays the held face image on the display unit 103 in the manner shown in Figure 11. The content displayed at this time shall include at least one registered face image and one temporarily saved face image. The holding unit 304 also displays on the display unit 103 text 1101 indicating the registered face image, text 1102 indicating the temporarily saved face image, a selection frame 1103 for selecting the temporarily saved face image to be used for updating, and text 1104 for confirming whether or not to update. Note that text 1104 may be displayed in advance, or it may be displayed in the form of a dialog box triggered by the acceptance of a specific operation in the processing of S1004 described later. Of course, these are merely examples and do not limit this embodiment.

[0062] In S1003, the storage unit 304 displays an error message on the display unit 103 indicating that it does not have a temporarily stored face image or temporarily stored feature quantity. This error message may be dismissed, for example, by user action or after a certain period of time has elapsed. After the completion of the process in S1003, the storage unit 304 terminates the series of processes shown in Figure 10.

[0063] In S1004, the holding unit 304 receives user input via the display unit 103 and the operation dial 104. In S1005, the holding unit 304 determines whether or not the user has instructed the user to update the registered face image and registered feature quantities. If the holding unit 304 determines in S1005 that it has been instructed to update the registered face image and registered feature quantities, it proceeds to S1006. On the other hand, if the holding unit 304 determines in S1005 that it has not been instructed to update the registered face image and registered feature quantities (for example, it has received another instruction), it proceeds to S1007.

[0064] In S1006, the holding unit 304 updates the registered face image and registered feature quantity using the temporarily stored face image enclosed by the selection frame 1103 and the temporarily stored feature quantity corresponding to the temporarily stored image. In this process, updating means replacing the registered face image and registered feature quantity with the temporarily stored face image and temporarily stored feature quantity, or registering an additional temporarily stored face image and temporarily stored feature quantity. If the registered face image and registered feature quantity are replaced by this process, the registered face image and registered feature quantity are temporarily stored as the temporarily stored face image and temporarily stored feature quantity. After the completion of process S1006, the holding unit 304 completes the series of processes shown in Figure 10.

[0065] In S1007, the holding unit 304 determines whether or not the termination of the process has been instructed. If the holding unit 304 determines in S1007 that it has been instructed to terminate the process, it terminates the series of processes shown in Figure 10. On the other hand, if the holding unit 304 determines that the termination of processing has not been instructed in S1007, it proceeds to S1004. In this case, the processing from S1004 onward will be executed again.

[0066] (Effects and Benefits) With the above configuration, it becomes possible to continue using face images and features that have been added and improved in the past without discarding them, thus further improving tracking performance.

[0067] <Other Embodiments> In the first embodiment, the simultaneous fulfillment of the shooting intent criterion and the score criterion was used as the criterion for adding or updating face images and features. In contrast, in the second embodiment, the simultaneous fulfillment of the shooting intent criterion and the context criterion was used as the criterion for adding or updating face images and features. In other embodiments, when integrating the three criteria described above, namely the shooting intent criterion, the score criterion, and the context criterion, control may be applied that uses a composite index rather than a rule-based one. For example, the percentage of image frames that meet the criterion within a certain period may be used as the score value for that criterion, and the sum or multiplication of these score criterion values ​​may be applied as the integrated score. Such an integrated score may be compared with a predetermined threshold, and a decision on updating face images and features may be made based on the result. Furthermore, as exemplified by the shooting intent criterion, score criterion, and context criterion, if the decision to add or update face images and features is made based on at least two of the multiple criteria, the combination to be applied may be changed as appropriate. In other words, the combination of two or more criteria applied to the decision to add or update face images and features may be changed as appropriate depending on the scene or use case to which the technology relating to this disclosure is applied.

[0068] In the embodiments described above, various explanations were given based on the premise that imaging is performed by the user controlling the information processing device 100 (camera). However, in reality, situations in which imaging is automatically controlled by a robot or the like can also be envisioned. In such situations, for example, the simultaneous fulfillment of the score criterion and the context criterion may be used as the criterion for additional registration or update. Specifically, suppose that the camera is repeatedly performing automatic pan-tilt imaging in a musical and it is determined that the score criterion and the context criterion are simultaneously fulfilled. In such a case, it is conceivable that the camera would perform additional registration at that point and switch to tracking the target subject from that point onward. In this case, it is preferable that the user pre-set conditions for the context criterion.

[0069] In the first and second embodiments, examples were described in which, when no additional registered face images and additional registered features exist, the matching face image and matching features are used as the additional registered face images and additional registered features. Furthermore, in these embodiments, examples were also described in which, when additional registered face images and additional registered features exist, the additional registered face images and additional registered features are updated using the matching face image and matching features. On the other hand, it is not necessarily required that updates be permitted only when additional registered face images and additional registered features exist. For example, the registered face images and registered features may be updated using the matching face image and matching features without retaining the additional registered face images and additional registered features. In other words, in the examples shown in Figures 4 and 8, the process of updating the registered face images and registered features using the matching face image and matching features may be applied without applying the process in S409.

[0070] In the first and second embodiments, examples were described in which, if the criteria set by the additional registration / update determination unit 309 are met when additional registration has already been performed, additional registered face images and registered feature quantities are updated. In contrast, if the criteria are met when additional registration has already been performed, control may be applied to perform further additional registration instead of updating additional registered face images and registered feature quantities.

[0071] In the first and second embodiments, whether or not the shutter button 101 was pressed was used as an element for determining whether or not the shooting intent criterion was met. On the other hand, the information processing device 100 may be provided with a dedicated button for determining the shooting intent criterion. In this case, the determination of whether or not the shooting intent criterion is met can be made based on whether or not the button is pressed.

[0072] In the first and second embodiments, examples were described in which the additional registered face image and additional registered feature quantities are discarded after a certain period of time has elapsed since their last update. In contrast, for example, control may be applied such that the additional registered face image and additional registered feature quantities are discarded when the half-press or full-press state of the shutter is released. Another example is that the information processing device 100 may be provided with a dedicated button for discarding additional registration information, and control may be applied such that the additional registered face image and additional registered feature quantities are discarded immediately when the button is pressed. Yet another example is that a specific discard means may be set in advance, and control may be applied such that the additional registered face image and additional registered feature quantities are discarded immediately when the discard means is applied by user operation.

[0073] In the third embodiment, an example was described in which a registered face image and registered feature are updated using a temporarily stored face image and temporarily stored feature. In contrast, control may be applied such that the temporarily stored face image and temporarily stored feature are discarded when instructions are received from the user, for example, via the UI shown in Figure 11.

[0074] In the first and second embodiments, examples were described of displaying a face frame or a matching face image in a separate frame when score criteria or context criteria are met, or when additional registration / update is performed. On the other hand, it is conceivable that the user may be so focused on the target of tracking that it may be difficult to determine from this display alone whether the desired action has been performed. In light of such a situation, control such as outputting a sound effect from speaker 105 may also be applied to inform the user that the criteria have been met.

[0075] In the first and second embodiments, examples were described in which face images and feature quantities are immediately added or updated when the criteria set by the additional registration / update determination unit 309 are met. In contrast, control may be applied in which face images and feature quantities are not added at that time, and the matching face image and matching feature quantities of the subject that meets the criteria are temporarily stored in the storage unit 304 as a temporarily stored face image and temporarily stored feature quantities. In this case, the control described later in the third embodiment will be applied, making it possible for the user to register at their own discretion.

[0076] Furthermore, while Figure 1 illustrates an example where the information processing device 100 is comprised of a single device, each component may be comprised of multiple devices that are different from one another.

[0077] The present invention can take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, it may be applied to a system consisting of multiple devices (e.g., a host computer, interface devices, imaging devices, web applications, etc.) or to an apparatus consisting of a single device. Furthermore, the present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program. In this case, the program code read from the storage media itself realizes the functions of the embodiments described above, and the storage media on which the program code is recorded constitutes the present invention.

[0078] Furthermore, the disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing device comprising: an acquisition means for acquiring a face image from a target image; an extraction means for extracting feature quantities from the face image; a holding means for holding the face image and the feature quantities extracted from the face image; a calculation means for calculating a matching score by comparing a first feature quantity extracted by the extraction means and a second feature quantity held in the holding means; an authentication means for determining that authentication is successful when the matching score exceeds a threshold; a determination means for determining whether at least two of the following criteria are met: a shooting intent criterion related to determining the user's shooting intent, a score criterion related to determining the matching score, and a context criterion related to determining the shooting conditions; and an additional update means for adding the first face image acquired by the acquisition means and the first feature quantities to the holding means, or updating the second face image and the second feature quantities held in the holding means based on the first face image and the first feature quantities, when the determination means determines that the two or more criteria are met. (Configuration 2) The information processing device according to Configuration 1, characterized in that the shooting intent criterion is a criterion for determining whether at least one of the following series of conditions is met: the shutter button is pressed, the button related to determining the shooting intent is pressed, and the subject's face is within a predetermined range of the image, or whether at least one of the following series of conditions is met at a predetermined frequency or higher. (Configuration 3) The information processing device according to Configuration 2, characterized in that the determination means controls the display of the predetermined range when determining whether the shooting intent criterion is met based on whether or not the subject's face is present within a predetermined range of the image. (Configuration 4) The information processing apparatus according to any one of Configurations 1 to 3, characterized in that the score criterion is a criterion for determining whether the matching score satisfies the condition that it is less than an authentication threshold and greater than or equal to an acceptable threshold, or whether the condition is satisfied at a predetermined frequency or more. (Configuration 5) The information processing apparatus according to Configuration 4, characterized in that the allowable threshold is a value smaller than the authentication threshold. (Configuration 6) The information processing apparatus according to Configuration 4 or 5, characterized in that the determination means controls the display of a face frame surrounding the face of a subject whose matching score is less than the authentication threshold and greater than or equal to the tolerance threshold. (Configuration 7) The information processing apparatus according to any one of Configurations 1 to 6, characterized in that the context criterion is a criterion for determining whether at least one of the following conditions is met: the subject's face is depicted in a predetermined manner; the number of faces depicted in the image is less than or equal to a predetermined number; and the subject's face is located at a predetermined spatial position; or whether at least one of the following conditions is met at a predetermined frequency or higher. (Configuration 8) The information processing apparatus according to Configuration 7, wherein the determination means controls the display of a face frame surrounding the face of a subject that satisfies the context criteria. (Configuration 9) The information processing apparatus according to Configuration 7 or 8, characterized in that the determination means controls the display of auxiliary information indicating the extent to which the context criteria are met. (Configuration 10) The information processing apparatus according to any one of Configurations 1 to 9, wherein the additional update means adds the first face image and the first feature to the holding means when the holding means does not hold any other face images and other feature quantities that have been registered in advance. (Configuration 11) The information processing apparatus according to any one of Configurations 1 to 10, wherein the additional update means updates the second face image held in the holding means with the first face image, and updates the second feature amount held in the holding means based on the first feature amount, the matching score, and the camera parameters related to the image acquisition. (Configuration 12) The information processing apparatus according to any one of Configurations 1 to 11, wherein the additional update means, when a predetermined period has elapsed since the last additional registration of face images and feature quantities to the holding means or the last update of face images and feature quantities held by the holding means, stops using other face images and feature quantities registered in addition to the previously registered face images and feature quantities held by the holding means, and temporarily holds such other face images and feature quantities in the holding means as face images and feature quantities to be deleted after the predetermined period has elapsed. (Configuration 13) The information processing apparatus according to Configuration 12, wherein the holding means updates the second face image and the second feature quantity using the temporarily stored face image and feature quantity. (Configuration 14) The information processing apparatus according to any one of Configurations 1 to 13, wherein the additional update means adds and registers a plurality of first face images, the average of a plurality of first feature quantities extracted from each of the plurality of first face images, or a feature quantity obtained by aggregating the plurality of first feature quantities by a weighted average using the matching scores of the plurality of first feature quantities to the holding means. (Configuration 15) The information processing apparatus according to any one of Configurations 1 to 14, wherein, when there are multiple subjects in the image that are candidates for additional registration or updating of face images and feature quantities, the additional update means is characterized in that, among the multiple subjects, the subject with the highest matching score, the subject whose face is closer to the center of the image, or the subject whose face is most prominently displayed, is the subject for additional registration or updating of face images and feature quantities. (Configuration 16) The information processing apparatus according to any one of Configurations 1 to 15, wherein the holding means discards the held face image and feature quantities in accordance with instructions from the user. (Configuration 17) The information processing apparatus according to any one of Configurations 1 to 16, wherein the additional update means controls the face frame surrounding the face of a subject used for additional registration of a face image to the holding means or for updating a face image held in the holding means to be highlighted or to be displayed in a separate frame. (Configuration 18) The information processing apparatus according to any one of Configurations 1 to 17, wherein the additional update means controls the display of a separate frame that is enlarged to encompass the faces of each of the multiple subjects in the image when there are multiple subjects in the image that are candidates for additional registration or update of face images and feature quantities. (Configuration 19) The information processing apparatus according to any one of Configurations 1 to 18, characterized in that the determination means notifies by sound that the criteria have been met in at least one of the following cases: when it determines that two or more of the criteria have been met, or when an additional face image is registered to the holding means or an update is performed to a face image held in the holding means. (Method 1) A method for controlling an information processing device, comprising: an acquisition step of acquiring a face image from a target image; an extraction step of extracting feature quantities from the face image; a holding step of holding the face image and the feature quantities extracted from the face image in a holding unit; a calculation step of calculating a matching score by comparing a first feature quantity extracted in the extraction step and a second feature quantity held in the holding unit; an authentication step of determining that authentication is successful if the matching score exceeds a threshold; a determination step of determining whether at least two of the following criteria are met: a shooting intent criterion related to determining the user's shooting intent, a score criterion related to determining the matching score, and a context criterion related to determining the shooting conditions; and, if the determination step determines that the two or more criteria are met, an additional update step of adding the first face image acquired in the acquisition step and the first feature quantities to the holding unit, or updating the second face image and the second feature quantities held in the holding unit based on the first face image and the first feature quantities. (Program 1) A program for causing a computer to function as an information processing device, comprising: acquisition means for acquiring a face image from a target image; extraction means for extracting feature quantities from the face image; holding means for holding the face image and the feature quantities extracted from the face image; calculation means for calculating a matching score by comparing a first feature quantity extracted by the extraction means and a second feature quantity held in the holding means; authentication means for determining authentication success when the matching score exceeds a threshold; determination means for determining whether at least two of the following criteria are met: a shooting intent criterion related to determining the user's shooting intent, a score criterion related to determining the matching score, and a context criterion related to determining the shooting conditions; and, when the determination means determines that the two or more criteria are met, additional registration of the first face image acquired by the acquisition means and the first feature quantities in the holding means, or updating the second face image and the second feature quantities held in the holding means based on the first face image and the first feature quantities. [Explanation of Symbols]

[0079] 100 Information Processing Devices 302 Face Image Acquisition Unit 303 Feature Extraction Unit 304 Holding part 305 Score Calculation Unit 309 Additional Registration / Update Judgment Unit 310 Additional Registration / Update Section

Claims

1. A means for obtaining a face image from a target image, An extraction means for extracting feature quantities from the aforementioned facial image, A means for storing a face image and feature quantities extracted from the face image, A calculation means that calculates a matching score by comparing the first feature quantity extracted by the extraction means with the second feature quantity held in the holding means, Authentication means that determines authentication is successful when the matching score exceeds a threshold, A determination means for determining whether at least two of the following criteria are met: a shooting intent criterion related to determining the user's shooting intent, a score criterion related to determining the matching score, and a context criterion related to determining the shooting conditions. If the determination means determines that two or more of the above criteria are met, the addition update means either registers the first face image acquired by the acquisition means and the first feature quantity in the storage means, or updates the second face image and the second feature quantity held in the storage means based on the first face image and the first feature quantity. An information processing device characterized by having

2. The aforementioned criteria for photographic intent are: The shutter button is pressed, The button related to determining the shooting intention has been pressed, The subject's face is located within a predetermined area of ​​the aforementioned image, Whether or not at least one of the following conditions is met, Alternatively, whether at least one of the aforementioned set of conditions is met at a predetermined frequency or higher, The criteria for making that determination The information processing apparatus according to claim 1, characterized in that

3. The information processing apparatus according to claim 2, characterized in that the determination means controls the display of the predetermined range when determining whether the shooting intent criterion is met based on whether or not the subject's face is present within a predetermined range of the image.

4. The aforementioned scoring criteria are: Whether the matching score satisfies the condition that it is below the authentication threshold and above the acceptable threshold, Alternatively, whether the above condition is met at a predetermined frequency or higher, The criteria for making that determination The information processing apparatus according to claim 1, characterized in that

5. The information processing apparatus according to claim 4, characterized in that the allowable threshold is a value smaller than the authentication threshold.

6. The information processing apparatus according to claim 4, characterized in that the determination means controls the display of a face frame surrounding the face of a subject whose matching score is less than the authentication threshold and greater than or equal to the tolerance threshold.

7. The aforementioned contextual criteria are: The subject's face is captured in the specified way, The number of faces in the aforementioned image is less than or equal to a predetermined number, The subject's face is located in a predetermined spatial position, Whether or not at least one of the following conditions is met, Alternatively, whether at least one of the aforementioned set of conditions is met at a predetermined frequency or higher, The criteria for making that determination The information processing apparatus according to claim 1, characterized in that

8. The information processing apparatus according to claim 7, characterized in that the determination means controls the display of a face frame surrounding the face of a subject that satisfies the context criteria.

9. The information processing apparatus according to claim 7, characterized in that the determination means controls the display of auxiliary information indicating the extent to which the context criteria are met.

10. The information processing apparatus according to claim 1, wherein the additional update means adds the first face image and the first feature to the holding means when the holding means does not hold any other face images and other feature quantities that have been registered in advance other than the face images and feature quantities that have been registered in advance.

11. The aforementioned additional update means is: The second face image held in the holding means is updated with the first face image. Based on the first feature quantity, the matching score, and the camera parameters related to image acquisition, the second feature quantity held in the holding means is updated. The information processing apparatus according to claim 1, characterized in that

12. The aforementioned additional update means is: If a predetermined period has elapsed since the last addition of face images and feature quantities to the holding means or the last update of face images and feature quantities held in the holding means, the use of other face images and feature quantities registered in addition to the previously registered face images and feature quantities held by the holding means will be stopped. The other facial images and other feature quantities are temporarily stored in the storage means as facial images and feature quantities to be deleted after a predetermined period of time has elapsed. The information processing apparatus according to claim 1, characterized in that

13. The information processing apparatus according to claim 12, characterized in that the holding means updates the second face image and the second feature quantity using the temporarily stored face image and feature quantity.

14. The aforementioned additional update means is: Multiple first facial images and, The average of multiple first feature quantities extracted from each of the multiple first face images, or a feature quantity obtained by aggregating the multiple first feature quantities using a weighted average of the matching scores of the multiple first feature quantities, Register the above holding means in addition The information processing apparatus according to claim 1, characterized in that

15. The aforementioned additional update means is: If the aforementioned image contains multiple subjects that are candidates for additional registration or updating of facial images and feature quantities, Among the multiple subjects, the subject with the highest matching score, The subject whose face is closer to the center of the aforementioned image, The subject whose face is shown most prominently, One of the following will be subject to additional registration or updating of facial images and feature quantities. The information processing apparatus according to claim 1, characterized in that

16. The information processing apparatus according to claim 1, wherein the holding means discards the held face image and feature quantities in accordance with instructions from the user.

17. The information processing apparatus according to claim 1, characterized in that the additional update means controls the face frame surrounding the face of a subject used for additional registration of a face image to the holding means or for updating a face image held in the holding means to be highlighted or to be displayed in a separate frame.

18. The information processing apparatus according to claim 1, characterized in that, when there are multiple subjects in the image that are candidates for additional registration or updating of face images and feature quantities, the additional update means controls the display of a separate frame that is enlarged to encompass the faces of each of the multiple subjects.

19. The determination means is, If it is determined that two or more of the above criteria are met, When a face image is added to the holding means, or when a face image held in the holding means is updated, In at least one of the following cases, the system will notify that the criteria have been met by sound. The information processing apparatus according to claim 1, characterized in that

20. A method for controlling an information processing device, The acquisition step involves obtaining a face image from the target image, An extraction step of extracting features from the aforementioned facial image, A holding step in which a face image and feature quantities extracted from the face image are held in a holding unit, A calculation step which involves comparing the first feature quantity extracted in the extraction step with the second feature quantity held in the holding unit to calculate a matching score, An authentication step in which authentication is determined to be successful if the matching score exceeds a threshold, A determination step that determines whether at least two of the following criteria are met: a shooting intent criterion related to determining the user's shooting intent, a score criterion related to determining the matching score, and a context criterion related to determining the shooting conditions. If the determination step determines that two or more of the above criteria are met, an additional update step is performed to add the first face image acquired in the acquisition step and the first feature quantity to the storage unit, or to update the second face image and the second feature quantity stored in the storage unit based on the first face image and the first feature quantity. A control method for an information processing device, characterized by including the following:

21. Computers, A means for obtaining a face image from a target image, An extraction means for extracting feature quantities from the aforementioned facial image, A means for storing a face image and feature quantities extracted from the face image, A calculation means that calculates a matching score by comparing the first feature quantity extracted by the extraction means with the second feature quantity held in the holding means, Authentication means that determines authentication is successful when the matching score exceeds a threshold, A determination means for determining whether at least two of the following criteria are met: a shooting intent criterion related to determining the user's shooting intent, a score criterion related to determining the matching score, and a context criterion related to determining the shooting conditions. If the determination means determines that two or more of the above criteria are met, the addition update means either registers the first face image acquired by the acquisition means and the first feature quantity in the storage means, or updates the second face image and the second feature quantity held in the storage means based on the first face image and the first feature quantity. A program for causing an information processing device to function, characterized by having the following features.

Images