Information processing device, system, information processing method, and program

Abstract

To enable more efficient handling of biometric information and / or information extracted from the biometric information while protecting privacy of users.SOLUTION: Biometric information of a person is acquired. Consent information indicative of the consent status of the person regarding the use of feature quantities extracted from the biometric information is obtained. At least any of the acquired biometric information, identification information of the person corresponding to the biometric information, feature quantities extracted from the biometric information, and the acquired consent information is sent to a first external device according to the consent status.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a system, an information processing method, and a program. [Background technology]

[0002] Biometric authentication technology is known that identifies individuals based on acquired biometric information. However, biometric authentication may violate personal privacy depending on the application, so certain restrictions are required on how it can be used. Patent Document 1 discloses a method for obtaining consent from individuals before registering them for biometric authentication. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-119549 [Non-patent literature]

[0004] [Non-Patent Document 1] Deng, Jiankang, et al. “Retinaface: Single-shot multi-level face localization in the wild.” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a risk of inconvenience when handling biometric information and / or information extracted from the biometric information on multiple devices. In other words, when registering a subject of biometric authentication on each of multiple devices in Patent Document 1, consent must be obtained for each device, which poses a problem of cumbersome processing.

[0006] An object of the present invention is to enable more efficient handling of biometric information and / or information extracted from biometric information while protecting the privacy of users. [Means for solving the problem]

[0007] To achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: biometric information acquisition means for acquiring biometric information of a person, consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of features extracted from the biometric information, and first transmission means for transmitting at least one of the biometric information acquired by the biometric information acquisition means, identification information of the person corresponding to the biometric information, the features extracted from the biometric information, and the consent information acquired by the consent information acquisition means to an external device in accordance with the consent state. [Effects of the Invention]

[0008] It becomes possible to handle biometric information and / or information extracted from biometric information more efficiently while protecting the user's privacy. [Brief explanation of the drawings]

[0009] [Figure 1] 1A and 1B are diagrams for explaining situations in which an information processing apparatus according to an embodiment is used; [Figure 2] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing apparatus according to an embodiment. [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing apparatus according to an embodiment. [Figure 4] 10 is a flowchart showing an example of a registration process by the information processing apparatus according to the embodiment. [Figure 5] FIG. 2 is a diagram for explaining a registration dictionary in the information processing apparatus according to the embodiment. [Figure 6] 10 is a flowchart showing an example of authentication processing by the information processing apparatus according to the embodiment. [Figure 7]1A and 1B are views showing examples of images output by the information processing apparatus according to the embodiment. [Figure 8] FIG. 1 is a diagram showing an example of a system configuration according to an embodiment. [Figure 9] 10 is a flowchart showing an example of output processing by the information processing apparatus according to the embodiment. [Figure 10] FIG. 4 is a diagram for explaining a comparison of registered dictionaries between devices according to the embodiment. [Figure 11] FIG. 1 is a diagram showing an example of a system configuration according to an embodiment. [Figure 12] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing apparatus according to an embodiment. [Figure 13] FIG. 2 is a diagram for explaining a registration dictionary in the information processing apparatus according to the embodiment. [Figure 14] FIG. 10 is a diagram for explaining consent conditions by the information processing device according to the embodiment. [Figure 15] 1 is a flowchart showing an example of AI processing by the information processing device according to the embodiment. [Figure 16] 10 is a flowchart showing an example of transmission and reception of information between information processing devices. [Figure 17] FIG. 10 is a block diagram showing an example of the functional configuration of an information processing device according to a fourth embodiment. [Figure 18] 10 is a flowchart showing another example of transmission and reception of information between information processing devices. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [Embodiment 1] [overview] The information processing device according to the present embodiment acquires biometric information of a person, acquires consent information indicating consent information of the person regarding use of features extracted from the biometric information, and then transmits at least one of the acquired biometric information, identification information of the person corresponding to the biometric information, the features extracted from the biometric information, and the acquired consent information to an external device according to the consent state.

[0012] The following describes a method for restricting the use of biometric information in a biometric authentication registration process for an image of a person captured by a digital camera. In this embodiment, the digital camera is described as having an information processing device and an image capture device, but the present invention is not limited to this. For example, the digital camera having the image capture device and the information processing device may be connected via a network.

[0013] 1A and 1B are diagrams for explaining situations in which an information processing device according to this embodiment is used, and Fig. 1A shows a situation in which a person is photographed using a digital camera.

[0014] The information processing device 1 executes processing by each functional unit shown in FIG. 3, which will be described later. The information processing device 1 according to this embodiment is part of a digital camera, and controls the processing of the entire digital camera. In the following description, it is assumed that an imaging device 14, which is part of the digital camera, operates in response to instructions from the information processing device 1. Subject 2 is a first subject, a person in this case, whose image is captured by the imaging device 14 in the situation shown in FIG. 1(a). Subject 3 is a second subject, a person in this case, whose image is captured by the imaging device 14 in the situation shown in FIG. 1(a). In the following description, a person who is a subject in an image may be referred to as an "individual."

[0015] Button 11 is a shutter button that a user (not shown) of imaging device 14 presses to instruct information processing device 1 to take a photograph. Panel 12 is a display panel that displays the results of processing to the user of imaging device 14. Here, panel 12 displays an image to allow the user to check the composition or the state of the subject when taking a photograph. There is no particular limit to the type of panel 12 as long as it can display an image, and it may be, for example, a liquid crystal panel, an organic EL (Electro Luminescence) panel, or a projector.

[0016] The operation keys 13 are used to obtain input from the user. There are no particular limitations on the type of the operation keys 13 as long as they can obtain input from the user. For example, the operation keys 13 may be buttons provided on the imaging device 14, or may be a keyboard and mouse connected to the information processing device 1, or the panel 12, which is a touch panel, may also serve as the operation keys 13.

[0017] 1(b) shows a digital camera having an imaging device 14 and an information processing device 1, viewed from the opposite side of the orientation in FIG. 1(a). The imaging device 14 is an imaging unit consisting of a lens and a sensor, and captures an image in response to a request from the information processing device 1.

[0018] [Hardware configuration] 2 is a block diagram showing an example of a hardware configuration of the information processing device 1. The information processing device 1 includes a CPU 21, a ROM 22, a RAM 23, an external memory 24, an input unit 25, a display unit 26, a communication I / F 27, an I / O 28, and a communication bus 29.

[0019] The CPU 21 is a central processing unit that controls various devices connected to the system bus 29 and executes a program that implements the processing of the present invention. The ROM 22 stores a BIOS program and a boot program. The RAM 23 is a memory used as the main storage device of the CPU 21. The external memory 24 stores various data such as programs processed by the information processing device 1 and captured images.

[0020] The input unit 25 acquires input from the user via the buttons 11 or operation keys 13 mounted on the information processing device 1. The display unit 26 outputs the calculation results of the information processing device 1 to the display panel 12 in accordance with instructions from the CPU 21. The communication I / O 27 is a communication interface and performs information communication with the outside. The communication I / O 27 may perform communication via wired communication using a USB or the like, or via wireless communication such as a local area network or serial communication, and the type of communication is not limited. The I / O 28 inputs data from the imaging device 14.

[0021] [Explanation of the configuration diagram] FIG. 3 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. The biometric information acquisition unit 31 acquires biometric information. Here, the biometric information acquisition unit 31 acquires a captured image in which a subject appears as biometric information. The consent information management unit 32 acquires and manages consent information indicating whether or not a person has given consent to the use of the acquired biometric information. Here, the consent information management unit 32 can acquire and manage consent information based on input via the operation keys 13. Note that, hereinafter, the term "consent" simply refers to consent to the use of the biometric information as described above. Also, here, biometric information will be described as referring to the entire captured image in which the subject appears. However, for example, a rectangular area (bounding box) in which the subject is detected, or a rectangular area in which the subject's face is detected, etc. may also be used as biometric information, and is not particularly limited as long as it is information in an image that contains information about the subject.

[0022] The feature extraction unit 33 extracts feature amounts from biometric information. Here, the feature extraction unit can extract feature amounts from a captured image, which is biometric information, for a person who has consented to the use of feature amounts based on consent information.

[0023] The output unit 34 outputs (transmits) the consent information, biometric information, and feature amounts to an external device. The registration unit 35 registers the biometric information and consent information. Specifically, the registration unit 35 associates the biometric information, feature amounts extracted from the biometric information, and consent information and stores them in the external memory 24. The registration unit 35 can also associate a person's name obtained by input via the operation keys 13 with the feature amounts and consent information and record them in the external memory 24. Note that instead of outputting the consent information, biometric information, and feature amounts to an external device, the output unit 34 may output the consent information or the biometric information and feature amounts to an external device.

[0024] The matching unit 36 ​​compares the input biometric information with the biometric information registered by the registration unit 35, and determines whether the input biometric information corresponds to anyone of the registered biometric information. The matching result utilization unit 37 includes a control unit 38, a display unit 39, and a recording unit 40, and performs various processes using the matching results determined by the matching unit 36. The control unit 38 controls the imaging device 14 based on the matching results. The display unit 39 displays the matching results on the display panel 12. The recording unit 40 records the matching results in the external memory 24.

[0025] Each process according to this embodiment will be described below with reference to a flowchart, but the process procedure is not limited to that shown. For example, the order of processes may be changed as long as the same processing results are obtained, multiple processes may be integrated, a process described as a single step may be subdivided, or some processes may be omitted. Furthermore, each process may be individually extracted and function independently as a single functional element, and may be used in combination with processes other than those shown.

[0026] [Data structure explanation] The consent information according to this embodiment is information indicating whether or not there is consent regarding the use of feature quantities, as described above, and here, consent / non-consent is expressed as a Boolean value. The initial value of the consent information is set to no consent.

[0027] The registration dictionary according to this embodiment is a database in which information about people that the user of the information processing device 1 wants to match is registered. Here, the registration dictionary stores, for each person, the person's name (character string), consent information, facial image, and feature data in association with each other. Note that multiple pieces of facial image and feature data may be stored for each person.

[0028] [Explanation of the processing flow chart] The biometric authentication process performed by the information processing device 1 according to this embodiment is divided into a registration process for registering biometric information of a person to be authenticated, and an authentication process for determining which of the registered people the input biometric information corresponds to. Furthermore, the information processing device 1 according to this embodiment restricts the use of biometric information of a certain person in the registration process based on the person's consent information acquired based on the biometric information. Hereinafter, the process for restricting the use of biometric information in such registration process will be described with reference to FIG. 4.

[0029] Fig. 4 is a flowchart showing an example of registration processing by the information processing device 1. The processing described in Fig. 4 starts when, for example, a user of the information processing device 1 instructs execution of the registration processing (for example, by pressing the operation button 13 on the display panel 12 shown in Fig. 1). The user according to this embodiment is assumed to be a photographer who photographs a subject using a digital camera including the information processing device 1.

[0030] In S101, the information processing device 1 performs initialization processing for the registration processing. Here, the CPU 21 of the information processing device 1 reads a program from the external memory 24 of FIG. 2 and makes the registration processing, which will be described later, operable. As part of the initialization processing, the CPU 21 of the information processing device 1 also loads a registration dictionary from the external memory 24 and makes it available for addition. The processing described below in FIG. 4 is realized by the CPU 21 of the information processing device 1 executing the necessary programs.

[0031] In S102, the biometric information acquisition unit 31 acquires biometric information (here, an image of an individual's face captured by the imaging device 14 in FIG. 1). At this time, the biometric information acquisition unit 31 controls the imaging device 14 using known imaging technology so that the focus and exposure are suited to the target's face, and acquires the captured image. Note that the following description will be given assuming that the captured image that becomes the biometric information contains one subject, but the captured image may contain multiple subjects, and the processing described below may be performed on each of them individually.

[0032] In S103, the consent information management unit 32 acquires and manages consent information for individuals (subjects) appearing in the captured images acquired in S102. As described above, the consent information acquired here indicates whether or not there is consent to the use of images showing the individual's face, i.e., their biometric information.

[0033] The "use of biometric information" according to this embodiment includes at least one of photographing and recording a face image, extracting features from the face image, performing a registration process to register the features, performing an authentication process based on the features (here, a process to identify a person), and performing some kind of control using the results of the authentication process. Details of these usage methods will be described later. The information processing device 1 according to this embodiment restricts the use of this biometric information based on consent information.

[0034] The consent information management unit 32 according to this embodiment can acquire consent information based on, for example, a user's operation via the display panel 12 or the operation keys 13. Specifically, the consent information management unit 32 can display, on the display panel 12, a message indicating that biometric information will be used and a message for accepting a choice as to whether or not to agree to the use, and acquire the result of the choice as consent information.

[0035] It should be noted that the user name is assumed to be set in advance (for example, based on the user's login or the user's input during operation). However, in order to prevent so-called spoofing, in which a person other than the user arbitrarily performs an operation agreeing to the user's consent, the user may be set based on a captured image different from the biometric information. For example, an imaging device (not shown) may be further provided to capture an image of the operator of the operation keys 13, and the user may be set based on the image captured by such an imaging device (by a known person recognition process). In such a case, it may be additionally determined whether the set user and the person in the biometric information acquired in S102 are the same person, and if they are not the same person, the process of FIG. 4 may be terminated at that point.

[0036] Furthermore, although the explanation has been given assuming that consent information is acquired based on user input via the operation keys 13, the present invention is not limited to such processing as long as it is possible to acquire whether or not a user has consented. For example, if it is detected that a user has performed a predetermined action (e.g., a gesture indicating an instruction by voice, etc.), it may be determined that the user has consented to the use of their biometric information, and image capture by the image capture device 14 and processing for using the biometric information (e.g., authentication processing) may be performed. Such processing allows consent to be expressed simply by making a gesture, thereby improving convenience. The gesture may be, for example, a peace sign using the hand, or a gesture of raising or lowering the hand, and the gesture may be recognized using known motion recognition technology.

[0037] In S104, the feature extraction unit 33 determines whether or not the user has consented to the use of the biometric information based on the consent information acquired in S103. If the user has consented, the process proceeds to S105, and if not, the process proceeds to S107.

[0038] In S105, the feature extraction unit 33 extracts features from the biometric information of a user who has consented to the use of the biometric information. In this embodiment, the biometric information is a captured image of an individual's face, and the feature extraction unit 33 identifies the position of the face in the captured image and then extracts features from the face. Any known image processing technology can be used in the process of identifying the position of the face in the captured image and the process of extracting features from the person's face. For example, the technology described in Non-Patent Document 1 may be used to identify the position of the face in the captured image. Furthermore, for example, a method using deep learning may be used to extract features from the person's face, or a method such as Local Binary Pattern (LBP) or HoG (Histogram Oriented Gradient) may be used.

[0039] In S106, the registration unit 35 associates the extracted feature with the biometric information, consent information, and personal name, and registers them in a registration dictionary. The registration unit 35 can acquire the personal name by accepting user input from an input unit (not shown) via the operation keys 13 in FIG. 1. The format of the personal name according to this embodiment is not particularly limited as long as it is set so as to enable distinction between registered persons. For example, the personal name may be a real name, a nickname, a number, a symbol, or the like. Furthermore, although this embodiment will be described assuming that the registration unit 35 registers the consent information, the processing may be performed by a different functional unit, such as the consent information management unit 32 registering the consent information.

[0040] In S107, the registration unit 35 records the biometric information, consent information, and person's name in association with each other. Here, feature extraction is not performed for individuals who have not given consent, and only biometric information, i.e., a face image, is registered for the consent information and person's name.

[0041] In S108, the registration unit 35 determines whether to end the registration process. Here, for example, if the user has input to end the registration process, it may be determined that the registration process is to be ended. If it is determined that the registration process is to be ended, the process of FIG. 4 ends; if not, the process returns to S102.

[0042] FIG. 5 is a diagram showing an example of the contents of the registration dictionary after the registration process. In FIG. 5, each column displays a "No." indicating an index, a "Person's Name" character string indicating a person's name, "Consent Information" that displays a check mark if consent to the use of biometric information is indicated in the consent information, a "Facial Image" displaying an image of the subject, and a "Feature Amount" that schematically displays the extracted and recorded feature amount. Each row and column represents one person, and three people, "Mr. A," "Mr. B," and "Mr. C," are registered in Nos. 1 to 3, respectively. In FIG. 5, consent to the use of biometric information has been obtained for "Mr. A" and "Mr. C," and feature amounts have been extracted and registered. Consent has not been obtained for "Mr. B," and feature amounts have not been registered. This registration dictionary is used in the authentication process, which will be described later.

[0043] [effect] This type of processing can restrict the use of biometric information for people who have not given their consent. In particular, by preventing feature registration, it is possible to prevent individuals from being identified through the authentication processing described below (i.e., restricting face authentication processing). Furthermore, by registering only an image when consent has not been given, if consent is obtained later, it becomes possible to extract and register feature information without obtaining new biometric information, improving convenience (since it is no longer necessary to obtain an image and consent at the same time, convenience is improved for both the user and the person being registered).

[0044] [Variation 1-1] [Variations in how consent is obtained] In the present embodiment, the information processing device 1 acquires consent from a subject by displaying on its display panel 12 a message indicating that biometric information will be used and a message (an image prompting consent) that accepts the user's consent. However, the method of acquiring consent information is not limited to this, as long as it is possible to confirm whether the user has consented. For example, the information processing device 1 may acquire consent information based on an input from an application on a mobile terminal such as a smartphone (not shown). In such a case, the information processing device 1 may communicate with the mobile terminal and request the mobile terminal to transmit consent information when acquiring consent information (S103 in FIG. 4). Upon receiving the request, the mobile terminal displays an image prompting consent on its screen and receives input indicating consent from the user of the mobile terminal, thereby acquiring consent information and transmitting it to the information processing device 1. At this time, to detect impersonation and prevent consent by a different person, the information processing device 1 may verify that the person being registered and the user of the mobile terminal are the same person (perform identity authentication). For this purpose, the information processing device 1 may perform identity authentication on its own or using a security function of the mobile terminal. For example, the information processing device 1 can transmit a facial image of a person to be registered to a mobile terminal, and the mobile terminal can use a known facial authentication method to verify whether the user and the facial image are the same person. The facial image of the user of the mobile terminal is captured, for example, by the mobile terminal's internal camera. With this configuration, consent obtained through impersonation can be avoided and consent information can be obtained from the subject. Note that the personal authentication is not limited to such facial authentication, and can be any authentication process using, for example, fingerprint authentication provided in the mobile terminal, or authentication process using an electronic certificate, etc.

[0045] [Variation 1-2] [Processing when no image is registered] In the present embodiment, it has been described that a facial image is registered even if consent is not obtained. However, it is also possible to prevent registration of a facial image without consent. According to such processing, by not storing a facial image without consent, it is possible to perform processing that takes user privacy into greater consideration (respects individual wishes more). Note that, compared to consent information, facial images can often be acquired relatively easily through imaging processing or acquisition processing via SNS (Social Networking Service), etc. Therefore, if priority is given to light processing speed and ease of operation in the registration processing, facial images may be registered regardless of whether consent is obtained, as described in S107.

[0046] [Variation 1-3] [Images and names are not required for the registered dictionary] In the first embodiment, it has been described that information such as a facial image and a person's name is linked to consent information and registered in the registration dictionary. However, the information registered in the registration dictionary for consent information is not particularly limited to information necessary for matching, and a person's name is not essential. For example, in the registration dictionary, only a facial image may be linked to consent information and registered, or a feature may be linked to consent information and registered, or both of these may be linked to consent information and registered.

[0047] [Embodiment 2] The information processing device 1 according to the first embodiment restricts the use of biometric information in the above-described registration process based on the presence or absence of consent. The information processing device 1 according to the second embodiment acquires consent information in the same manner as the first embodiment, and restricts the use of biometric information in the authentication process based on the presence or absence of consent. The information processing device 1 according to this embodiment has the same hardware configuration and functional configuration as the information processing device 1 according to the first embodiment, and can execute the same processes, so redundant explanations will be omitted.

[0048] 6 is a flowchart showing an example of authentication processing performed by the information processing device 1 according to this embodiment. For example, the authentication processing is started when a user of the information processing device 1 instructs the execution of the authentication processing (for example, by pressing the operation button 13 on the display panel 12 shown in FIG. 1). Note that the authentication processing according to this embodiment is performed after the registration processing described in the first embodiment is executed.

[0049] In S111, the information processing device 1 performs initialization. The initialization process is performed for the registration process. Here, the information processing device 1 reads a program from the external memory 24 of FIG. 2 and makes the authentication process, which will be described later, operable. As part of the initialization process, the information processing device 1 also loads a registration dictionary from the external memory 24 and makes it available for use.

[0050] In S112, the biometric information acquisition unit 31 acquires biometric information (here, an image of an individual's face from the imaging device 14 in FIG. 1). Here, the biometric information acquisition unit 31 acquires, as biometric information, an image of a candidate to be recorded by the digital camera from the imaging device 14. The captured image acquired here is displayed on the display panel 12 as a live view in the processing described below. The user (photographer) can determine the composition of the image to be recorded or the timing to press the shutter button 11 while checking the live view. Note that the captured image acquired here may include multiple faces.

[0051] In S113, the feature extraction unit 33 extracts features from the biometric information. Here, as described in the registration process, the feature extraction unit 33 identifies the position of the face in the captured image and then extracts features from the face of the user who has agreed to the use of their biometric information. If the image includes multiple faces, features are extracted from each of the multiple faces.

[0052] In S114, the matching unit 36 ​​identifies a person by comparing the feature values ​​extracted in S113 with the feature values ​​of individuals registered in the registration dictionary. Here, the matching unit 36 ​​compares the feature values ​​extracted in S113 with the biometric information registered by the registration unit 35 to determine whether the extracted feature value matches or does not match any of the registered biometric information. For example, the matching unit 36 ​​calculates the similarity between the extracted feature value and the registered feature value. If the similarity value exceeds a predetermined threshold, the matching unit 36 ​​determines that the extracted person is the individual. If no person exceeds the threshold, the matching unit 36 ​​determines that the extracted person is not a person. Any known method can be used to calculate the similarity. For example, cosine similarity or L2 distance may be used as the similarity. Since the L2 distance is a measure in which the value decreases as the distance between feature values ​​decreases, the reciprocal of the L2 distance may be converted into the similarity value. Here, as described in the description of FIG. 5 of the registration process, since "Mr. B" does not have any registered feature values, matching cannot be performed for Mr. B. In addition, in the case of Mr. B, since he has not consented to the use of his biometric information, no features are extracted.

[0053] In the following S115 to S117, the matching result utilization unit 37 performs various processes using the matching result output by the matching unit 36. These processes will be described in detail below. In S115, the display unit 39 displays the matching result on the display panel 12. FIG. 7 is a schematic diagram showing an example of the matching result displayed by the display unit 39. In FIG. 7, a face frame G101 of a person displayed as "Unknown", a face frame G102 of a person displayed as "Mr. C", and a face frame G103 displayed as "Unknown" are arranged on the display panel 12.

[0054] Here, G101 is "Mr. B," but as mentioned above, "Mr. B" has not consented to the use of his or her biometric information, so the matching process was not performed. Therefore, the result is determined to be "Unknown," meaning that the person is not a member of the registered dictionary. Also, here, G102 is "Mr. C," and as mentioned above, "Mr. C" has consented to the use of his or her biometric information, so the result is determined to be "Mr. C" by matching the feature values ​​of "Mr. C" in the registered dictionary. G103 indicates that the person was not matched with anyone registered in the registered dictionary and was determined to be "Unknown." In other words, in this example, only the name of "Mr. C," who consented, is displayed; the matching results for other people are displayed as "Unknown," with no indication of their identity. In this way, the use of biometric authentication can be restricted by preventing matching of people without consent. The display shown in Figure 7 is a live view display on a digital camera, and the user can decide whether to save the image by pressing the shutter button while viewing this display.

[0055] In S116, the control unit 38 controls the imaging parameters of the imaging device 14 (here, focus control and exposure control) based on the matching result. For example, the control unit 38 can control the imaging parameters so that the focus and exposure are controlled for the face of the person whose name has been identified based on the matching result. In other words, the control unit 38 may control the imaging parameters so that a person whose name has not been identified (who has not consented to the use of their biometric information) is not used as a reference in the control of imaging parameters. Here, the control of focus and exposure for a specific person can be performed using known AF / AE technology, and a detailed description thereof will be omitted. Note that, here, if there are multiple people whose names have been identified in the captured image, it is assumed that it is possible to determine in advance which of the people to focus and expose (for example, a priority can be set for each person).

[0056] In S117, the recording unit 40 records the captured image using the matching result. Here, the recording unit 40 captures a captured image when the user presses the shutter button 11, and records only the matching results of persons who have given their consent as tag information for the captured image (i.e., the recording unit 40 can control so that the matching results of persons who have not given their consent are not recorded in association with the image). In the example of FIG. 7, as described above, information about "Mr. C," who has been identified with consent, is recorded as tag information. As shown in FIG. 7, information about the face frame of G102 for "Mr. C" may also be recorded together with the captured image. According to this processing, it is possible to obtain a captured image in which the matching results are linked and recorded only for persons who have given their consent.

[0057] In step 118, the matching result utilization unit 37 determines whether or not to terminate the authentication process. Here, for example, if the user has input to terminate the authentication process, it may be determined that the registration process is terminated. If it is determined that the authentication process is terminated, the process in FIG. 6 is terminated; if not, the process returns to S112.

[0058] [effect] According to this type of processing, it is possible to restrict the use of biometric information for persons who have not given their consent. In particular, it is possible to restrict the use of biometric authentication processing or processing results for persons who have not given their consent. In the authentication processing, biometric authentication processing is performed on the input face image to determine whether or not consent has been given, but the results are not used for persons who have not given their consent, which effectively restricts the biometric authentication processing. Therefore, it is possible to restrict the biometric authentication processing while respecting the individual's will.

[0059] [Variation 2-1] [Extract features regardless of consent] In the registration process according to this embodiment, the feature information of a person who has not given consent is not registered. However, the feature information of a person may be registered even if the person has not given consent. In this case, the information processing device 1 restricts the use of biometric information by not using the matching results of a person who has not given consent during authentication processing, rather than not extracting feature information from the person.

[0060] [Variation 2-2] [Variations of consent information] In the present embodiment, the consent information has been described as information indicating only whether or not consent has been given, but it may also be possible to specify the type of consent, such as what consent is given for. The type of consent may be, for example, consent to using acquired biometric information to control imaging parameters such as AF / AE for the subject, consent to recording images of the subject, or consent to displaying the matching results, or other consent to various processes. Such processing can achieve restrictions that respect the individual's wishes in more detail.

[0061] [Variation 2-3] [Features are extracted during authentication without registering them] In this embodiment, the feature amounts of the registered dictionary are described as being extracted and stored during the registration process, but the feature amounts extracted during the authentication process may also be registered in the registered dictionary. In that case, during the authentication process, the consent information of the registered dictionary can be referenced, and the feature amounts can be extracted only from persons who have consented, and compared with the input feature amounts.

[0062] [Variation 2-4] [If no match is found during the matching process, register as no consent] In the comparison process, if the extracted feature is determined to match no one by comparing it with the registered dictionary, the information processing device 1 may additionally register the person corresponding to the feature in the registered dictionary as a person without consent. In this case, in the example of FIG. 7, G103 is a person determined to match no one of the people registered in the registered dictionary, so the facial image of G103 is registered. Here, the name set for a person without consent is assumed to be a predetermined character string that is set in advance. In addition, in this case, the person determined to match no one may be registered in association with the feature extracted from the facial image. According to this process, it is possible to explicitly identify the person as having not given consent by looking at the registered dictionary, thereby improving convenience for the user of the information processing device 1.

[0063] [Variation 2-5] [If consent is not given, re-acquire] Furthermore, the information processing device 1 may redisplay a display for accepting a person who has not given consent, asking them to choose whether or not to consent to the use of their biometric information, and prompt them to consent to the use of their biometric information again (re-acquire the consent). That is, the consent information management unit 32 of the information processing device 1 can accept an instruction to change the consent information. In this case, if consent is obtained by instructing the subject to make a gesture, consent information can be obtained simultaneously from multiple people in the photo, which is highly convenient. For a person whose consent information indicating consent has been reacquired, features are extracted from the facial image, and the features and consent information are newly registered in the registration dictionary. Furthermore, when previously acquired consent information indicates non-consent, the consent information management unit 32 of the information processing device 1 can also accept an instruction to change the consent status from non-consent to consent. In response to the consent information management unit 32 accepting the above-mentioned change instruction, the feature extraction unit 33 of the information processing device 1 extracts features from the registered biometric information (facial image) of the person corresponding to the consent information. Then, the registration unit 35 associates the facial image with the changed consent information and registers it. According to this modified example, there is no need to obtain consent and acquire a facial image at the same time, or it becomes possible to change a consent decision once made, thereby improving convenience for users who obtain consent and for people who give consent.

[0064] [Variation 2-6] [Variations in the use of authentication results] Furthermore, the recording of an image of a person who has not given consent may be restricted. For example, in the example of FIG. 7, the face frames G101 and G103 of the person who has not given consent may be filled in (for example, with a single color of black) and output as an image. In this way, the information processing device 1 can restrict the use of biometric information by outputting an image from which areas containing biometric information have been deleted. According to such processing, an image can be recorded with the consent of all people whose face areas appear in the image. Also, for example, the information processing device 1 may be configured not to record an image unless the consent of all people appearing in the image has been obtained.

[0065] Although this method of restricting the recording of images is effective in terms of respecting the will of the individual, it is a strong restriction. From this perspective, for example, images can be recorded (output as is) for people who have not given their consent, but images of people who have refused to use their biometric information cannot be recorded (by blacking out the face area), so that images can be output that, to some extent, balances convenience and respect for the will of the individual.

[0066] Furthermore, in the present embodiment, when displaying the matching result, the explanation has been given assuming that the presence or absence of consent is displayed on a live view image as shown in FIG. 7. However, such display of the presence or absence of consent is not limited to the live view, but may be displayed on an output image. The information processing device 1 in this embodiment has a function for displaying recorded captured images, as is generally provided in digital cameras, and consent information may be superimposed on such captured images in the same manner as in the live view of FIG. 7. This type of processing makes it possible to check the presence or absence of consent for people whose images have been captured in the past.

[0067] Furthermore, as described in Modification 2-2, when obtaining consent for each process, whether or not to use the matching results for each process may be switched depending on the content of the consent. For example, if a person consents to the recording of an image but does not consent to the adjustment of focus or exposure using the matching results, restrictions on the use of biometric information are controlled for each such process. In this case, for example, the focus or exposure is not adjusted for that person, and an image of that person is recorded. In this way, restrictions can be set that are more in line with the person's intentions.

[0068] [Variation 2-7] [Variations of information processing devices] Although the information processing device 1 according to this embodiment has been described as being part of a digital camera, the present invention is not limited to such a configuration as long as it can perform similar processing. For example, the information processing device 1 may be a smartphone with a camera or a network camera with a pan-tilt-zoom function that allows the angle of view to be adjusted. In this case, one possible method for using the matching results is to control the pan-tilt-zoom to capture an identified individual. When the information processing device 1 is used for such purposes, restricting the use of biometric information for persons who have not consented can prevent invasion of privacy and prevent misuse. Furthermore, when the information processing device 1 is a network camera, a separate server may be provided to manage the network camera and record captured video, and the server may perform the consent information acquisition or registration process. In such a configuration, when acquiring consent information, a UI for acquiring consent information on the server, i.e., a display indicating the use of biometric information and a display allowing the user to choose whether or not to consent, may be displayed on a display device. The user may then input their consent using an input device such as a mouse or keyboard.

[0069] [Embodiment 3] In this embodiment, an example will be described in which consent information, biometric information, and feature amounts registered by the information processing device 1 according to embodiment 1 are output to another device. In addition, a case will be described in which a digital camera, like the information processing device 1, is used as the other device used here.

[0070] 8 is a schematic diagram showing an example of use of the information processing device 1 according to this embodiment. The information processing device 1 is the same as that in the first embodiment, so a duplicated description will be omitted.

[0071] The network 15 is a network through which the devices communicate with each other. The network 15 may be, for example, a local area network (LAN), but the form of the network is not particularly limited as long as it can connect the devices so that they can communicate with each other. For example, the network 15 may be wireless or wired.

[0072] The information processing device 16 is a second information processing device to which the information processing device 1 according to this embodiment outputs consent information, and is assumed to be a digital camera in this example. The information processing device 16 according to this embodiment has the same hardware configuration and functional configuration as the information processing device 1, and redundant explanations will be omitted. The information processing device 16 is also capable of sending and receiving requests and data from other devices.

[0073] The mobile terminal 17 is a smartphone to which the information processing device 1 according to this embodiment outputs consent information. The mobile terminal 17 is capable of sending and receiving requests and data from other devices, similar to the information processing device 16. The information processing device 1 according to this embodiment outputs, in addition to consent information, feature amounts of persons who have given consent to the mobile terminal 17. This allows the mobile terminal 17 to use the acquired feature amounts of persons who have given consent. Conversely, for persons who have not given consent, the information processing device 1 cannot output consent information or feature amounts.

[0074] The server 18 is a server to which the information processing device 1 outputs consent information, and functions as an external device that processes biometric information. The server 18 according to this embodiment is capable of sending and receiving requests and data from other devices, similar to the information processing device 16. Furthermore, the information processing device 1 outputs the feature amounts of the consenting person to the server 18 in addition to the consent information. This allows the server 18 to use the acquired feature amounts of the consenting person.

[0075] Fig. 9 is a flowchart showing an example of output processing by the information processing device 1 according to this embodiment. The processing described in Fig. 9 starts when, for example, a user of the information processing device 1 instructs execution of the output processing (for example, by pressing the operation button 13 on the display panel 12 shown in Fig. 1). Here, an example will be described in which the information processing device 1 described in Fig. 8 outputs data included in the registered dictionary, that is, consent information, biometric information, and feature amounts, to the information processing device 16.

[0076] In S121, the output unit 34 acquires the registered dictionary registered by the registration unit 35. As described above, the registered dictionary includes consent information, biometric information, and feature amounts. As described in the first embodiment, this consent information is acquired by the consent information management unit 32. The output unit 34 according to this embodiment acquires the consent information held in the registered dictionary, but may also acquire consent from an individual as in the first embodiment.

[0077] At S122, the output unit 34 establishes a connection with the destination information processing device 16 and makes it ready to output data. At S123, the output unit 34 references the registered dictionary held by the destination information processing device 16 and compares it with the registered dictionary held by the information processing device 1 to detect differences. FIG. 10 is a schematic diagram illustrating the above-described difference detection and synchronization process performed by the information processing device 1 according to this embodiment. FIG. 10(a) shows the registered dictionary held by the information processing device 1, and FIG. 10(b) shows the registered dictionary held by the destination information processing device 16. Here, the output unit 34 compares the registered dictionaries and detects differences. Here, the process of detecting differences is a process of identifying information not stored in the external device as a difference. In this example, although "Mr. A" at No. 1 in each registered dictionary is the same person, the difference is that different facial images and feature values ​​are registered in each. Furthermore, for Nos. 2 and 3 in FIG. 10(a), the person not registered in FIG. 10(b) is also a difference.

[0078] In order to determine whether or not the two people are the same person, the matching unit 36 ​​can perform the above-mentioned matching process on each feature. The matching unit 36 ​​may also store unique numbers, symbols, etc. that identify individuals in a registered dictionary, and determine whether or not the two people are the same person by comparing such information. As mentioned above, the "Person's Name" field contains a person's name or nickname, and is not necessarily a unique number or symbol, so it may not be used to determine whether or not the two people are the same person.

[0079] In S124, the output unit 34 displays to the user which information to output, and then confirms whether or not it is OK to output the information. Here, the output unit 34 displays a screen such as that shown in Fig. 10(a) and Fig. 10(b). In this example, the output unit 34 outputs the facial image and feature amount of No. 1 and the facial image and feature amount of No. 3 from the information processing device 1. Along with this display, the information processing device 1 receives input from the user via an input unit (not shown) regarding whether or not to output the information.

[0080] In S125, if there is an input from the user permitting output (Yes in S125), the output unit 34 proceeds to S126, otherwise it terminates the processing in Fig. 10. In S126, the output unit 34 outputs the consent information, biometric information, and feature amounts.

[0081] FIG. 10(c) shows the registration dictionary of the information processing device 16 after output. The registration dictionary shown in FIG. 10(c) reflects information from the registration dictionary (a) of the information processing device 1, which was not held before the output. In this embodiment, the information processing device 1 also acquires the difference from the information processing device 16, and synchronization with the registration dictionary held in the information processing device 16 is performed. Therefore, the registration dictionaries of the information processing device 1 and the information processing device 16 are synchronized, and both information processing devices store the registration dictionary shown in FIG. 10(c). Note that, once the registration dictionary of the information processing device 1 is output, the registration dictionaries of both information processing devices do not necessarily need to be synchronized, and only the registration dictionary held in the information processing device 16 may be updated.

[0082] In FIG. 10(c), multiple facial images and feature amounts are registered for one person ("Mr. A"), and matching can be performed by matching each feature amount and calculating a representative matching result from the multiple matching results. Any known method for matching data can be used to calculate the representative matching result. For example, matching can be performed by selecting the matching result with the highest similarity or by averaging the similarities. According to this processing, by registering multiple facial images and feature amounts for one person, matching can be performed based on facial images and feature amounts of that person taken under multiple shooting conditions, thereby improving matching accuracy.

[0083] Fig. 16 is a diagram showing an example of transmission and reception of information between the information processing device 1 and the information processing device 16, which is executed in S123 to S126. The processing in Fig. 16 is started when a connection is established between the information processing device 1 and the information processing device 16 in S122, for example.

[0084] In S140, the information processing device 1 requests the information processing device 16 to transmit a registered dictionary. In S141, the information processing device 16 transmits part or all of the registered dictionary held by the information processing device 16 to the information processing device 1, and in S142, the information processing device 1 receives part or all of the registered dictionary transmitted from the information processing device 16. The information of the registered dictionary transmitted and received in S141 and S142 includes, for example, one or more of biometric information, person identification information, feature amounts, and consent information. Here, it is assumed that the information of the registered dictionary transmitted and received in S141 and S142 is the same type of information (i.e., consent information) as the item output in S126 (corresponding to S146 described later) (i.e., consent information, biometric information, and feature amounts in the example described with reference to FIG. 9).

[0085] At S143, the information processing device 1 compares the registered dictionary received from the information processing device 16 with the registered dictionary held by the information processing device 1. At S143, the process of detecting differences by comparing the registered dictionaries, which was described as being performed at S123, is executed. At S144, the information processing device 1 executes a process of synchronizing the registered dictionary held therein with the received registered dictionary.

[0086] Here, the synchronization process involves the information processing device 1 adding up the information for each entry in the registration dictionary for each person. For example, in FIG. 10(c), a synchronized registration dictionary is generated for the person's name "Mr. A," including both the facial image and feature values ​​contained in the registration dictionary of the information processing device 1 and the facial image and feature values ​​contained in the registration dictionary of the information processing device 16. Note that the synchronization process is not limited to this, and any known method used to integrate two databases may be used. For example, information on the last update date and time for each entry in the registration dictionary may be linked and registered, and the registration dictionary may be synchronized between multiple registration dictionaries by overwriting each entry with the information of the most recent update date and time. Furthermore, this synchronization process may involve transmitting synchronized data and overwriting the data in the destination device, or it may involve transmitting only the difference from the destination (data not stored in the destination) and performing an update process on the destination device. When only information on the difference with the destination is transmitted in the synchronization process, information that does not differ between the information processing device 1 and the information processing device 16 (for example, feature amounts if the feature amounts are equal) is not transmitted or received in the synchronization process.

[0087] In S145, the information processing device 1 outputs information about the synchronized registered dictionary to the information processing device 16, in S146 the information processing device 16 receives the synchronized registered dictionary output in S145, and in S147 the information processing device 16 performs a comparison process and a synchronization process based on the registered dictionary received from the information processing device 1, thereby ending the process. The comparison process and synchronization process performed in S147 are equivalent to the processes in S143 and S144. Here, the output process in S145 corresponds to S126 in FIG. 9, and is assumed to output information with consent for a person. Therefore, when the process of FIG. 16 is performed for each of multiple people, the type of information transmitted and received may differ for each person depending on the items with consent.

[0088] In FIG. 16 , the information processing device 1 receives the registered dictionary before synchronization, performs synchronization processing, and outputs the synchronized registered dictionary to the information processing device 16. However, as long as information is transmitted and received between the information processing device 1 and the information processing device 16 and the information contained in the registered dictionary is synchronized and updated, the device performing the synchronization processing is not limited to the information processing device 1. For example, the information processing device 1 and the information processing device 16 may be interchanged in the processes described in FIG. 16 . Also, in this embodiment, as shown in S140 of FIG. 16 , the information processing device 1 transmits a request for a registered dictionary to the information processing device 16. However, the information processing device 1 may transmit the registered dictionary to the information processing device 1 without a request. Also, in this embodiment, as shown in S145 of FIG. 16 , the information processing device 1 transmits the synchronized registered dictionary to the information processing device 16 without a request from the information processing device 16. However, the information processing device 1 may transmit the synchronized registered dictionary in response to a request from the information processing device 16. Also, before executing steps S140 to S147 shown in FIG. 16 , a step of obtaining confirmation from the device user may be provided. In addition, in the present embodiment, the explanation has been given mainly on an example in which the information processing device 1 synchronizes the registered dictionaries and then transmits the synchronized registered dictionaries to the information processing device 16, but the information processing device 1 and the information processing device 16 may exchange their registered dictionaries and then each may simultaneously execute synchronization processing in parallel. As such, it should be noted that there are various variations on the processing in Fig. 16 and it is not limited to a specific method.

[0089] [effect] With this configuration, the consent information, biometric information, and feature amounts acquired by the information processing device 1 can be output to another device. By outputting the consent information to another device and making it available on the output destination device, it is possible to eliminate the need to acquire consent information by making some kind of contact with the person to be processed. For example, in a group of family or friends who want to share consent information, the consent information can be synchronized between the information processing devices, thereby reducing the number of cumbersome consent acquisitions.

[0090] Furthermore, by outputting the biometric information and feature quantities together with the consent information, the procedure of acquiring the biometric information and feature quantities again on the output destination device can be omitted. By outputting the feature quantities, the feature quantities can be used for facial recognition on another device. Additionally, by outputting the biometric information (here, a facial image), the feature quantities can be extracted again from the facial image when the facial recognition model is updated. Furthermore, with this configuration, the user of the device can visually check the registration status, such as which individuals have consented or are registered, thereby improving convenience from a management perspective.

[0091] [Variation 3-1] [Output Destination Variations] In the present embodiment, the information processing device 1 and the information processing device 16 are both digital cameras. However, other devices capable of executing similar processing may be used. For example, a smartphone, a personal computer, or other device may be used as the information processing device (1 or 16). Such information processing devices may be required to search for a specific person from a large number of captured images. Therefore, the information processing device may be configured to search only for people who have given consent based on the output consent information. That is, when a user specifies a person to search, the consent information may be referenced and the user may specify only people who have given consent. This processing allows only images of people who have given consent to be searched, thereby realizing restrictions that better respect individual intentions. Furthermore, the information processing device 16, which is the output destination, may be a device that provides cloud services. Even in such a case, the information processing device 1 according to this embodiment can output consent information to the output destination device via a network.

[0092] [Variation 3-2] [Variations of consent information] The information processing device 1 may also be configured to allow the user to specify consent to outputting the consent information or biometric information to another device. With such a configuration, the consent information or biometric information can be prevented from being inadvertently output to another device, while allowing the target device to use the information or biometric information.

[0093] [Variation 3-3] [Select consent information to output] Although the information processing device 1 according to the present embodiment has been described as automatically synchronizing and outputting the output destination and consent information, the user may be able to select the information to be synchronized. In this case, the items that the user can select are limited to those for which consent information indicates consent. With this configuration, the user can select and synchronize only the consent information that is necessary.

[0094] [Variation 3-4] [Verification of output information] In addition, although the present embodiment has been described with reference to an example in which the registered dictionary is output to another device and synchronized, it is also possible to output and synchronize only a portion of the information contained in the registered dictionary. For example, only consent information and personal names may be output and synchronized, or only biometric information or features may be output and synchronized.

[0095] Furthermore, the information processing device 1 may output to an external device, for a certain person, the identification information and consent information indicating the person in association with each other, regardless of whether consent has been given. In this case, in addition to the identification information and consent information, if there is information for which consent to use is given based on the consent information, that information is additionally associated and output.

[0096] [Embodiment 4] In the third embodiment, an example was described in which the information processing device 1 outputs consent information, biometric information, and feature quantities to another device. On the other hand, in the present embodiment, information (permission information) for permitting data transmission and reception is transmitted from the information processing device 1 to an external device, and then the biometric information and feature quantities are output from the information processing device 1 to the external device that transmitted the permission information according to the consent state, and biometric authentication processing is performed in the external device. In particular, as the permission information, a request for device authentication between the information processing device 1 and the external device, as described below, is transmitted, and when the information processing device 1 receives approval information, described below, in response to such request, it outputs the biometric information and feature quantities to the external device. Such an information processing device and external device will be described below.

[0097] The server 18, which is an external device according to this embodiment, acquires biometric information and feature amounts from the information processing device 1 and performs various processes based on the acquired feature amounts. Here, the server 18 is a server that provides an image storage service, and stores images transmitted from other devices for each user, and provides various services such as sharing or managing images in response to requests from other devices.

[0098] Fig. 17 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment has the same functional units as those shown in Fig. 3 and can execute the same processes, except that it has an output unit 64, a device authentication unit 66, and a processing result utilization unit 67 instead of the output unit 34, the matching unit 36, and the matching result utilization unit 37, and therefore a duplicated description will be omitted.

[0099] The output unit 64 outputs the registered dictionary registered by the registration unit 35 (i.e., here, including consent information, biometric information (facial image), feature amounts, and personal name information) to an external device (here, server 18). As a prerequisite for this output, the output unit 64 needs to have previously obtained approval information from the device authentication unit 66. If approval information has not been obtained, the output unit 64 restricts the output of the registered dictionary to the external device. Here, the output unit 64 may restrict the output of the registered dictionary to the external device by not outputting the registered dictionary, or may control the server 18 so that it cannot receive data including the registered dictionary.

[0100] The device authentication unit 66 communicates with the external device and performs an authorization process. Specifically, the device authentication unit 66 transmits information requesting authorization to the external device, and the external device transmits authorization information to the device authentication unit 66, thereby enabling transmission and reception of data such as biometric information or feature quantities. The authorization information is not particularly limited as long as it is information used to determine whether or not to permit transmission and reception of data in communication with another device, such as an authentication token used to authenticate the device. Hereinafter, this series of processes for receiving authorization information from an external device and enabling transmission and reception of data with the external device may be referred to as "device authentication." Furthermore, the enabling of data transmission and reception through device authentication may be expressed as "device authentication (is performed)."

[0101] The processing result utilization unit 67 acquires and utilizes the processing results from the external device that has undergone the approval process by the device authentication unit 66. A specific example of the processing by the processing result utilization unit 67 will be described later.

[0102] A specific example of the biometric authentication process performed in the external device will be described below with reference to FIG.

[0103] Fig. 18 is a flowchart showing an example of authentication processing performed by the information processing device 1 and server 18 according to this embodiment. In the example of Fig. 18, the information processing device 1 outputs biometric information to the server 18, and the server 18 executes the authentication processing. Note that the processing shown in Fig. 18 is assumed to start, for example, when a user of the information processing device 1 issues an instruction to the information processing device 1 to output the biometric information and feature amounts to an external device.

[0104] First, in S151, the device authentication unit 66 identifies a device (output destination device) that outputs the biometric information and feature quantities. The device authentication unit 66 can identify the output destination device based on input from the user of the information processing device 1. Specifically, the device authentication unit 66 stores information linking services available to the information processing device 1, i.e., services permitted to process biometric information or feature quantities, with devices that provide those services, and can identify the output destination device by accepting a user's selection from among devices linked to such services. Note that, here, the user's input is assumed to be an input for directly selecting a device (e.g., selecting from a list of device names). However, this type of input is not particularly limited as long as the device is selected in a similar manner based on the user's input. For example, the device authentication unit 66 may accept a user's selection of a service (or a process performed by the service) as described above, and identify a device linked to the selected service as the output destination device. In this way, the permission information may include information specifying a service (or a process corresponding to the service) to be executed in the external device that is the destination. Through this process, the device that outputs the biometric information and feature quantities is identified. By allowing the user to select the output destination device, it becomes possible to specify what processing is to be performed based on the biometric information or feature amounts. In the example of Fig. 18, it is assumed that the user has specified the image classification processing using feature amounts, which will be described later, as the service, and has selected the server 18 as the external device that will perform that processing.

[0105] In S152, the device authentication unit 66 performs approval processing with the device identified in S151, in this case the server 18. Specifically, the device authentication unit 66 transmits information to the server 18 requesting approval for communication of data such as biometric information or feature quantities. At this time, a device identifier capable of identifying the information processing device 1 is output to the server 18 as the information requesting approval, so that the server 18 that has acquired the information can execute approval. Here, the device identifier is an identifier for identifying the device, and may be prepared independently or an existing one may be used. For example, the device identifier may be information unique to the device, such as the device's serial number or MAC address.

[0106] In S153, the server 18 performs device authentication based on the acquired device identifier. Specifically, the server 18 determines whether or not to permit communication of biometric information or feature amounts with the information processing device 1. For example, the server 18 may store in advance a list of device identifiers for which communication is permitted, and permit communication if the acquired device identifier is included in the list.

[0107] By performing device authentication in this manner, biometric information and feature amounts can be output to a device that executes a service that is intended by the user of the information processing device 1. In other words, it is possible to prevent the biometric information or feature amounts from being output to a device that executes an unexpected service, thereby preventing unintended use of the biometric information or feature amounts. For example, it is possible to prevent situations such as the biometric information or feature amounts being used ignoring the consent information or being used in an unexpected way.

[0108] In S154, if device authentication has been performed in S153, that is, if communication between the information processing device 1 and the server 18 is permitted (Yes in S154), the server 18 proceeds to S155. If device authentication has not been performed (No in S154), the processing of this flowchart ends.

[0109] In S155, the server 18 outputs the approval information to the device authentication unit 66. In S156, the output unit 64 outputs the registration dictionary and the captured image to the server 18, assuming that the device authentication unit 66 has received the approval information. As described above, the registration dictionary is data composed of a person's name, a facial image, and feature amounts. The captured image is a large number of facial images including the face of the subject, i.e., biometric information. When the output unit 64 outputs information, it may also output a certificate obtained from the device authentication unit 66 to the server 18, and the server 18 may permit reception of the registration dictionary and the captured image by verifying such certificate. Note that here, as in the first to third embodiments, the information processing device 1 outputs registration information whose consent information indicates consent to the server 18.

[0110] In S157, the server 18 acquires the registered dictionary and the captured image. In S158, the output unit 64 specifies (requests) processing for the registered dictionary and biometric information acquired by the server 18 in S157. As described above, information indicating available services (processing) is stored in advance in the information processing device 1, and execution of that processing is requested here. Note that a format may be adopted in which a plurality of services available to the information processing device 1 are prepared, and a request is made to execute processing for one or more selected services from among those services.

[0111] In S159, the server 18 performs the process requested in S158. In this example, the server 18 performs image classification processing (service) using feature amounts. First, the server 18 searches the captured images acquired in S157 (here, for each of the many captured images) for people registered in the registration dictionary also acquired in S157. The server 18 can identify who appears in which image by, for example, performing the face detection or face authentication process described above. Information about the identified person is stored as metadata associated with the captured image.

[0112] In S160, the processing result utilization unit 67 obtains the processing result of S159, i.e., metadata, from the server 18, displays the processing result, and terminates the processing of FIG. 16. This display of the processing result involves displaying a list of all people whose information is included in the metadata obtained in S159 along with their facial images, accepting a user selection of which person's image the user wants to display, and displaying an image featuring the selected person. This improves convenience by allowing users to display images featuring only specific people, or to perform operations such as processing, editing, or deletion on photos featuring only specific people. Note that, here, it is assumed that, prior to S160, the output unit 64 requests the server 18 to transmit the processing result, and a process (not shown) is performed to enter a standby state for obtaining the processing result.

[0113] [effect] According to this configuration, when device authentication is established between the information processing device 1 and the external device, the biometric information and feature values ​​acquired by the information processing device 1 can be output to the external device, and the biometric authentication process can be performed in the external device. This, as in the third embodiment, can be expected to reduce the effort required to acquire consent information. Furthermore, it is also expected to have the effect of offloading processing to another device and distributing the load. Even when using a standalone device such as a camera, which often has low processing power and runs on a battery, it is possible to distribute the load by offloading processing and having the external device perform biometric authentication. Therefore, it is expected to have the effect of allowing the camera to focus on its primary processing, such as image capture, or to conserve battery power. Furthermore, by performing device authentication, the biometric information and feature values ​​can be output only to devices that execute predetermined services, preventing unexpected processing.

[0114] [Variation 4-1] In this embodiment, the external device, i.e., the server 18, provides a service using image classification. However, other services using biometric information and features may also be used. For example, a service may identify a person from a large number of images using features from a registered dictionary, and then erase the person's face if the person's consent information indicates non-consent. The process of erasing a person's face can be performed using techniques such as masking, mosaicing, or cropping the face to prevent it from appearing. This process automates the tedious task of erasing unauthorized people from a large number of images, thereby improving convenience. Furthermore, by having an external device perform such erasure processing rather than the information processing device 1, the processing load on the information processing device 1 can be reduced.

[0115] Another possible service provided by the server 18 is to list persons who have not given consent from a large number of images and provide a method for accessing information for obtaining consent. Specifically, a web page for obtaining consent is prepared and its URL (Uniform Resource Locator) is issued, so that the user of the information processing device 1 can send the URL to the subject (person who has not given consent) by email or using a communication tool. This configuration allows for the cumbersome process of obtaining consent to proceed smoothly.

[0116] [Variation 4-2] Although the permission information according to the present embodiment has been described as a request for device authentication, it is not limited thereto as long as it is information for permitting transmission and reception of data between devices. For example, when transmitting biometric information and feature quantities, the information processing device 1 may simultaneously transmit permission information indicating permission to execute processing using the biometric information, linked to the biometric information. Specifically, a permission information creation unit (not shown) included in the information processing device 1 creates permission information for processing the biometric information in an external device based on the consent information (i.e., when the consent information indicates consent for the person to be processed). The output unit 64 links the created permission information to the biometric information and outputs it to the server 18. The server 18 executes biometric authentication processing in response to acquiring the permission information.

[0117] According to this type of processing, by linking the permission information to the biometric information, the server 18 can determine whether or not it is okay to execute processing using the biometric information and feature values ​​without querying the information processing device 1, which increases the flexibility in the timing of the processing, thereby improving convenience.

[0118] Note that the permission information linked to the biometric information may include information about the service specified in S158. In this way, when multiple services are available on the server 18, the server 18 alone can determine which processing is permitted for which biometric information, which increases the flexibility in the timing of processing and improves convenience.

[0119] [Other variations] In the present embodiment, the information processing device 1 simultaneously outputs the registered dictionary and the captured image to the server 18 in S156, but these pieces of information do not have to be transmitted simultaneously. For example, once the registered dictionary is transmitted to the server 18, it may not be output to the server 18 unless there is a difference (unless there is an update). Furthermore, if the captured image has already been transmitted to the server 18, the transmission process from the second time onwards may be omitted.

[0120] Furthermore, in the present embodiment, the description has been given assuming that the registered dictionary, i.e., the biometric information and the feature amounts, are output to the server 18, but either one of these pieces of information may be output. When only the biometric information is output, the server 18 may extract and use the feature amounts from the captured image. In this case, however, it is preferable that the algorithm for extracting the feature amounts is the same as that of the information processing device 1, and such an operation is made executable in advance. When only the feature amounts are output, the processing may be performed as shown in this embodiment.

[0121] Furthermore, in this embodiment, the information processing device 1 requests execution of processing on the registered dictionary and captured images immediately after outputting them to the server 18 in S156, but the timing of sending the request is not particularly limited as long as it occurs after S156. For example, if captured images are output to the server 18 every few days and stored, and a request to execute processing is made (S158) when a certain number of images have accumulated, this is efficient because processing can be performed on a large number of stored images. Furthermore, by doing so, processing can be performed at a timing of the user's choosing, thereby improving convenience.

[0122] In addition, in the present embodiment, the case where the device that captures the image and the device that outputs the image to the external device are the same device has been described, but these processes may be performed by different devices. In this case, the device that outputs the image to the external device receives biometric information and / or feature amounts extracted from the biometric information from another device such as a camera, and outputs them all together to the external device. This is more efficient because it requires only one output rather than outputting from each device individually.

[0123] In addition, in the present embodiment, an example has been described in which unexpected processing using biometric information is prevented by controlling data output only to external devices that have been authenticated. However, use restrictions may be more securely enforced by allowing data transmission and reception using techniques such as encryption. Specifically, the information processing device 1 encrypts the biometric information or feature quantity before outputting it to the external device. The information processing device 1 then outputs a decryption key for the encryption to the external device, and the external device uses the biometric information or feature quantity after decrypting it with the decryption key. This configuration prevents data from being extracted from the information processing device 1 and used unintentionally. In the above-described embodiment, the information processing device 1 is a digital camera, and biometric information is often recorded on a small portable medium. In such a case, even if the portable medium is removed, the biometric information cannot be used unless the decryption key is obtained, thereby preventing unintended use of the biometric information.

[0124] In the present embodiment, when device authentication is performed, the information processing device 1 requests communication permission from the server 18, and the server 18 approves it, but the sender and receiver may be reversed. That is, the server 18 may request communication permission from the information processing device 1, and the information processing device 1 may approve it. In the case of a device such as a server that is intended to operate 24 hours a day, if communication permission and processing using biometric information and feature amounts are periodically requested rather than by a user's instruction, processing of biometric information is automatically performed without the user of the information processing device 1 being aware of it, thereby improving convenience.

[0125] [Embodiment 5] [Consent to AI processing and use of the processing results] In the first to fourth embodiments, examples of restricting the use of biometric information were described. However, in this embodiment, a case where the target of use restriction is not limited to biometric authentication, but the use of AI processing and its processing results is restricted will be described. In this embodiment, a method for restricting AI processing targeting customers who have not given consent by identifying customers (users) appearing in images captured by cameras installed in a commercial facility and consent information related to those customers will be described. It is assumed here that multiple services using AI processing are provided in the commercial facility, and consent is obtained for each of those services.

[0126] In this embodiment, the term "AI processing" refers to processing that performs intelligent processing, identification, judgment, etc. using technology such as machine learning, and specifically, AI processing is assumed to include face detection or face recognition, age / gender estimation, etc. Furthermore, the AI ​​processing according to this embodiment is not limited to image processing, but also includes text analysis, analysis of other sensing data, etc.

[0127] The processing performed by the information processing device 1 according to this embodiment will be described below. FIG. 11 is a schematic diagram showing an example of the system configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment functions as a server that controls the system of this embodiment. The information processing device 1 according to this embodiment is also equipped with input devices such as a mouse and keyboard, as well as a screen display panel for operation, and can accept operations from the administrator of the system of this embodiment. The network 15 is the same as that of embodiment 3. The other hardware configurations are the same as those of the information processing device 1 of embodiment 1, so duplicated explanations will be omitted.

[0128] Imaging device 41 is a camera for facial recognition payment, and is composed of a camera unit and a communication device. Imaging device 41 transmits the captured facial image to information processing device 1 via network 15. Surveillance camera 42 and surveillance camera 43 are a first surveillance camera and a second surveillance camera, respectively. Surveillance camera 42 and surveillance camera 43 are each composed of a camera unit and a communication device, and transmit the captured image to information processing device 1 via network 15.

[0129] 12 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. The biometric information acquisition unit 51 acquires images of an individual, in this case, a customer, from the face recognition payment camera 41, the monitoring camera 42, and the second monitoring camera 43.

[0130] The individual identification unit 52 identifies the individual appearing in the image acquired by the biometric information acquisition unit 51. The consent information management unit 53 acquires and manages consent information indicating what AI processing an individual has consented to and the use of the processing results.

[0131] The AI ​​processing unit 54 is an AI processing unit that performs AI processing on the image acquired by the biometric information acquisition unit 51. The AI ​​processing unit 54 according to this embodiment refers to the consent information and executes only the AI ​​processing to which the individual who is the target of the AI ​​processing has consented. In other words, the AI ​​processing unit 54 according to this embodiment restricts the AI ​​processing based on the consent. The content of the AI ​​processing will be specifically described later.

[0132] The AI ​​processing result utilization unit 55 utilizes the results processed by the AI ​​processing unit 54. The AI ​​processing result utilization unit 55 creates information that will be the basis for services to be provided to individuals.

[0133] The consent condition storage unit 56 stores consent conditions, including conditions related to consent for individuals, and presents them to the user. The consent conditions according to this embodiment include information such as the content of the AI ​​processing, the data used by the AI ​​processing, the purpose of the AI ​​processing, information about the organization performing the AI ​​processing, information about the organization managing the data used by the AI ​​processing, the services provided to the customer, and the period during which the AI ​​processing will be performed. These consent conditions are presented to the customer in advance and consent is obtained from the customer, and are conditions for implementing the AI ​​processing described below. The output unit 57 outputs and displays the information created by the AI ​​processing result utilization unit 55 and the consent conditions stored in the consent condition storage unit 56 to an external device.

[0134] The specific steps of the processing performed by the information processing device 1 according to this embodiment are described below. The processing according to this embodiment is divided into two steps: a pre-processing step in which consent conditions are presented to a customer and consent information is registered in a registration dictionary together with a facial image or feature amount, and an AI processing step in which AI processing is performed based on the consent information registered in the pre-processing step.

[0135] In the pre-processing according to this embodiment, similar to the registration processing described in embodiment 1, feature amounts, biometric information, consent information, and personal names are linked and registered in a registration dictionary. Since this embodiment targets customers who use commercial facilities, this registration processing is performed, for example, using the customer's smartphone when entering the commercial facility, or the registration processing is performed in advance on a website provided by the commercial facility. Another difference between the processing performed by the information processing device 1 according to this embodiment and that in embodiment 1 is that the consent information includes consent to multiple AI processes.

[0136] FIG. 13 is a diagram schematically illustrating the contents of a registered dictionary in this embodiment. The columns for No., person's name, facial image, and feature value shown in FIG. 13 are the same as those in embodiment 1, and therefore will not be described here. The consent information column displays services using AI processing, such as "Facial Recognition Payment," "Crowd Analysis," and "Visitor Trend Analysis," with checkboxes indicating consent for each service. In the example shown in FIG. 13, "Mr. A" consents to all services, while "Mr. B" does not. On the other hand, "Mr. C" does not consent to "Facial Recognition Payment" and "Visitor Trend Analysis," but does consent to "Crowd Analysis." Thus, the consent information according to this embodiment describes the purpose (type) of AI processing and includes information indicating whether the user consents to each AI processing. Note that the AI ​​processing shown in FIG. 13 is merely an example; other types of AI processing, such as face detection, facial recognition, or age / gender estimation, may also be listed.

[0137] This consent information is acquired by the operator of the commercial facility presenting the conditions of each consent information (consent conditions) to the customer when the customer enters the commercial facility and confirming the customer's willingness to consent. For example, the address of a web page indicating the consent conditions is presented to the customer, and the customer who accesses the address is asked to confirm the consent conditions on a smartphone or other device, and consent is acquired based on the user's input. The address may be presented as text on a poster or the like, or as a two-dimensional barcode to reduce the customer's input effort. The form of the address is not particularly limited as long as it is accessible to the user. The address may also be distributed to the customer's smartphone or other device at a specific location using contactless communication technology.

[0138] Figure 14 is a schematic diagram showing an example of consent conditions presented to a customer. The consent conditions are presented in this way, and consent information is acquired by having the customer input whether or not they consent. The consent conditions shown in Figure 14 present information about services that use the three AI processes mentioned above: "Facial Recognition Payment," "Congestion Analysis," and "Visitor Trend Analysis."

[0139] "Facial recognition payment" is a service that identifies people in an image and uses payment information (such as bank account or credit card information) pre-linked to that person to make payments such as purchasing goods. "Crowd analysis" is a service that counts the number of people in an image, visualizes the number of people within the camera's field of view, and presents the visualization results to customers. "Visitor trend analysis" is a service that estimates the age and gender of customers to analyze trends among visitors and provide advertisements based on their age and gender on signage in commercial facilities, etc. All of these services that use AI processing can be realized by using well-known AI processing such as face detection, face recognition, or age and gender estimation.

[0140] In the example of the service item "1. Facial Recognition Payment" in Figure 14, the information presented includes the AI ​​processing details 201, the data processed by the AI ​​202, the service provided to the customer by the AI ​​processing 203, and the consent period 204. The consent period indicates the period during which the consent setting is valid. In the example of Figure 14, the customer can enter the details, but a specific date or period may be presented and the user's selection accepted. After confirming these conditions, the customer can indicate their consent by selecting a check box 205. In this case, an unselected check box means that the user has not consented. For other service items, the same information as "1. Facial Recognition Payment" is presented. Note that if a length is set as the consent period, the period for executing the AI ​​processing can be set by obtaining the current time when the AI ​​processing is executed.

[0141] When the customer has finished inputting whether or not he / she consents to all services, he / she presses the consent button 206 to decide his / her intention to consent to the consent conditions, i.e., the consent information, and the consent information is acquired by the information processing device 1.

[0142] This configuration makes it possible to obtain consent information from customers after having them confirm the services they will receive and the data they will provide (data processed by AI). Therefore, it is possible to provide only AI-processed services to customers that the customer has consented to, and conversely, it is possible to prevent AI processing that the customer has not consented to from being performed, i.e., to restrict the execution of AI processing. Therefore, it is possible to provide services that are in line with the customer's wishes, improving convenience for both the service provider and the customer.

[0143] [AI processing] Next, the AI ​​processing according to this embodiment will be described. Fig. 15 is a flowchart showing an example of the AI ​​processing executed in this embodiment. Note that the processing shown in Fig. 15 is executed after the above-mentioned preliminary processing is completed. If the processing shown in Fig. 15 is executed before the preliminary processing is completed, there is no person who is said to have consented to the AI ​​processing, and the AI ​​processing with consent described below will not be executed.

[0144] In S131, the information processing device 1 performs initialization processing for AI processing. Here, the information processing device 1 loads the registered dictionary registered in the pre-processing and makes it readable.

[0145] In S132, the biometric information acquisition unit 51 acquires an image of an individual from a camera. Here, it is assumed that the biometric information acquisition unit 51 acquires images from the monitoring cameras 42 and 43.

[0146] In S133, the individual identification unit 52 identifies who the person in the acquired image is. Here, the individual identification unit 52 identifies, for all people in the image, whether they correspond to any person registered in the registration dictionary or whether they do not match any person. This identification method can be performed in the same way as in S114 of the second embodiment. The information processing device 1 according to this embodiment extracts features from the captured image regardless of whether consent is obtained, to identify an individual, and restricts the use of subsequent biometric information for that individual based on the consent information.

[0147] In S134, the consent information management unit 53 acquires the consent information of the identified person. In the example of Fig. 13, the feature amount and the consent information are registered in association with each other, so the consent information management unit 53 refers to the consent information of the identified person. Note that here too, for a person who is determined not to match anyone by the matching process, no consent information exists, and therefore the person is considered to have not consented to any services.

[0148] In S135, the AI ​​processing unit 54 restricts the use of the biometric information based on the consent information, and then executes AI processing on the acquired image (including the face). Here, the AI ​​processing unit 54 executes AI processing for which consent has been obtained, assuming that AI processing for which consent has not been obtained will not be executed based on the consent information.

[0149] It should be noted that the (type of) imaging device and the type of AI processing to be executed are associated and set in advance, and only AI processing for which consent has been obtained is executed from among the AI ​​processing types thus set in advance. Here, images are acquired from surveillance camera 42 and surveillance camera 43, and the AI ​​processing to be executed is "congestion analysis" and "visitor trend analysis" associated with those surveillance cameras (of the type "surveillance camera"), but "face recognition payment" is not executed.

[0150] For example, if a person identified as "Mr. C" (shown in FIG. 13) is present in the captured image being processed, age and gender estimation for the "Visitor Trend Analysis," the only AI processing for which consent is granted, is executed based on Mr. C's consent information. On the other hand, if a person detected in the captured image does not consent to the "Visitor Trend Analysis," age and gender estimation is not executed for that person. For example, if a person identified as "Mr. A" (shown in FIG. 13) is present in the captured image, all executable AI processing is executed because Mr. A's consent information indicates consent for all AI processing. This processing is executed in accordance with the consent conditions stored in the consent condition storage unit 56 and presented to the customer, as shown in FIG. 14. The information processing device 1 may be configured to allow the system administrator to confirm whether the processing content, data to be processed, or purpose described in FIG. 14 is in accordance with the processing content, data to be processed, or purpose.

[0151] In S136, the AI ​​processing result utilization unit 55 performs various processes using the results of the AI ​​processing. For example, the AI ​​processing result utilization unit 55 may convert and process the results of AI processing, such as face detection or age and gender estimation, into a form that can be used for services and output it to another device as needed. For example, when the AI ​​processing result utilization unit 55 performs a "visitor trend analysis," it may output data counting the number of people for each estimated age and gender. Here, the age and gender are estimated only for people who have consented to the "visitor trend analysis" in the captured image, and only those who consent are counted. This count number may be accumulated for a predetermined period while the system is operating, and the number of visitors for each age and gender during that period may be calculated and output to another device as needed. This information is useful for commercial facility operators because it can be used as data to improve the operation of the commercial facility.

[0152] Furthermore, the AI ​​processing result utilization unit 55 may provide a service to customers by displaying advertisements based on the estimated age and gender on a signage terminal near the camera that captured the image. This type of processing is beneficial because it can provide advertisements that are likely to be suitable for the customer. In this way, AI processing and the use of processing results that are beneficial to both the operator of the commercial facility and the customer can be carried out with their consent.

[0153] Furthermore, when the AI ​​processing result utilization unit 55 executes "crowding analysis," it can count the number of people in the store in the same way as when executing "visitor trend analysis," and output the count results to another device. This allows the congestion status to be presented to customers, improving customer convenience. Such congestion status can be presented on a display in the store or via an application on the customer's smartphone, etc.

[0154] In S137, the information processing device 1 determines whether to terminate the AI ​​processing. Here, the information processing device 1 determines to terminate the AI ​​processing if an instruction to terminate has been received from the administrator of the system, and determines not to terminate the AI ​​processing if not. If it is determined in S137 that the AI ​​processing should be terminated, the processing of FIG. 15 ends, and if not, the processing returns to S132.

[0155] This type of processing makes it possible to limit the processing performed on customer data to only those that have been consented to. In particular, since AI processing can be restricted after obtaining the customer's consent to the execution of AI processing for each AI processing service, it becomes possible to perform processing that better respects the individual's will. Therefore, from the customer's perspective, it becomes possible to indicate whether or not to consent to AI processing for each AI processing service, and from the service provider's perspective, it is possible to provide only the AI ​​processing and services that the customer desires, thereby improving convenience.

[0156] It is also possible to offer more dynamic conditions for restricting, allowing, and rewarding AI processing, such as offering customers a time-limited online coupon if they allow a business to use their anonymized in-store purchase history to improve services.

[0157] Furthermore, by presenting the terms of consent to customers in advance, it is possible to explain what information about them will be used in AI processing and what benefits they will receive as a result. This allows customers to choose whether or not to consent to AI processing for each service. This allows customers to choose whether or not to receive a service based on AI processing, taking into account the advantages and disadvantages in accordance with their own wishes, thereby improving convenience.

[0158] [Variation 5-1] In the fifth embodiment, an example was described in which an image from a surveillance camera was acquired and AI processing was performed using the surveillance camera, but each process can be performed in the same way when using an imaging device of a different type than a surveillance camera. Below, we will explain a case in which each process is performed using an image acquired from the facial recognition payment camera 41 instead of the surveillance cameras 42 and 43. Here, it is assumed that "facial recognition payment" is associated with the facial recognition payment camera as the AI ​​processing to be performed.

[0159] Here, in S132, an image is acquired from the face recognition payment camera 41, and in the subsequent processing, the AI ​​processing of "face recognition payment" is executed. In the following, the explanation of the processing that overlaps with that explained with reference to FIG. 15 will be omitted.

[0160] In S135, the AI ​​processing unit 54 executes AI processing for the user who has consented to "face authentication payment" based on the consent information. Here, the user is identified using face authentication, but the result of identifying the user performed in S133 may be reused.

[0161] In S136, the AI ​​processing result utilization unit 55 performs payment based on the facial recognition result, referring to payment information such as pre-registered credit card or bank account information, and transmits the payment result to the payment terminal of the commercial facility. Note that prior to this process, there is assumed to be processing such as a store clerk at the commercial facility determining the payment amount or obtaining the customer's intention to use facial recognition payment. Note that for persons who have not consented, payment cannot be made, and therefore information indicating that the payment has failed is output to the payment terminal.

[0162] This type of processing also makes it possible to restrict use based on consent for services that use relatively confidential information, such as payment information. Some customers may feel uneasy about using such information, so by making it possible to restrict use based on consent, it becomes possible to process services that take into consideration the customer's wishes.

[0163] [Variation 5-2] In this embodiment, it has been described that all consent information of persons not in the registered dictionary is processed as if they have not consented, but this processing is not particularly limited to this. For example, the information processing device 1 may perform each process assuming that persons not in the registered dictionary have consented to all AI processing. In this case, however, it is desirable to notify customers by displaying a message such as "If a customer enters a commercial facility according to operating regulations, they will be considered to have consented to AI processing" so that customers do not feel uncomfortable that AI processing has been performed without their consent.

[0164] [Variation 5-3] In this embodiment, biometric information, i.e., an image of a face, is used as the data input to AI processing, but it does not have to be biometric information. For example, personal information such as an individual's name, address, email address, telephone number, or attribute information (race, gender, age, occupation) may be registered in advance in a registration dictionary as text or numerical values ​​and used for AI processing. This personal information may be an ID number assigned a unique number to each individual. In this case, an example of AI processing and services may include behavioral and purchasing predictions using textual personal information. Even if the input data is text or numerical values, because it is personal information, restricting its use with the individual's consent can provide a service that is highly satisfactory to customers.

[0165] Furthermore, in the present embodiment, biometric information is used to identify the individual, but biometric information does not necessarily have to be used as long as the individual can be identified. For example, the individual may be identified by information such as a membership number or a user name, and consent information in a registered dictionary may be referenced. Furthermore, for example, the individual may be identified by an RFID tag or device-to-device communication with a smartphone carried by the user.

[0166] [Variation 5-4] In this embodiment, information such as the content of AI processing, the data to be used, or the purpose is presented as a condition for consent, but the information presented here is not limited to this. For example, as described above, the name of the organization or company that manages the AI ​​processing and the data to be used may be presented as part of the condition for consent. Such processing allows the customer to decide whether to consent by taking into account whether the organization or company is trustworthy. In this case, organizations and companies may be assigned a score in advance indicating their trustworthiness, and this score may be presented to the customer.

[0167] Information about the location of use may also be presented as a condition of consent. For example, information about whether the user's information will be used only at that commercial facility or at commercial facilities in other locations may be presented as part of the condition of consent. Furthermore, the screen presenting the condition of consent may allow the customer to select the location where their information will be used. This type of processing makes it possible to configure the system so that customers can collectively indicate their consent for all commercial facilities operated by the same operator.

[0168] [Variation 5-5] In this embodiment, an example has been described in which the information processing device 1 is used for AI processing and services in a commercial facility as shown in Fig. 14, but the location, AI processing, and services are not limited to those described here. For example, when using biometric information as data for improving the operation of the commercial facility, a process of restricting the use of biometric information in the information processing device 1 may be performed. In this case, the information processing device 1 according to this embodiment can be used, for example, when performing AI processing to constantly detect and track customers and visualize their movement paths within the commercial facility.

[0169] Furthermore, by using facial recognition to manage entry to specific locations or rooms, it becomes possible to eliminate the need for membership cards or keys. From this perspective, the information processing device 1 may be used in commercial facilities such as sports gyms, schools, hospitals, and other facilities. When the information processing device 1 is used in a school or hospital, in addition to the AI ​​processing for entry management described above, it is expected that the information processing device 1 will be used for AI processing and services such as operating an AI to detect suspicious individuals for security services, or operating an AI to detect abnormal behavior such as falls for monitoring services. Even in such cases, by obtaining the user's consent for each AI processing and service, it becomes possible to provide services that reflect the customer's wishes.

[0170] However, some services that are of public interest or highly urgent nature, such as crime prevention services, monitoring services, or guidance services based on an analysis of the number or location of passersby during a disaster, may be provided without consent. In such cases, a separate situation notification module (not shown) may be provided to control the services provided by the information processing device 1 based on the notification results so that the services will operate only under appropriate conditions in accordance with current or future laws and regulations. This configuration allows customers to always receive urgent services without having to wait for consent each time, thereby improving convenience.

[0171] For example, in a group setting such as a school, if a service is provided in which automatic photographing of faces using face detection is performed and the photographs are then classified by person and sold, consent may be obtained for both automatic photographing using face detection and for person identification processing to classify the photographs. This type of processing allows AI processing to be restricted according to individual wishes. In particular, it becomes possible to perform processing that respects the detailed wishes of individuals, such as allowing face detection alone but not allowing classification to identify individuals.

[0172] Although the present embodiment has been described assuming that consent information is obtained from the user, the consent information does not have to be entered by the user himself / herself, as long as it indicates whether or not the user has given consent. For example, a parent or guardian of a user (especially a child) may submit consent information on behalf of the user. Such processing can accommodate cases where, for example, when a child gets lost in a commercial facility, the parent or guardian enters consent information on the child's behalf when AI processing (face detection and face recognition) is performed on surveillance camera images. Furthermore, the present embodiment has been described assuming that consent is obtained upon entering a commercial facility. However, it is also possible for a parent or guardian to access the system and enter consent to AI processing to find the lost child when the child becomes lost. In this way, the person entering the consent information does not have to be the user himself / herself; the consent information may be entered by a person deemed to have the right to enter consent. Furthermore, the time or place for consent to be entered is not limited and may be any time before AI processing is performed. For example, the consent information may be entered when a service is needed.

[0173] [Variation 5-6] In this embodiment, consent information for the use of AI processing is acquired for each commercial facility. However, for example, by sharing consent information between facilities, it is possible to eliminate the need for users to enter consent information one by one at each facility. Furthermore, for example, similar to setting security items for each trust level in Internet web browsing, multiple trust levels and corresponding consent conditions may be associated in advance, and processing may be performed based on these settings. In this case, facility users can set consent simply by specifying which trust level the facility they are using corresponds to. Furthermore, (default) consent conditions may be set in the absence of user input. Furthermore, in this embodiment, consent information is described as being entered in advance, but it may also be possible to edit it, for example, while using the facility. This configuration allows users to change the simultaneous conditions at a certain timing, improving convenience.

[0174] [Other embodiments] In the above-described first to fifth embodiments, an example of acquiring a face image as biometric information has been described. However, the biometric information is not limited to an image containing a face, as long as it contains information about the user. For example, the biometric information may be an image containing information that can identify the user, such as an individual's iris, fingerprint, or veins. Even when such biometric information is used, it is possible to similarly extract features from the biometric information. Note that the device for acquiring the captured image may be an imaging device including an appropriate sensor corresponding to each type of biometric information.

[0175] The disclosure of this specification includes the following information processing device, system, information processing method, and program. (Item 1) a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; a first transmitting means for transmitting at least one of the biometric information acquired by the biometric information acquiring means, the identification information of the person corresponding to the biometric information, the feature extracted from the biometric information, and the consent information acquired by the consent information acquiring means to a first external device in accordance with the consent state; An information processing device comprising: (Item 2) The information processing device described in item 1, characterized in that if the consent information acquired by the consent information acquisition means indicates consent, the first transmission means associates the biometric information and / or feature of the person corresponding to the consent information with the consent information and transmits them to a first external device. (Item 3) 3. The information processing device according to item 1 or 2, characterized in that, when the consent information acquired by the consent information acquisition means indicates consent, the first transmission means transmits the features of the person corresponding to the consent information to a first external device. (Item 4) The information processing device described in any one of items 1 to 3, characterized in that the first transmission means associates the consent information acquired by the consent information acquisition means with identification information of the person corresponding to the consent information and transmits the associated information to the first external device. (Item 5) Item 5. The information processing device according to item 4, wherein the first transmission means associates the consent information and the identification information with the biometric information or the feature amount and transmits them to the first external device based on the consent information. (Item 6) a specifying means for specifying information not stored in the first external device; The information processing device described in any one of items 1 to 5, wherein the first transmitting means transmits to the first external device at least one of the information identified by the identifying means, which is the person's biometric information, the identification information, the consent information, and the feature amount. (Item 7) a presentation means for presenting the information identified by the identification means to a user; a permission acquisition means for acquiring permission from the user to transmit the presented information to the first external device; The information processing device described in item 6, characterized in that the first transmission means, when the permission acquisition means acquires permission from the user, executes transmission of at least one of the biometric information, identification information that identifies the person, features extracted from the biometric information, and the consent information to a first external device. (Item 8) when the information stored as the feature amount of the first person in the information processing device and the first external device differ, the first transmission means transmits the feature amount of the first person to the first external device; 8. The information processing device according to item 6 or 7, wherein, when it is determined that the information stored as the feature of the first person in the information processing device and the first external device is the same person, the first transmission means does not transmit the feature of the first person to the first external device. (Item 9) a selection means for selecting information to be output to the first external device from among the biometric information, identification information for identifying the person, features extracted from the biometric information, and the consent information; 9. The information processing device according to any one of items 1 to 8, wherein the first transmitting means transmits the information selected by the selecting means to the first external device. (Item 10) when the information stored as biometric information of the first person in the information processing device and the first external device differ, the first transmission means transmits the biometric information of the first person to the first external device; The information processing device described in any one of items 6 to 9, characterized in that if the information stored as biometric information of the first person in the information processing device and the first external device is the same, the first transmission means does not transmit the biometric information of the first person to the first external device. (Item 11) a display control means for displaying the information specified by the specifying means on a display means; The information processing device described in any one of items 6 to 8, characterized in that the first transmission means transmits the information to the first external device in response to receiving a user instruction to allow the transmission of the information displayed by the display control means. (Item 12) the first transmitting means has a transmission permission information acquiring means for acquiring transmission consent information indicating a consent state regarding the transmission of information to the first external device, The information processing device described in any one of items 1 to 11, characterized in that the first transmission means transmits to the first external device at least one of biometric information of a person corresponding to the transmission consent information that agrees to the transmission, identification information of the person corresponding to the biometric information, features extracted from the biometric information, and consent information acquired by the consent information acquisition means. (Item 13) a receiving unit for receiving designation of information that is permitted to be transmitted to the first external device by the first transmitting unit; The information processing device described in any one of items 1 to 12, characterized in that the first transmitting means transmits information specified by the receiving means among biometric information, features, and consent information to the first external device. (Item 14) 14. The information processing device according to any one of items 1 to 13, wherein the biometric information is a captured image of the person's face, iris, fingerprint, or veins. (Item 15) a second transmitting means for transmitting, to a second external device, information for permitting transmission and reception of data with the second external device; An information processing device described in any one of items 1 to 14, characterized in that the first external device is the same as the second external device to which information for permitting the transmission and reception of the data is sent. (Item 16) Item 16. The information processing device according to item 15, wherein the information for permitting transmission and reception of data is a request for device authentication with a second external device. (Item 17) Item 17. The information processing device according to item 16, wherein the information for permitting transmission and reception of data includes information specifying a service to be executed by the second external device. (Item 18) The information processing device described in item 15, characterized in that the information for permitting the transmission and reception of the data is information indicating that the execution of processing using the biometric information is permitted, and is transmitted to the second external device simultaneously with the biometric information. (Item 19) 19. The information processing device according to any one of items 15 to 18, further comprising a receiving unit for receiving a result of processing using the biometric information by the second external device. (Item 20) A system including an information processing device and an external device different from the information processing device, The information processing device includes: a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; a first transmitting means for transmitting, to the external device, information for permitting transmission and reception of data with the external device; a second transmitting means for transmitting at least one of the biometric information acquired by the biometric information acquiring means and a feature extracted from the biometric information to the external device in accordance with the consent state; Equipped with The external device is a processing means for executing processing based on the biometric information and / or the feature amount received from the information processing device; A system comprising: (Item 21) 21. The system according to item 20, wherein the first transmission means further includes information specifying processing based on the biometric information and / or the feature amount to be executed by the external device. (Item 22) The system described in item 21, characterized in that the information for permitting the transmission and reception of data is information indicating that the execution of processing using the biometric information is permitted, and is transmitted to the external device simultaneously with the biometric information. (Item 23) 23. The system according to any one of items 20 to 22, wherein the information processing device further comprises a utilization unit that acquires and utilizes the processing means by the processing unit. (Item 24) An information processing method performed by an information processing device, a biometric information acquisition step of acquiring biometric information of a person; a consent information acquisition step of acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; a transmitting step of transmitting at least one of the biometric information acquired in the biometric information acquiring step, the identification information of the person corresponding to the biometric information, the feature extracted from the biometric information, and the consent information acquired in the consent information acquiring step to a first external device in accordance with the consent state; An information processing method comprising: (Item 25) A program for causing a computer to function as each means of the information processing device described in any one of items 1 to 19.

[0176] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0177] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0178] 1: information processing device, 14: imaging device

Claims

1. a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; a first transmitting means for transmitting at least one of the biometric information acquired by the biometric information acquiring means, the identification information of the person corresponding to the biometric information, the feature extracted from the biometric information, and the consent information acquired by the consent information acquiring means to a first external device in accordance with the consent state; An information processing device comprising:

2. The information processing device described in claim 1, characterized in that when the consent information acquired by the consent information acquisition means indicates consent, the first transmission means associates the biometric information and / or features of the person corresponding to the consent information with the consent information and transmits them to a first external device.

3. 2. The information processing device according to claim 1, wherein, when the consent information acquired by the consent information acquisition means indicates consent, the first transmission means transmits the feature amount of the person corresponding to the consent information to a first external device.

4. 2. The information processing apparatus according to claim 1, wherein the first transmission means transmits the consent information acquired by the consent information acquisition means and identification information of the person corresponding to the consent information in association with each other to a first external device.

5. 5. The information processing device according to claim 4, wherein the first transmission means transmits the biometric information or the feature amount in addition to the consent information and the identification information in association with each other based on the consent information to the first external device.

6. a specifying means for specifying information not stored in the first external device; The information processing device according to claim 1, characterized in that the first transmitting means transmits to the first external device at least one of the information identified by the identifying means, which is the person's biometric information, the identification information, the consent information, and the feature amount.

7. a presentation means for presenting the information identified by the identification means to a user; a permission acquisition means for acquiring permission from the user to transmit the presented information to the first external device; 7. The information processing device according to claim 6, wherein the first transmitting means, when the permission obtaining means obtains permission from the user, performs transmission of at least one of the biometric information, identification information for identifying the person, features extracted from the biometric information, and the consent information to a first external device.

8. when the information stored as the feature amount of the first person in the information processing device and the first external device differ, the first transmission means transmits the feature amount of the first person to the first external device; The information processing device according to claim 6, characterized in that, when it is determined that the information stored as the characteristic amount of the first person in the information processing device and the first external device is the same person, the first transmission means does not transmit the characteristic amount of the first person to the first external device.

9. a selection unit that selects information to be output to the first external device from among the biometric information, identification information for identifying the person, features extracted from the biometric information, and the consent information; 2. The information processing apparatus according to claim 1, wherein said first transmitting means transmits the information selected by said selecting means to said first external device.

10. When the information stored as biometric information of the first person in the information processing device and the first external device differ, the first transmission means transmits the biometric information of the first person to the first external device; The information processing device according to claim 6, characterized in that, when the information stored as biometric information of the first person in the information processing device and the first external device is the same, the first transmission means does not transmit the biometric information of the first person to the first external device.

11. a display control means for displaying the information specified by the specifying means on a display means; 7. The information processing apparatus according to claim 6, wherein the first transmitting means transmits the information to the first external device in response to receiving a user instruction permitting transmission of the information displayed by the display control means.

12. the first transmitting means has a transmission permission information acquiring means for acquiring transmission consent information indicating a consent state regarding the transmission of information to the first external device, The information processing device described in claim 1, characterized in that the first transmission means transmits to the first external device at least one of biometric information of a person corresponding to the transmission consent information that agrees to the transmission, identification information of the person corresponding to the biometric information, features extracted from the biometric information, and consent information acquired by the consent information acquisition means.

13. a receiving unit for receiving designation of information that is permitted to be transmitted to the first external device by the first transmitting unit; 2. The information processing apparatus according to claim 1, wherein the first transmitting means transmits information designated by the accepting means out of biometric information, feature amounts, and consent information to the first external device.

14. The information processing apparatus according to claim 1 , wherein the biometric information is a captured image of the person's face, iris, fingerprint, or veins.

15. a second transmitting means for transmitting, to a second external device, information for permitting transmission and reception of data with the second external device; 2. The information processing apparatus according to claim 1, wherein the first external device is the same as a second external device to which the information for permitting transmission and reception of the data is transmitted.

16. 16. The information processing apparatus according to claim 15, wherein the information for permitting transmission and reception of data is a request for performing device authentication with a second external device.

17. 17. The information processing apparatus according to claim 16, wherein the information for permitting transmission and reception of data includes information for specifying a service to be executed by the second external device.

18. 16. The information processing device according to claim 15, wherein the information for permitting transmission and reception of data is information indicating that execution of a process using the biometric information is permitted, and is transmitted to the second external device simultaneously with the biometric information.

19. 16. The information processing apparatus according to claim 15, further comprising: a receiving unit for receiving a result of processing using the biometric information by the second external device.

20. A system including an information processing device and an external device different from the information processing device, The information processing device includes: a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; a first transmitting means for transmitting, to the external device, information for permitting transmission and reception of data with the external device; a second transmitting means for transmitting at least one of the biometric information acquired by the biometric information acquiring means and a feature extracted from the biometric information to the external device in accordance with the consent state; Equipped with The external device is a processing means for executing processing based on the biometric information and / or the feature amount received from the information processing device; A system comprising:

21. 21. The system according to claim 20, wherein the first transmission means further includes information specifying a process to be executed by the external device based on the biometric information and / or the feature amount.

22. The system described in claim 21, characterized in that the information for permitting transmission and reception of data is information indicating that execution of processing using the biometric information is permitted, and is transmitted to the external device simultaneously with the biometric information.

23. 21. The system according to claim 20, wherein the information processing device further comprises a utilization unit for acquiring and utilizing the processing means by the processing unit.

24. An information processing method performed by an information processing device, a biometric information acquisition step of acquiring biometric information of a person; a consent information acquisition step of acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; a transmitting step of transmitting at least one of the biometric information acquired by the biometric information acquiring step, the identification information of the person corresponding to the biometric information, the feature extracted from the biometric information, and the consent information acquired by the consent information acquiring step to a first external device according to the consent state; An information processing method comprising:

25. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 19.

Images