Information processing device, information processing method, and program

Abstract

To enable effective protection of privacy of a person regarding feature quantities obtained form biometric information.SOLUTION: Biometric information of a person is acquired. Consent information indicative of the consent status of the person regarding the use of feature quantities extracted from the biometric information is obtained. The feature quantities are extracted from the biometric information of the person. At least either of extraction and use of the feature quantities is restricted according to the acquired consent information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] AI processing is known, which performs detection, estimation, and other processes using parameters learned from data such as images. AI processing is capable of performing intelligent processing and judgments like humans, and has been increasingly applied in various fields in recent years. However, there are also calls for restrictions on the unauthorized use of personal information (e.g., facial images, fingerprints, irises, etc.) by AI, and certain considerations are required when using AI processing. In particular, biometric authentication, which is one type of AI processing, may violate personal privacy depending on the application, so further restrictions on its use are required. Patent Document 1 discloses a method for obtaining individual consent before registering a person to be biometrically authenticated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-119549 [Non-patent literature]

[0004] [Non-Patent Document 1] Deng, Jiankang, et al. “Retinaface: Single-shot multi-level face localization in the wild.” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] Although Patent Document 1 describes obtaining prior consent for the acquisition of facial images, there is a risk that extraction of features from biometric information such as facial images and use of those features may be carried out without the person's consent.

[0006] An object of the present invention is to more effectively protect the privacy of an individual with respect to feature quantities obtained from biometric information. [Means for solving the problem]

[0007] To achieve the object of the present invention, for example, an information processing device according to one embodiment includes the following configuration: biometric information acquisition means for acquiring biometric information of a person, consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of feature amounts extracted from the biometric information, extraction means for extracting the feature amounts from the biometric information of the person, and restriction means for restricting at least one of the extraction and use of the feature amounts based on the consent information acquired by the consent information acquisition means. [Effects of the Invention]

[0008] This makes it possible to more effectively protect the privacy of individuals regarding features obtained from biometric information. [Brief explanation of the drawings]

[0009] [Figure 1] 1A and 1B are diagrams for explaining situations in which an information processing apparatus according to an embodiment is used; [Figure 2] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing apparatus according to an embodiment. [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing apparatus according to an embodiment. [Figure 4] 10 is a flowchart showing an example of a registration process by the information processing apparatus according to the embodiment. [Figure 5] FIG. 2 is a diagram for explaining a registration dictionary in the information processing apparatus according to the embodiment. [Figure 6] 10 is a flowchart showing an example of authentication processing by the information processing apparatus according to the embodiment. [Figure 7] 1A and 1B are views showing examples of images output by the information processing apparatus according to the embodiment. [Figure 8] FIG. 1 is a diagram showing an example of a system configuration according to an embodiment. [Figure 9] 10 is a flowchart showing an example of output processing by the information processing apparatus according to the embodiment. [Figure 10] FIG. 4 is a diagram for explaining a comparison of registered dictionaries between devices according to the embodiment. [Figure 11] FIG. 1 is a diagram showing an example of a system configuration according to an embodiment. [Figure 12] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing apparatus according to an embodiment. [Figure 13] FIG. 2 is a diagram for explaining a registration dictionary in the information processing apparatus according to the embodiment. [Figure 14] FIG. 10 is a diagram for explaining consent conditions by the information processing device according to the embodiment. [Figure 15] 1 is a flowchart showing an example of AI processing by the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [Embodiment 1] [overview] An information processing device according to this embodiment acquires biometric information of a person, acquires consent information indicating consent information from the person regarding the use of features extracted from the biometric information, extracts features from the person's biometric information, and restricts at least one of the extraction and use of the features based on the acquired consent information.

[0012] The following describes a method for restricting the use of biometric information in a biometric authentication registration process for an image of a person captured by a digital camera. In this embodiment, the digital camera is described as having an information processing device and an image capture device, but the present invention is not limited to this. For example, the digital camera having the image capture device and the information processing device may be connected via a network.

[0013] 1A and 1B are diagrams for explaining situations in which an information processing device according to this embodiment is used, and Fig. 1A shows a situation in which a person is photographed using a digital camera.

[0014] The information processing device 1 executes processing by each functional unit shown in FIG. 3, which will be described later. The information processing device 1 according to this embodiment is part of a digital camera, and controls the processing of the entire digital camera. In the following description, it is assumed that an imaging device 14, which is part of the digital camera, operates in response to instructions from the information processing device 1. Subject 2 is a first subject, a person in this case, whose image is captured by the imaging device 14 in the situation shown in FIG. 1(a). Subject 3 is a second subject, a person in this case, whose image is captured by the imaging device 14 in the situation shown in FIG. 1(a). In the following description, a person who is a subject in an image may be referred to as an "individual."

[0015] Button 11 is a shutter button that a user (not shown) of imaging device 14 presses to instruct information processing device 1 to take a photograph. Panel 12 is a display panel that displays the results of processing to the user of imaging device 14. Here, panel 12 displays an image to allow the user to check the composition or the state of the subject when taking a photograph. There is no particular limit to the type of panel 12 as long as it can display an image, and it may be, for example, a liquid crystal panel, an organic EL (Electro Luminescence) panel, or a projector.

[0016] The operation keys 13 are used to obtain input from the user. There are no particular limitations on the type of the operation keys 13 as long as they can obtain input from the user. For example, the operation keys 13 may be buttons provided on the imaging device 14, or may be a keyboard and mouse connected to the information processing device 1, or the panel 12, which is a touch panel, may also serve as the operation keys 13.

[0017] 1(b) shows a digital camera having an imaging device 14 and an information processing device 1, viewed from the opposite side of the orientation in FIG. 1(a). The imaging device 14 is an imaging unit consisting of a lens and a sensor, and captures an image in response to a request from the information processing device 1.

[0018] [Hardware configuration] 2 is a block diagram showing an example of a hardware configuration of the information processing device 1. The information processing device 1 includes a CPU 21, a ROM 22, a RAM 23, an external memory 24, an input unit 25, a display unit 26, a communication I / F 27, an I / O 28, and a communication bus 29.

[0019] The CPU 21 is a central processing unit that controls various devices connected to the system bus 29 and executes a program that implements the processing of the present invention. The ROM 22 stores a BIOS program and a boot program. The RAM 23 is a memory used as the main storage device of the CPU 21. The external memory 24 stores various data such as programs processed by the information processing device 1 and captured images.

[0020] The input unit 25 acquires input from the user via the buttons 11 or operation keys 13 mounted on the information processing device 1. The display unit 26 outputs the calculation results of the information processing device 1 to the display panel 12 in accordance with instructions from the CPU 21. The communication I / O 27 is a communication interface and performs information communication with the outside. The communication I / O 27 may perform communication via wired communication using a USB or the like, or via wireless communication such as a local area network or serial communication, and the type of communication is not limited. The I / O 28 inputs data from the imaging device 14.

[0021] [Explanation of the configuration diagram] FIG. 3 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. The biometric information acquisition unit 31 acquires biometric information. Here, the biometric information acquisition unit 31 acquires a captured image in which a subject appears as biometric information. The consent information management unit 32 acquires and manages consent information indicating whether or not a person has given consent to the use of the acquired biometric information. Here, the consent information management unit 32 can acquire and manage consent information based on input via the operation keys 13. Note that, hereinafter, the term "consent" simply refers to consent to the use of the biometric information as described above. Also, here, biometric information will be described as referring to the entire captured image in which the subject appears. However, for example, a rectangular area (bounding box) in which the subject is detected, or a rectangular area in which the subject's face is detected, etc. may also be used as biometric information, and is not particularly limited as long as it is information in an image that contains information about the subject.

[0022] The feature extraction unit 33 extracts feature amounts from biometric information. Here, the feature extraction unit can extract feature amounts from a captured image, which is biometric information, for a person who has consented to the use of feature amounts based on consent information.

[0023] The output unit 34 outputs (transmits) the consent information, biometric information, and feature amounts to an external device. The registration unit 35 registers the biometric information and consent information. Specifically, the registration unit 35 associates the biometric information, feature amounts extracted from the biometric information, and consent information and stores them in the external memory 24. The registration unit 35 can also associate a person's name obtained by input via the operation keys 13 with the feature amounts and consent information and record them in the external memory 24. Note that instead of outputting the consent information, biometric information, and feature amounts to an external device, the output unit 34 may output the consent information or the biometric information and feature amounts to an external device.

[0024] The matching unit 36 ​​compares the input biometric information with the biometric information registered by the registration unit 35, and determines whether the input biometric information corresponds to anyone of the registered biometric information. The matching result utilization unit 37 includes a control unit 38, a display unit 39, and a recording unit 40, and performs various processes using the matching results determined by the matching unit 36. The control unit 38 controls the imaging device 14 based on the matching results. The display unit 39 displays the matching results on the display panel 12. The recording unit 40 records the matching results in the external memory 24.

[0025] Each process according to this embodiment will be described below with reference to a flowchart, but the process procedure is not limited to that shown. For example, the order of processes may be changed as long as the same processing results are obtained, multiple processes may be integrated, a process described as a single step may be subdivided, or some processes may be omitted. Furthermore, each process may be individually extracted and function independently as a single functional element, and may be used in combination with processes other than those shown.

[0026] [Data structure explanation] The consent information according to this embodiment is information indicating whether or not there is consent regarding the use of feature quantities, as described above, and here, consent / non-consent is expressed as a Boolean value. The initial value of the consent information is set to no consent.

[0027] The registration dictionary according to this embodiment is a database in which information about people that the user of the information processing device 1 wants to match is registered. Here, the registration dictionary stores, for each person, the person's name (character string), consent information, facial image, and feature data in association with each other. Note that multiple pieces of facial image and feature data may be stored for each person.

[0028] [Explanation of the processing flow chart] The biometric authentication process performed by the information processing device 1 according to this embodiment is divided into a registration process for registering biometric information of a person to be authenticated, and an authentication process for determining which of the registered people the input biometric information corresponds to. Furthermore, the information processing device 1 according to this embodiment restricts the use of biometric information of a certain person in the registration process based on the person's consent information acquired based on the biometric information. Hereinafter, the process for restricting the use of biometric information in such registration process will be described with reference to FIG. 4.

[0029] Fig. 4 is a flowchart showing an example of registration processing by the information processing device 1. The processing described in Fig. 4 starts when, for example, a user of the information processing device 1 instructs execution of the registration processing (for example, by pressing the operation button 13 on the display panel 12 shown in Fig. 1). The user according to this embodiment is assumed to be a photographer who photographs a subject using a digital camera including the information processing device 1.

[0030] In S101, the information processing device 1 performs initialization processing for the registration processing. Here, the CPU 21 of the information processing device 1 reads a program from the external memory 24 of FIG. 2 and makes the registration processing, which will be described later, operable. As part of the initialization processing, the CPU 21 of the information processing device 1 also loads a registration dictionary from the external memory 24 and makes it available for addition. The processing described below in FIG. 4 is realized by the CPU 21 of the information processing device 1 executing the necessary programs.

[0031] In S102, the biometric information acquisition unit 31 acquires biometric information (here, an image of an individual's face captured by the imaging device 14 in FIG. 1). At this time, the biometric information acquisition unit 31 controls the imaging device 14 using known imaging technology so that the focus and exposure are suited to the target's face, and acquires the captured image. Note that the following description will be given assuming that the captured image that becomes the biometric information contains one subject, but the captured image may contain multiple subjects, and the processing described below may be performed on each of them individually.

[0032] In S103, the consent information management unit 32 acquires and manages consent information for individuals (subjects) appearing in the captured images acquired in S102. As described above, the consent information acquired here indicates whether or not there is consent to the use of images showing the individual's face, i.e., their biometric information.

[0033] The "use of biometric information" according to this embodiment includes at least one of photographing and recording a face image, extracting features from the face image, performing a registration process to register the features, performing an authentication process based on the features (here, a process to identify a person), and performing some kind of control using the results of the authentication process. Details of these usage methods will be described later. The information processing device 1 according to this embodiment restricts the use of this biometric information based on consent information.

[0034] The consent information management unit 32 according to this embodiment can acquire consent information based on, for example, a user's operation via the display panel 12 or the operation keys 13. Specifically, the consent information management unit 32 can display, on the display panel 12, a message indicating that biometric information will be used and a message for accepting a choice as to whether or not to agree to the use, and acquire the result of the choice as consent information.

[0035] It should be noted that the user name is assumed to be set in advance (for example, based on the user's login or the user's input during operation). However, in order to prevent so-called spoofing, in which a person other than the user arbitrarily performs an operation agreeing to the user's consent, the user may be set based on a captured image different from the biometric information. For example, an imaging device (not shown) may be further provided to capture an image of the operator of the operation keys 13, and the user may be set based on the image captured by such an imaging device (by a known person recognition process). In such a case, it may be additionally determined whether the set user and the person in the biometric information acquired in S102 are the same person, and if they are not the same person, the process of FIG. 4 may be terminated at that point.

[0036] Furthermore, although the explanation has been given assuming that consent information is acquired based on user input via the operation keys 13, the present invention is not limited to such processing as long as it is possible to acquire whether or not a user has consented. For example, if it is detected that a user has performed a predetermined action (e.g., a gesture indicating an instruction by voice, etc.), it may be determined that the user has consented to the use of their biometric information, and image capture by the image capture device 14 and processing for using the biometric information (e.g., authentication processing) may be performed. Such processing allows consent to be expressed simply by making a gesture, thereby improving convenience. The gesture may be, for example, a peace sign using the hand, or a gesture of raising or lowering the hand, and the gesture may be recognized using known motion recognition technology.

[0037] In S104, the feature extraction unit 33 determines whether or not the user has consented to the use of the biometric information based on the consent information acquired in S103. If the user has consented, the process proceeds to S105, and if not, the process proceeds to S107.

[0038] In S105, the feature extraction unit 33 extracts features from the biometric information of a user who has consented to the use of the biometric information. In this embodiment, the biometric information is a captured image of an individual's face, and the feature extraction unit 33 identifies the position of the face in the captured image and then extracts features from the face. Any known image processing technology can be used in the process of identifying the position of the face in the captured image and the process of extracting features from the person's face. For example, the technology described in Non-Patent Document 1 may be used to identify the position of the face in the captured image. Furthermore, for example, a method using deep learning may be used to extract features from the person's face, or a method such as Local Binary Pattern (LBP) or HoG (Histogram Oriented Gradient) may be used.

[0039] In S106, the registration unit 35 associates the extracted feature with the biometric information, consent information, and personal name, and registers them in a registration dictionary. The registration unit 35 can acquire the personal name by accepting user input from an input unit (not shown) via the operation keys 13 in FIG. 1. The format of the personal name according to this embodiment is not particularly limited as long as it is set so as to enable distinction between registered persons. For example, the personal name may be a real name, a nickname, a number, a symbol, or the like. Furthermore, although this embodiment will be described assuming that the registration unit 35 registers the consent information, the processing may be performed by a different functional unit, such as the consent information management unit 32 registering the consent information.

[0040] In S107, the registration unit 35 records the biometric information, consent information, and person's name in association with each other. Here, feature extraction is not performed for individuals who have not given consent, and only biometric information, i.e., a face image, is registered for the consent information and person's name.

[0041] In S108, the registration unit 35 determines whether to end the registration process. Here, for example, if the user has input to end the registration process, it may be determined that the registration process is to be ended. If it is determined that the registration process is to be ended, the process of FIG. 4 ends; if not, the process returns to S102.

[0042] FIG. 5 is a diagram showing an example of the contents of the registration dictionary after the registration process. In FIG. 5, each column displays a "No." indicating an index, a "Person's Name" character string indicating a person's name, "Consent Information" that displays a check mark if consent to the use of biometric information is indicated in the consent information, a "Facial Image" displaying an image of the subject, and a "Feature Amount" that schematically displays the extracted and recorded feature amount. Each row and column represents one person, and three people, "Mr. A," "Mr. B," and "Mr. C," are registered in Nos. 1 to 3, respectively. In FIG. 5, consent to the use of biometric information has been obtained for "Mr. A" and "Mr. C," and feature amounts have been extracted and registered. Consent has not been obtained for "Mr. B," and feature amounts have not been registered. This registration dictionary is used in the authentication process, which will be described later.

[0043] [effect] This type of processing can restrict the use of biometric information for people who have not given their consent. In particular, by preventing feature registration, it is possible to prevent individuals from being identified through the authentication processing described below (i.e., restricting face authentication processing). Furthermore, by registering only an image when consent has not been given, if consent is obtained later, it becomes possible to extract and register feature information without obtaining new biometric information, improving convenience (since it is no longer necessary to obtain an image and consent at the same time, convenience is improved for both the user and the person being registered).

[0044] [Variation 1-1] [Variations in how consent is obtained] In the present embodiment, the information processing device 1 acquires consent from a subject by displaying on its display panel 12 a message indicating that biometric information will be used and a message (an image prompting consent) that accepts the user's consent. However, the method of acquiring consent information is not limited to this, as long as it is possible to confirm whether the user has consented. For example, the information processing device 1 may acquire consent information based on an input from an application on a mobile terminal such as a smartphone (not shown). In such a case, the information processing device 1 may communicate with the mobile terminal and request the mobile terminal to transmit consent information when acquiring consent information (S103 in FIG. 4). Upon receiving the request, the mobile terminal displays an image prompting consent on its screen and receives input indicating consent from the user of the mobile terminal, thereby acquiring consent information and transmitting it to the information processing device 1. At this time, to detect impersonation and prevent consent by a different person, the information processing device 1 may verify that the person being registered and the user of the mobile terminal are the same person (perform identity authentication). For this purpose, the information processing device 1 may perform identity authentication on its own or using a security function of the mobile terminal. For example, the information processing device 1 can transmit a facial image of a person to be registered to a mobile terminal, and the mobile terminal can use a known facial authentication method to verify whether the user and the facial image are the same person. The facial image of the user of the mobile terminal is captured, for example, by the mobile terminal's internal camera. With this configuration, consent obtained through impersonation can be avoided and consent information can be obtained from the subject. Note that the personal authentication is not limited to such facial authentication, and can be any authentication process using, for example, fingerprint authentication provided in the mobile terminal, or authentication process using an electronic certificate, etc.

[0045] [Variation 1-2] [Processing when no image is registered] In the present embodiment, it has been described that a facial image is registered even if consent is not obtained. However, it is also possible to prevent registration of a facial image without consent. According to such processing, by not storing a facial image without consent, it is possible to perform processing that takes user privacy into greater consideration (respects individual wishes more). Note that, compared to consent information, facial images can often be acquired relatively easily through imaging processing or acquisition processing via SNS (Social Networking Service), etc. Therefore, if priority is given to light processing speed and ease of operation in the registration processing, facial images may be registered regardless of whether consent is obtained, as described in S107.

[0046] [Variation 1-3] [Images and names are not required for the registered dictionary] In the first embodiment, it has been described that information such as a facial image and a person's name is linked to consent information and registered in the registration dictionary. However, the information registered in the registration dictionary for consent information is not particularly limited to information necessary for matching, and a person's name is not essential. For example, in the registration dictionary, only a facial image may be linked to consent information and registered, or a feature may be linked to consent information and registered, or both of these may be linked to consent information and registered.

[0047] [Embodiment 2] The information processing device 1 according to the first embodiment restricts the use of biometric information in the above-described registration process based on the presence or absence of consent. The information processing device 1 according to the second embodiment acquires consent information in the same manner as the first embodiment, and restricts the use of biometric information in the authentication process based on the presence or absence of consent. The information processing device 1 according to this embodiment has the same hardware configuration and functional configuration as the information processing device 1 according to the first embodiment, and can execute the same processes, so redundant explanations will be omitted.

[0048] 6 is a flowchart showing an example of authentication processing performed by the information processing device 1 according to this embodiment. For example, the authentication processing is started when a user of the information processing device 1 instructs the execution of the authentication processing (for example, by pressing the operation button 13 on the display panel 12 shown in FIG. 1). Note that the authentication processing according to this embodiment is performed after the registration processing described in the first embodiment is executed.

[0049] In S111, the information processing device 1 performs initialization. The initialization process is performed for the registration process. Here, the information processing device 1 reads a program from the external memory 24 of FIG. 2 and makes the authentication process, which will be described later, operable. As part of the initialization process, the information processing device 1 also loads a registration dictionary from the external memory 24 and makes it available for use.

[0050] In S112, the biometric information acquisition unit 31 acquires biometric information (here, an image of an individual's face from the imaging device 14 in FIG. 1). Here, the biometric information acquisition unit 31 acquires, as biometric information, an image of a candidate to be recorded by the digital camera from the imaging device 14. The captured image acquired here is displayed on the display panel 12 as a live view in the processing described below. The user (photographer) can determine the composition of the image to be recorded or the timing to press the shutter button 11 while checking the live view. Note that the captured image acquired here may include multiple faces.

[0051] In S113, the feature extraction unit 33 extracts features from the biometric information. Here, as described in the registration process, the feature extraction unit 33 identifies the position of the face in the captured image and then extracts features from the face of the user who has agreed to the use of their biometric information. If the image includes multiple faces, features are extracted from each of the multiple faces.

[0052] In S114, the matching unit 36 ​​identifies a person by comparing the feature values ​​extracted in S113 with the feature values ​​of individuals registered in the registration dictionary. Here, the matching unit 36 ​​compares the feature values ​​extracted in S113 with the biometric information registered by the registration unit 35 to determine whether the extracted feature value matches or does not match any of the registered biometric information. For example, the matching unit 36 ​​calculates the similarity between the extracted feature value and the registered feature value. If the similarity value exceeds a predetermined threshold, the matching unit 36 ​​determines that the extracted person is the individual. If no person exceeds the threshold, the matching unit 36 ​​determines that the extracted person is not a person. Any known method can be used to calculate the similarity. For example, cosine similarity or L2 distance may be used as the similarity. Since the L2 distance is a measure in which the value decreases as the distance between feature values ​​decreases, the reciprocal of the L2 distance may be converted into the similarity value. Here, as described in the description of FIG. 5 of the registration process, since "Mr. B" does not have any registered feature values, matching cannot be performed for Mr. B. In addition, in the case of Mr. B, since he has not consented to the use of his biometric information, no features are extracted.

[0053] In the following S115 to S117, the matching result utilization unit 37 performs various processes using the matching result output by the matching unit 36. These processes will be described in detail below. In S115, the display unit 39 displays the matching result on the display panel 12. FIG. 7 is a schematic diagram showing an example of the matching result displayed by the display unit 39. In FIG. 7, a face frame G101 of a person displayed as "Unknown", a face frame G102 of a person displayed as "Mr. C", and a face frame G103 displayed as "Unknown" are arranged on the display panel 12.

[0054] Here, G101 is "Mr. B," but as mentioned above, "Mr. B" has not consented to the use of his or her biometric information, so the matching process was not performed. Therefore, the result is determined to be "Unknown," meaning that the person is not a member of the registered dictionary. Also, here, G102 is "Mr. C," and as mentioned above, "Mr. C" has consented to the use of his or her biometric information, so the result is determined to be "Mr. C" by matching the feature values ​​of "Mr. C" in the registered dictionary. G103 indicates that the person was not matched with anyone registered in the registered dictionary and was determined to be "Unknown." In other words, in this example, only the name of "Mr. C," who consented, is displayed; the matching results for other people are displayed as "Unknown," with no indication of their identity. In this way, the use of biometric authentication can be restricted by preventing matching of people without consent. The display shown in Figure 7 is a live view display on a digital camera, and the user can decide whether to save the image by pressing the shutter button while viewing this display.

[0055] In S116, the control unit 38 controls the imaging parameters of the imaging device 14 (here, focus control and exposure control) based on the matching result. For example, the control unit 38 can control the imaging parameters so that the focus and exposure are controlled for the face of the person whose name has been identified based on the matching result. In other words, the control unit 38 may control the imaging parameters so that a person whose name has not been identified (who has not consented to the use of their biometric information) is not used as a reference in the control of imaging parameters. Here, the control of focus and exposure for a specific person can be performed using known AF / AE technology, and a detailed description thereof will be omitted. Note that, here, if there are multiple people whose names have been identified in the captured image, it is assumed that it is possible to determine in advance which of the people to focus and expose (for example, a priority can be set for each person).

[0056] In S117, the recording unit 40 records the captured image using the matching result. Here, the recording unit 40 captures a captured image when the user presses the shutter button 11, and records only the matching results of persons who have given their consent as tag information for the captured image (i.e., the recording unit 40 can control so that the matching results of persons who have not given their consent are not recorded in association with the image). In the example of FIG. 7, as described above, information about "Mr. C," who has been identified with consent, is recorded as tag information. As shown in FIG. 7, information about the face frame of G102 for "Mr. C" may also be recorded together with the captured image. According to this processing, it is possible to obtain a captured image in which the matching results are linked and recorded only for persons who have given their consent.

[0057] In step 118, the matching result utilization unit 37 determines whether or not to terminate the authentication process. Here, for example, if the user has input to terminate the authentication process, it may be determined that the registration process is terminated. If it is determined that the authentication process is terminated, the process in FIG. 6 is terminated; if not, the process returns to S112.

[0058] [effect] According to this type of processing, it is possible to restrict the use of biometric information for persons who have not given their consent. In particular, it is possible to restrict the use of biometric authentication processing or processing results for persons who have not given their consent. In the authentication processing, biometric authentication processing is performed on the input face image to determine whether or not consent has been given, but the results are not used for persons who have not given their consent, which effectively restricts the biometric authentication processing. Therefore, it is possible to restrict the biometric authentication processing while respecting the individual's will.

[0059] [Variation 2-1] [Extract features regardless of consent] In the registration process according to this embodiment, the feature information of a person who has not given consent is not registered. However, the feature information of a person may be registered even if the person has not given consent. In this case, the information processing device 1 restricts the use of biometric information by not using the matching results of a person who has not given consent during authentication processing, rather than not extracting feature information from the person.

[0060] [Variation 2-2] [Variations of consent information] In the present embodiment, the consent information has been described as information indicating only whether or not consent has been given, but it may also be possible to specify the type of consent, such as what consent is given for. The type of consent may be, for example, consent to using acquired biometric information to control imaging parameters such as AF / AE for the subject, consent to recording images of the subject, or consent to displaying the matching results, or other consent to various processes. Such processing can achieve restrictions that respect the individual's wishes in more detail.

[0061] [Variation 2-3] [Features are extracted during authentication without registering them] In this embodiment, the feature amounts of the registered dictionary are described as being extracted and stored during the registration process, but the feature amounts extracted during the authentication process may also be registered in the registered dictionary. In that case, during the authentication process, the consent information of the registered dictionary can be referenced, and the feature amounts can be extracted only from persons who have consented, and compared with the input feature amounts.

[0062] [Variation 2-4] [If no match is found during the matching process, register as no consent] In the comparison process, if the extracted feature is determined to match no one by comparing it with the registered dictionary, the information processing device 1 may additionally register the person corresponding to the feature in the registered dictionary as a person without consent. In this case, in the example of FIG. 7, G103 is a person determined to match no one of the people registered in the registered dictionary, so the facial image of G103 is registered. Here, the name set for a person without consent is assumed to be a predetermined character string that is set in advance. In addition, in this case, the person determined to match no one may be registered in association with the feature extracted from the facial image. According to this process, it is possible to explicitly identify the person as having not given consent by looking at the registered dictionary, thereby improving convenience for the user of the information processing device 1.

[0063] [Variation 2-5] [If consent is not given, re-acquire] Furthermore, the information processing device 1 may redisplay a display for accepting a person who has not given consent, asking them to choose whether or not to consent to the use of their biometric information, and prompt them to consent to the use of their biometric information again (re-acquire the consent). That is, the consent information management unit 32 of the information processing device 1 can accept an instruction to change the consent information. In this case, if consent is obtained by instructing the subject to make a gesture, consent information can be obtained simultaneously from multiple people in the photo, which is highly convenient. For a person whose consent information indicating consent has been reacquired, features are extracted from the facial image, and the features and consent information are newly registered in the registration dictionary. Furthermore, when previously acquired consent information indicates non-consent, the consent information management unit 32 of the information processing device 1 can also accept an instruction to change the consent status from non-consent to consent. In response to the consent information management unit 32 accepting the above-mentioned change instruction, the feature extraction unit 33 of the information processing device 1 extracts features from the registered biometric information (facial image) of the person corresponding to the consent information. Then, the registration unit 35 associates the facial image with the changed consent information and registers it. According to this modified example, there is no need to obtain consent and acquire a facial image at the same time, or it becomes possible to change a consent decision once made, thereby improving convenience for users who obtain consent and for people who give consent.

[0064] [Variation 2-6] [Variations in the use of authentication results] Furthermore, the recording of an image of a person who has not given consent may be restricted. For example, in the example of FIG. 7, the face frames G101 and G103 of the person who has not given consent may be filled in (for example, with a single color of black) and output as an image. In this way, the information processing device 1 can restrict the use of biometric information by outputting an image from which areas containing biometric information have been deleted. According to such processing, an image can be recorded with the consent of all people whose face areas appear in the image. Also, for example, the information processing device 1 may be configured not to record an image unless the consent of all people appearing in the image has been obtained.

[0065] Although this method of restricting the recording of images is effective in terms of respecting the will of the individual, it is a strong restriction. From this perspective, for example, images can be recorded (output as is) for people who have not given their consent, but images of people who have refused to use their biometric information cannot be recorded (by blacking out the face area), so that images can be output that, to some extent, balances convenience and respect for the will of the individual.

[0066] Furthermore, in the present embodiment, when displaying the matching result, the explanation has been given assuming that the presence or absence of consent is displayed on a live view image as shown in FIG. 7. However, such display of the presence or absence of consent is not limited to the live view, but may be displayed on an output image. The information processing device 1 in this embodiment has a function for displaying recorded captured images, as is generally provided in digital cameras, and consent information may be superimposed on such captured images in the same manner as in the live view of FIG. 7. This type of processing makes it possible to check the presence or absence of consent for people whose images have been captured in the past.

[0067] Furthermore, as described in Modification 2-2, when obtaining consent for each process, whether or not to use the matching results for each process may be switched depending on the content of the consent. For example, if a person consents to the recording of an image but does not consent to the adjustment of focus or exposure using the matching results, restrictions on the use of biometric information are controlled for each such process. In this case, for example, the focus or exposure is not adjusted for that person, and an image of that person is recorded. In this way, restrictions can be set that are more in line with the person's intentions.

[0068] [Variation 2-7] [Variations of information processing devices] Although the information processing device 1 according to this embodiment has been described as being part of a digital camera, the present invention is not limited to such a configuration as long as it can perform similar processing. For example, the information processing device 1 may be a smartphone with a camera or a network camera with a pan-tilt-zoom function that allows the angle of view to be adjusted. In this case, one possible method for using the matching results is to control the pan-tilt-zoom to capture an identified individual. When the information processing device 1 is used for such purposes, restricting the use of biometric information for persons who have not consented can prevent invasion of privacy and prevent misuse. Furthermore, when the information processing device 1 is a network camera, a separate server may be provided to manage the network camera and record captured video, and the server may perform the consent information acquisition or registration process. In such a configuration, when acquiring consent information, a UI for acquiring consent information on the server, i.e., a display indicating the use of biometric information and a display allowing the user to choose whether or not to consent, may be displayed on a display device. The user may then input their consent using an input device such as a mouse or keyboard.

[0069] [Embodiment 3] In this embodiment, an example will be described in which consent information, biometric information, and feature amounts registered by the information processing device 1 according to embodiment 1 are output to another device. In addition, a case will be described in which a digital camera, like the information processing device 1, is used as the other device used here.

[0070] 8 is a schematic diagram showing an example of use of the information processing device 1 according to this embodiment. The information processing device 1 is the same as that in the first embodiment, so a duplicated description will be omitted.

[0071] The network 15 is a network through which the devices communicate with each other. The network 15 may be, for example, a local area network (LAN), but the form of the network is not particularly limited as long as it can connect the devices so that they can communicate with each other. For example, the network 15 may be wireless or wired.

[0072] The information processing device 16 is a second information processing device to which the information processing device 1 according to this embodiment outputs consent information, and is assumed to be a digital camera in this example. The information processing device 16 according to this embodiment has the same hardware configuration and functional configuration as the information processing device 1, and redundant explanations will be omitted. The information processing device 16 is also capable of sending and receiving requests and data from other devices.

[0073] The mobile terminal 17 is a smartphone to which the information processing device 1 according to this embodiment outputs consent information. The mobile terminal 17 is capable of sending and receiving requests and data from other devices, similar to the information processing device 16. The information processing device 1 according to this embodiment outputs, in addition to consent information, feature amounts of persons who have given consent to the mobile terminal 17. This allows the mobile terminal 17 to use the acquired feature amounts of persons who have given consent. Conversely, for persons who have not given consent, the information processing device 1 cannot output consent information or feature amounts.

[0074] The PC 18 is a personal computer to which the information processing device 1 outputs consent information. The PC 18 according to this embodiment is capable of sending and receiving requests and data from other devices, similar to the information processing device 16. Furthermore, the information processing device 1 outputs, in addition to the consent information, the feature values ​​of the person who has given consent to the PC 18. This allows the PC 18 to use the acquired feature values ​​of the person who has given consent.

[0075] Fig. 9 is a flowchart showing an example of output processing by the information processing device 1 according to this embodiment. The processing described in Fig. 9 starts when, for example, a user of the information processing device 1 instructs execution of the output processing (for example, by pressing the operation button 13 on the display panel 12 shown in Fig. 1). Here, an example will be described in which the information processing device 1 described in Fig. 8 outputs data included in the registered dictionary, that is, consent information, biometric information, and feature amounts, to the information processing device 16.

[0076] In S121, the output unit 34 acquires the registered dictionary registered by the registration unit 35. As described above, the registered dictionary includes consent information, biometric information, and feature amounts. As described in the first embodiment, this consent information is acquired by the consent information management unit 32. The output unit 34 according to this embodiment acquires the consent information held in the registered dictionary, but may also acquire consent from an individual as in the first embodiment.

[0077] At S122, the output unit 34 establishes a connection with the destination information processing device 16 and makes it ready to output data. At S123, the output unit 34 references the registered dictionary held by the destination information processing device 16 and compares it with the registered dictionary held by the information processing device 1 to detect differences. FIG. 10 is a schematic diagram illustrating the above-described difference detection and synchronization process performed by the information processing device 1 according to this embodiment. FIG. 10(a) shows the registered dictionary held by the information processing device 1, and FIG. 10(b) shows the registered dictionary held by the destination information processing device 16. Here, the output unit 34 compares the registered dictionaries and detects differences. Here, the process of detecting differences is a process of identifying information not stored in the external device as a difference. In this example, although "Mr. A" at No. 1 in each registered dictionary is the same person, the difference is that different facial images and feature values ​​are registered in each. Furthermore, for Nos. 2 and 3 in FIG. 10(a), the person not registered in FIG. 10(b) is also a difference.

[0078] In order to determine whether or not the two people are the same person, the matching unit 36 ​​can perform the above-mentioned matching process on each feature. The matching unit 36 ​​may also store unique numbers, symbols, etc. that identify individuals in a registered dictionary, and determine whether or not the two people are the same person by comparing such information. As mentioned above, the "Person's Name" field contains a person's name or nickname, and is not necessarily a unique number or symbol, so it may not be used to determine whether or not the two people are the same person.

[0079] In S124, the output unit 34 displays to the user which information to output, and then confirms whether or not it is OK to output the information. Here, the output unit 34 displays a screen such as that shown in Fig. 10(a) and Fig. 10(b). In this example, the output unit 34 outputs the facial image and feature amount of No. 1 and the facial image and feature amount of No. 3 from the information processing device 1. Along with this display, the information processing device 1 receives input from the user via an input unit (not shown) regarding whether or not to output the information.

[0080] In S125, if there is an input from the user permitting output (Yes in S125), the output unit 34 proceeds to S126, otherwise it terminates the processing in Fig. 10. In S126, the output unit 34 outputs the consent information, biometric information, and feature amounts.

[0081] FIG. 10(c) shows the registration dictionary of the information processing device 16 after output. The registration dictionary shown in FIG. 10(c) reflects information from the registration dictionary (a) of the information processing device 1, which was not held before the output. In this embodiment, the information processing device 1 also acquires the difference from the information processing device 16, and synchronization with the registration dictionary held in the information processing device 16 is performed. Therefore, the registration dictionaries of the information processing device 1 and the information processing device 16 are synchronized, and both information processing devices store the registration dictionary shown in FIG. 10(c). Note that, once the registration dictionary of the information processing device 1 is output, the registration dictionaries of both information processing devices do not necessarily need to be synchronized, and only the registration dictionary held in the information processing device 16 may be updated.

[0082] In FIG. 10(c), multiple facial images and feature amounts are registered for one person ("Mr. A"), and matching can be performed by matching each feature amount and calculating a representative matching result from the multiple matching results. Any known method for matching data can be used to calculate the representative matching result. For example, matching can be performed by selecting the matching result with the highest similarity or by averaging the similarities. According to this processing, by registering multiple facial images and feature amounts for one person, matching can be performed based on facial images and feature amounts of that person taken under multiple shooting conditions, thereby improving matching accuracy.

[0083] [effect] With this configuration, the consent information, biometric information, and feature amounts acquired by the information processing device 1 can be output to another device. By outputting the consent information to another device and making it available on the output destination device, it is possible to eliminate the need to make some kind of contact with the person to be processed to acquire the consent information. For example, in a group of family or friends who want to share consent information, the consent information can be synchronized between the information processing devices, thereby reducing the number of cumbersome consent acquisitions.

[0084] Furthermore, by outputting the biometric information and feature quantities together with the consent information, the procedure of acquiring the biometric information and feature quantities again on the output destination device can be omitted. By outputting the feature quantities, the feature quantities can be used for facial recognition on another device. Additionally, by outputting the biometric information (here, a facial image), the feature quantities can be extracted again from the facial image when the facial recognition model is updated. Furthermore, with this configuration, the user of the device can visually check the registration status, such as which individuals have consented or are registered, thereby improving convenience from a management perspective.

[0085] [Variation 3-1] [Output Destination Variations] In the present embodiment, the information processing device 1 and the information processing device 16 are both digital cameras. However, other devices capable of executing similar processing may be used. For example, a smartphone, a personal computer, or other device may be used as the information processing device (1 or 16). Such information processing devices may be required to search for a specific person from a large number of captured images. Therefore, the information processing device may be configured to search only for people who have given consent based on the output consent information. That is, when a user specifies a person to search, the consent information may be referenced and the user may specify only people who have given consent. This processing allows only images of people who have given consent to be searched, thereby realizing restrictions that better respect individual intentions. Furthermore, the information processing device 16, which is the output destination, may be a device that provides cloud services. Even in such a case, the information processing device 1 according to this embodiment can output consent information to the output destination device via a network.

[0086] [Variation 3-2] [Variations of consent information] The information processing device 1 may also be configured to allow the user to specify consent to outputting the consent information or biometric information to another device. With such a configuration, the consent information or biometric information can be prevented from being inadvertently output to another device, while allowing the target device to use the information or biometric information.

[0087] [Variation 3-3] [Select consent information to output] Although the information processing device 1 according to the present embodiment has been described as automatically synchronizing and outputting the output destination and consent information, the user may be able to select the information to be synchronized. In this case, the items that the user can select are limited to those for which consent information indicates consent. With this configuration, the user can select and synchronize only the consent information that is necessary.

[0088] [Variation 3-4] [Verification of output information] In addition, although the present embodiment has been described with reference to an example in which the registered dictionary is output to another device and synchronized, it is also possible to output and synchronize only a portion of the information contained in the registered dictionary. For example, only consent information and personal names may be output and synchronized, or only biometric information or features may be output and synchronized.

[0089] [Embodiment 4] [Consent to AI processing and use of the processing results] In the first to third embodiments, examples of restricting the use of biometric information were described. However, in this embodiment, a case where the target of use restriction is not limited to biometric authentication, but the use of AI processing and its processing results is restricted will be described. In this embodiment, a method for restricting AI processing targeting customers who have not given consent by identifying customers (users) appearing in images captured by cameras installed in a commercial facility and consent information related to those customers will be described. It is assumed here that multiple services using AI processing are provided in the commercial facility, and consent is obtained for each of those services.

[0090] In this embodiment, the term "AI processing" refers to processing that performs intelligent processing, identification, judgment, etc. using technology such as machine learning, and specifically, AI processing is assumed to include face detection, face recognition, age / gender estimation, etc. Furthermore, the AI ​​processing according to this embodiment is not limited to image processing, but also includes text analysis, analysis of other sensing data, etc.

[0091] The processing performed by the information processing device 1 according to this embodiment will be described below. FIG. 11 is a schematic diagram showing an example of the system configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment functions as a server that controls the system of this embodiment. The information processing device 1 according to this embodiment is also equipped with input devices such as a mouse and keyboard, as well as a screen display panel for operation, and can accept operations from the administrator of the system of this embodiment. The network 15 is the same as that of embodiment 3. The other hardware configurations are the same as those of the information processing device 1 of embodiment 1, so duplicated explanations will be omitted.

[0092] Imaging device 41 is a camera for facial recognition payment, and is composed of a camera unit and a communication device. Imaging device 41 transmits the captured facial image to information processing device 1 via network 15. Surveillance camera 42 and surveillance camera 43 are a first surveillance camera and a second surveillance camera, respectively. Surveillance camera 42 and surveillance camera 43 are each composed of a camera unit and a communication device, and transmit the captured image to information processing device 1 via network 15.

[0093] 12 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. The biometric information acquisition unit 51 acquires images of an individual, in this case, a customer, from the face recognition payment camera 41, the monitoring camera 42, and the second monitoring camera 43.

[0094] The individual identification unit 52 identifies the individual appearing in the image acquired by the biometric information acquisition unit 51. The consent information management unit 53 acquires and manages consent information indicating what AI processing an individual has consented to and the use of the processing results.

[0095] The AI ​​processing unit 54 is an AI processing unit that performs AI processing on the image acquired by the biometric information acquisition unit 51. The AI ​​processing unit 54 according to this embodiment refers to the consent information and executes only the AI ​​processing to which the individual who is the target of the AI ​​processing has consented. In other words, the AI ​​processing unit 54 according to this embodiment restricts the AI ​​processing based on the consent. The content of the AI ​​processing will be specifically described later.

[0096] The AI ​​processing result utilization unit 55 utilizes the results processed by the AI ​​processing unit 54. The AI ​​processing result utilization unit 55 creates information that will be the basis for services to be provided to individuals.

[0097] The consent condition storage unit 56 stores consent conditions, including conditions related to consent for individuals, and presents them to the user. The consent conditions according to this embodiment include information such as the content of the AI ​​processing, the data used by the AI ​​processing, the purpose of the AI ​​processing, information about the organization performing the AI ​​processing, information about the organization managing the data used by the AI ​​processing, the services provided to the customer, and the period during which the AI ​​processing will be performed. These consent conditions are presented to the customer in advance and consent is obtained from the customer, and are conditions for implementing the AI ​​processing described below. The output unit 57 outputs and displays the information created by the AI ​​processing result utilization unit 55 and the consent conditions stored in the consent condition storage unit 56 to an external device.

[0098] The specific steps of the processing performed by the information processing device 1 according to this embodiment are described below. The processing according to this embodiment is divided into two steps: a pre-processing step in which consent conditions are presented to a customer and consent information is registered in a registration dictionary together with a facial image or feature amount, and an AI processing step in which AI processing is performed based on the consent information registered in the pre-processing step.

[0099] In the pre-processing according to this embodiment, similar to the registration processing described in embodiment 1, feature amounts, biometric information, consent information, and personal names are linked and registered in a registration dictionary. Since this embodiment targets customers who use commercial facilities, this registration processing is performed, for example, using the customer's smartphone when entering the commercial facility, or the registration processing is performed in advance on a website provided by the commercial facility. Another difference between the processing performed by the information processing device 1 according to this embodiment and that in embodiment 1 is that the consent information includes consent to multiple AI processes.

[0100] FIG. 13 is a diagram schematically illustrating the contents of a registered dictionary in this embodiment. The columns for No., person's name, facial image, and feature value shown in FIG. 13 are the same as those in embodiment 1, and therefore will not be described here. The consent information column displays services using AI processing, such as "Facial Recognition Payment," "Crowd Analysis," and "Visitor Trend Analysis," with checkboxes indicating consent for each service. In the example shown in FIG. 13, "Mr. A" consents to all services, while "Mr. B" does not. On the other hand, "Mr. C" does not consent to "Facial Recognition Payment" and "Visitor Trend Analysis," but does consent to "Crowd Analysis." Thus, the consent information according to this embodiment describes the purpose (type) of AI processing and includes information indicating whether the user consents to each AI processing. Note that the AI ​​processing shown in FIG. 13 is merely an example; other types of AI processing, such as face detection, facial recognition, or age / gender estimation, may also be listed.

[0101] This consent information is acquired by the operator of the commercial facility presenting the conditions of each consent information (consent conditions) to the customer when the customer enters the commercial facility and confirming the customer's willingness to consent. For example, the address of a web page indicating the consent conditions is presented to the customer, and the customer who accesses the address is asked to confirm the consent conditions on a smartphone or other device, and consent is acquired based on the user's input. The address may be presented as text on a poster or the like, or as a two-dimensional barcode to reduce the customer's input effort. The form of the address is not particularly limited as long as it is accessible to the user. The address may also be distributed to the customer's smartphone or other device at a specific location using contactless communication technology.

[0102] Figure 14 is a schematic diagram showing an example of consent conditions presented to a customer. The consent conditions are presented in this way, and consent information is acquired by having the customer input whether or not they consent. The consent conditions shown in Figure 14 present information about services that use the three AI processes mentioned above: "Facial Recognition Payment," "Congestion Analysis," and "Visitor Trend Analysis."

[0103] "Facial recognition payment" is a service that identifies people in an image and uses payment information (such as bank account or credit card information) pre-linked to that person to make payments such as purchasing goods. "Crowd analysis" is a service that counts the number of people in an image, visualizes the number of people within the camera's field of view, and presents the visualization results to customers. "Visitor trend analysis" is a service that estimates the age and gender of customers to analyze trends among visitors and provide advertisements based on their age and gender on signage in commercial facilities, etc. All of these services that use AI processing can be realized by using well-known AI processing such as face detection, face recognition, or age and gender estimation.

[0104] In the example of the service item "1. Facial Recognition Payment" in Figure 14, the information presented includes the AI ​​processing details 201, the data processed by the AI ​​202, the service provided to the customer by the AI ​​processing 203, and the consent period 204. The consent period indicates the period during which the consent setting is valid. In the example of Figure 14, the customer can enter the details, but a specific date or period may be presented and the user's selection accepted. After confirming these conditions, the customer can indicate their consent by selecting a check box 205. In this case, an unselected check box means that the user has not consented. For other service items, the same information as "1. Facial Recognition Payment" is presented. Note that if a length is set as the consent period, the period for executing the AI ​​processing can be set by obtaining the current time when the AI ​​processing is executed.

[0105] When the customer has finished inputting whether or not he / she consents to all services, he / she presses the consent button 206 to decide his / her intention to consent to the consent conditions, i.e., the consent information, and the consent information is acquired by the information processing device 1.

[0106] This configuration makes it possible to obtain consent information from customers after having them confirm the services they will receive and the data they will provide (data processed by AI). Therefore, it is possible to provide only AI-processed services to customers that the customer has consented to, and conversely, it is possible to prevent AI processing that the customer has not consented to from being performed, i.e., to restrict the execution of AI processing. Therefore, it is possible to provide services that are in line with the customer's wishes, improving convenience for both the service provider and the customer.

[0107] [AI processing] Next, the AI ​​processing according to this embodiment will be described. Fig. 15 is a flowchart showing an example of the AI ​​processing executed in this embodiment. Note that the processing shown in Fig. 15 is executed after the above-mentioned preliminary processing is completed. If the processing shown in Fig. 15 is executed before the preliminary processing is completed, there is no person who is said to have consented to the AI ​​processing, and the AI ​​processing with consent described below will not be executed.

[0108] In S131, the information processing device 1 performs initialization processing for AI processing. Here, the information processing device 1 loads the registered dictionary registered in the pre-processing and makes it readable.

[0109] In S132, the biometric information acquisition unit 51 acquires an image of an individual from a camera. Here, it is assumed that the biometric information acquisition unit 51 acquires images from the monitoring cameras 42 and 43.

[0110] In S133, the individual identification unit 52 identifies who the person in the acquired image is. Here, the individual identification unit 52 identifies, for all people in the image, whether they correspond to any person registered in the registration dictionary or whether they do not match any person. This identification method can be performed in the same way as in S114 of the second embodiment. The information processing device 1 according to this embodiment extracts features from the captured image regardless of whether consent is obtained, to identify an individual, and restricts the use of subsequent biometric information for that individual based on the consent information.

[0111] In S134, the consent information management unit 53 acquires the consent information of the identified person. In the example of Fig. 13, the feature amount and the consent information are registered in association with each other, so the consent information management unit 53 refers to the consent information of the identified person. Note that here too, for a person who is determined not to match anyone by the matching process, no consent information exists, and therefore the person is considered to have not consented to any services.

[0112] In S135, the AI ​​processing unit 54 restricts the use of the biometric information based on the consent information, and then executes AI processing on the acquired image (including the face). Here, the AI ​​processing unit 54 executes AI processing for which consent has been obtained, assuming that AI processing for which consent has not been obtained will not be executed based on the consent information.

[0113] It should be noted that the (type of) imaging device and the type of AI processing to be executed are associated and set in advance, and only AI processing for which consent has been obtained is executed from among the AI ​​processing types thus set in advance. Here, images are acquired from surveillance camera 42 and surveillance camera 43, and the AI ​​processing to be executed is "congestion analysis" and "visitor trend analysis" associated with those surveillance cameras (of the type "surveillance camera"), but "face recognition payment" is not executed.

[0114] For example, if a person identified as "Mr. C" (shown in FIG. 13) is present in the captured image being processed, age and gender estimation for the "Visitor Trend Analysis," the only AI processing for which consent is granted, is executed based on Mr. C's consent information. On the other hand, if a person detected in the captured image does not consent to the "Visitor Trend Analysis," age and gender estimation is not executed for that person. For example, if a person identified as "Mr. A" (shown in FIG. 13) is present in the captured image, all executable AI processing is executed because Mr. A's consent information indicates consent for all AI processing. This processing is executed in accordance with the consent conditions stored in the consent condition storage unit 56 and presented to the customer, as shown in FIG. 14. The information processing device 1 may be configured to allow the system administrator to confirm whether the processing content, data to be processed, or purpose described in FIG. 14 is in accordance with the processing content, data to be processed, or purpose.

[0115] In S136, the AI ​​processing result utilization unit 55 performs various processes using the results of the AI ​​processing. For example, the AI ​​processing result utilization unit 55 may convert and process the results of AI processing, such as face detection or age and gender estimation, into a form that can be used for services and output it to another device as needed. For example, when the AI ​​processing result utilization unit 55 performs a "visitor trend analysis," it may output data counting the number of people for each estimated age and gender. Here, the age and gender are estimated only for people who have consented to the "visitor trend analysis" in the captured image, and only those who consent are counted. This count number may be accumulated for a predetermined period while the system is operating, and the number of visitors for each age and gender during that period may be calculated and output to another device as needed. This information is useful for commercial facility operators because it can be used as data to improve the operation of the commercial facility.

[0116] Furthermore, the AI ​​processing result utilization unit 55 may provide a service to customers by displaying advertisements based on the estimated age and gender on a signage terminal near the camera that captured the image. This type of processing is beneficial because it can provide advertisements that are likely to be suitable for the customer. In this way, AI processing and the use of processing results that are beneficial to both the operator of the commercial facility and the customer can be carried out with their consent.

[0117] Furthermore, when the AI ​​processing result utilization unit 55 executes "crowding analysis," it can count the number of people in the store in the same way as when executing "visitor trend analysis," and output the count results to another device. This allows the congestion status to be presented to customers, improving customer convenience. Such congestion status can be presented on a display in the store or via an application on the customer's smartphone, etc.

[0118] In S137, the information processing device 1 determines whether to terminate the AI ​​processing. Here, the information processing device 1 determines to terminate the AI ​​processing if an instruction to terminate has been received from the administrator of the system, and determines not to terminate the AI ​​processing if not. If it is determined in S137 that the AI ​​processing should be terminated, the processing of FIG. 15 ends, and if not, the processing returns to S132.

[0119] This type of processing makes it possible to limit the processing performed on customer data to only those that have been consented to. In particular, since AI processing can be restricted after obtaining the customer's consent to the execution of AI processing for each AI processing service, it becomes possible to perform processing that better respects the individual's will. Therefore, from the customer's perspective, it becomes possible to indicate whether or not to consent to AI processing for each AI processing service, and from the service provider's perspective, it is possible to provide only the AI ​​processing and services that the customer desires, thereby improving convenience.

[0120] It is also possible to offer more dynamic conditions for restricting, allowing, and rewarding AI processing, such as offering customers a time-limited online coupon if they allow a business to use their anonymized in-store purchase history to improve services.

[0121] Furthermore, by presenting the terms of consent to customers in advance, it is possible to explain what information about them will be used in AI processing and what benefits they will receive as a result. This allows customers to choose whether or not to consent to AI processing for each service. This allows customers to choose whether or not to receive a service based on AI processing, taking into account the advantages and disadvantages in accordance with their own wishes, thereby improving convenience.

[0122] [Variation 4-1] In the fourth embodiment, an example was described in which an image from a surveillance camera was acquired and AI processing was performed using the surveillance camera, but each process can be performed in the same way when using an imaging device of a different type than a surveillance camera. Below, we will explain a case in which each process is performed using an image acquired from the facial recognition payment camera 41 instead of the surveillance cameras 42 and 43. Here, it is assumed that "facial recognition payment" is associated with the facial recognition payment camera as the AI ​​process to be executed.

[0123] Here, in S132, an image is acquired from the face recognition payment camera 41, and in the subsequent processing, the AI ​​processing of "face recognition payment" is executed. In the following, the explanation of the processing that overlaps with that explained with reference to FIG. 15 will be omitted.

[0124] In S135, the AI ​​processing unit 54 executes AI processing for the user who has consented to "face authentication payment" based on the consent information. Here, the user is identified using face authentication, but the result of identifying the user performed in S133 may be reused.

[0125] In S136, the AI ​​processing result utilization unit 55 performs payment based on the facial recognition result, referring to payment information such as pre-registered credit card or bank account information, and transmits the payment result to the payment terminal of the commercial facility. Note that prior to this process, there is assumed to be processing such as a store clerk at the commercial facility determining the payment amount or obtaining the customer's intention to use facial recognition payment. Note that for persons who have not consented, payment cannot be made, and therefore information indicating that the payment has failed is output to the payment terminal.

[0126] This type of processing also makes it possible to restrict use based on consent for services that use relatively confidential information, such as payment information. Some customers may feel uneasy about using such information, so by making it possible to restrict use based on consent, it becomes possible to process services that take into consideration the customer's wishes.

[0127] [Variation 4-2] In this embodiment, it has been described that all consent information of persons not in the registered dictionary is processed as if they have not consented, but this processing is not particularly limited to this. For example, the information processing device 1 may perform each process assuming that persons not in the registered dictionary have consented to all AI processing. In this case, however, it is desirable to notify customers by displaying a message such as "If a customer enters a commercial facility according to operating regulations, they will be considered to have consented to AI processing" so that customers do not feel uncomfortable that AI processing has been performed without their consent.

[0128] [Variation 4-3] In this embodiment, biometric information, i.e., an image of a face, is used as the data input to AI processing, but it does not have to be biometric information. For example, personal information such as an individual's name, address, email address, telephone number, or attribute information (race, gender, age, occupation) may be registered in advance in a registration dictionary as text or numerical values ​​and used for AI processing. This personal information may be an ID number assigned a unique number to each individual. In this case, an example of AI processing and services may include behavioral and purchasing predictions using textual personal information. Even if the input data is text or numerical values, because it is personal information, restricting its use with the individual's consent can provide a service that is highly satisfactory to customers.

[0129] Furthermore, in the present embodiment, biometric information is used to identify the individual, but biometric information does not necessarily have to be used as long as the individual can be identified. For example, the individual may be identified by information such as a membership number or a user name, and consent information in a registered dictionary may be referenced. Furthermore, for example, the individual may be identified by an RFID tag or device-to-device communication with a smartphone carried by the user.

[0130] [Variation 4-4] In this embodiment, information such as the content of AI processing, the data to be used, or the purpose is presented as a condition for consent, but the information presented here is not limited to this. For example, as described above, the name of the organization or company that manages the AI ​​processing and the data to be used may be presented as part of the condition for consent. Such processing allows the customer to decide whether to consent by taking into account whether the organization or company is trustworthy. In this case, organizations and companies may be assigned a score in advance indicating their trustworthiness, and this score may be presented to the customer.

[0131] Information about the location of use may also be presented as a condition of consent. For example, information about whether the user's information will be used only at that commercial facility or at commercial facilities in other locations may be presented as part of the condition of consent. Furthermore, the screen presenting the condition of consent may allow the customer to select the location where their information will be used. This type of processing makes it possible to configure the system so that customers can collectively indicate their consent for all commercial facilities operated by the same operator.

[0132] [Variation 4-5] In this embodiment, an example has been described in which the information processing device 1 is used for AI processing and services in a commercial facility as shown in Fig. 14, but the location, AI processing, and services are not limited to those described here. For example, when using biometric information as data for improving the operation of the commercial facility, a process of restricting the use of biometric information in the information processing device 1 may be performed. In this case, the information processing device 1 according to this embodiment can be used, for example, when performing AI processing to constantly detect and track customers and visualize their movement paths within the commercial facility.

[0133] Furthermore, by using facial recognition to manage entry to specific locations or rooms, it becomes possible to eliminate the need for membership cards or keys. From this perspective, the information processing device 1 may be used in commercial facilities such as sports gyms, schools, hospitals, and other facilities. When the information processing device 1 is used in a school or hospital, in addition to the AI ​​processing for entry management described above, it is expected that the information processing device 1 will be used for AI processing and services such as operating an AI to detect suspicious individuals for security services, or operating an AI to detect abnormal behavior such as falls for monitoring services. Even in such cases, by obtaining the user's consent for each AI processing and service, it becomes possible to provide services that reflect the customer's wishes.

[0134] However, some services that are of public interest or highly urgent nature, such as crime prevention services, monitoring services, or guidance services based on an analysis of the number or location of passersby during a disaster, may be provided without consent. In such cases, a separate situation notification module (not shown) may be provided to control the services provided by the information processing device 1 based on the notification results so that the services will operate only under appropriate conditions in accordance with current or future laws and regulations. This configuration allows customers to always receive urgent services without having to wait for consent each time, thereby improving convenience.

[0135] For example, in a group setting such as a school, if a service is provided in which automatic photographing of faces using face detection is performed and the photographs are then classified by person and sold, consent may be obtained for both automatic photographing using face detection and for person identification processing to classify the photographs. This type of processing allows AI processing to be restricted according to individual wishes. In particular, it becomes possible to perform processing that respects the detailed wishes of individuals, such as allowing face detection alone but not allowing classification to identify individuals.

[0136] Although the present embodiment has been described assuming that consent information is obtained from the user, the consent information does not have to be entered by the user himself / herself, as long as it indicates whether or not the user has given consent. For example, a parent or guardian of a user (especially a child) may submit consent information on behalf of the user. Such processing can accommodate cases where, for example, when a child gets lost in a commercial facility, the parent or guardian enters consent information on the child's behalf when AI processing (face detection and face recognition) is performed on surveillance camera images. Furthermore, the present embodiment has been described assuming that consent is obtained upon entering a commercial facility. However, it is also possible for a parent or guardian to access the system and enter consent to AI processing to find the lost child when the child becomes lost. In this way, the person entering the consent information does not have to be the user himself / herself; the consent information may be entered by a person deemed to have the right to enter consent. Furthermore, the time or place for consent to be entered is not limited and may be any time before AI processing is performed. For example, the consent information may be entered when a service is needed.

[0137] [Variation 4-6] In this embodiment, consent information for the use of AI processing is acquired for each commercial facility. However, for example, by sharing consent information between facilities, it is possible to eliminate the need for users to enter consent information one by one at each facility. Furthermore, for example, similar to setting security items for each trust level in Internet web browsing, multiple trust levels and corresponding consent conditions may be associated in advance, and processing may be performed based on these settings. In this case, facility users can set consent simply by specifying which trust level the facility they are using corresponds to. Furthermore, (default) consent conditions may be set in the absence of user input. Furthermore, in this embodiment, consent information is described as being entered in advance, but it may also be possible to edit it, for example, while using the facility. This configuration allows users to change the simultaneous conditions at a certain timing, improving convenience.

[0138] [Other embodiments] In the above-described first to fourth embodiments, an example of acquiring a face image as biometric information has been described. However, the biometric information is not limited to an image including a face, as long as it includes information about the user. For example, the biometric information may be an image including information that can identify the user, such as an individual's iris, fingerprint, or veins. Even when such biometric information is used, it is possible to similarly extract features from the biometric information. Note that the device for acquiring the captured image may be an imaging device including an appropriate sensor corresponding to each type of biometric information.

[0139] The disclosure of this specification includes the following information processing device, information processing method, and program. (Item 1) a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; extraction means for extracting the feature amount from the biometric information of the person; a restriction means for restricting at least one of extraction and use of the feature amount based on the consent information acquired by the consent information acquisition means; An information processing device comprising: (Item 2) 2. The information processing device according to item 1, wherein the restriction means controls the extraction means not to extract the feature when the consent information indicates that the person does not consent to the use of the feature. (Item 3) a registration means for registering the feature extracted by the extraction means and the consent information acquired by the consent information acquisition means in association with each other, 3. The information processing device according to item 1 or 2, wherein the restriction means controls the registration means not to register the feature when the consent information indicates that the person does not consent to the use of the feature. (Item 4) Item 3. The information processing device according to item 3, wherein the registration means, if the consent information indicates that the person does not agree to the use of the features, associates the person's biometric information with the consent information and registers them. (Item 5) a receiving means for receiving an instruction to change the consent information, 5. The information processing device according to item 4, wherein the extraction means extracts features from registered biometric information of a person corresponding to the consent information in response to the reception means receiving a change instruction to change the consent state indicated by the consent information from non-consent to consent. (Item 6) Item 6. The information processing device according to item 5, wherein the registration means associates and registers the biometric information acquired by the biometric information acquisition means, the consent information acquired by the consent information acquisition means, and the feature extracted by the extraction means. (Item 7) a matching means for performing a matching process using the feature extracted by the extraction means and a feature registered in advance, 7. The information processing device according to any one of items 1 to 6, wherein the restriction means restricts use of the matching result by the matching means in accordance with the consent information acquired by the consent information acquisition means. (Item 8) 8. The information processing device according to item 7, wherein the restriction means controls so that the results of the matching process relating to a person for whom the consent information does not indicate consent are not displayed. (Item 9) An imaging means; an imaging control means for controlling the imaging means based on a collation result by the collation means, 8. The information processing device according to item 7, wherein the restriction means controls the imaging control means not to control the imaging means for a person for whom the consent information does not indicate consent. (Item 10) 10. The information processing device according to item 9, wherein the control performed by the imaging control means includes at least one of focus control and exposure control. (Item 11) a first recording means for recording the person information obtained by the matching process by the matching means and the image captured by the imaging means in association with each other; 11. The information processing device according to item 9 or 10, wherein the restriction means controls so that personal information relating to a person for whom the consent information does not indicate consent is not recorded in association with the image. (Item 12) a second recording means for recording the image captured by the imaging means; The restriction means controls so that an image showing a person for whom the consent information does not indicate consent is not recorded by the second recording means. 10. The information processing device according to item 9, (Item 13) a transmitting means for transmitting the person information obtained by the matching process by the matching means to another device via a communication network, 8. The information processing device according to item 7, wherein the restriction means controls so that the transmission means does not transmit personal information of a person for whom the consent information does not indicate consent. (Item 14) 14. The information processing device according to any one of items 1 to 13, wherein the biometric information is a captured image of the person's face, iris, fingerprint, or veins. (Item 15) a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating the consent status of the person regarding AI processing of the biometric information; A processing means for performing AI processing on the biometric information of the person; a restriction means for restricting the AI ​​processing by the processing means based on the consent information acquired by the consent information acquisition means; An information processing device comprising: (Item 16) a presentation means for presenting consent conditions, which are conditions for consent by the person regarding the use of the biometric information in the AI ​​processing; Item 16. The information processing device according to item 15, wherein the consent information is information set based on the consent conditions. (Item 17) Item 17. The information processing device according to item 16, wherein the consent conditions include any one of the content of the AI ​​processing, the data used by the AI ​​processing, the period during which the AI ​​processing is performed, information about the organization that performs the AI ​​processing, and information about the organization that manages the data used by the AI ​​processing. (Item 18) Item 18. The information processing device according to item 17, characterized in that the information used in the AI ​​processing includes the person's name, address, email address, telephone number, ID number, or attribute information. (Item 19) An information processing method executed by an information processing device, a biometric information acquisition step of acquiring biometric information of a person; a consent information acquisition step of acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; an extraction step of extracting the feature amount from biometric information of the person; a limiting step of limiting at least one of extraction and use of the feature amount based on the consent information acquired in the consent information acquiring step; An information processing method comprising: (Item 20) An information processing method performed by an information processing device, a biometric information acquisition step of acquiring biometric information of a person; a consent information acquisition step of acquiring consent information indicating a consent status of the person regarding AI processing of the biometric information; a processing step of performing AI processing on the biometric information of the person; a limiting step of limiting the AI ​​processing performed by the processing step based on the consent information acquired by the consent information acquisition step; An information processing method comprising: (Item 21) A program for causing a computer to function as each means of the information processing device described in any one of items 1 to 18.

[0140] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0141] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0142] 1: information processing device, 14: imaging device

Claims

1. a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; extraction means for extracting the feature amount from the biometric information of the person; a restriction means for restricting at least one of extraction and use of the feature amount based on the consent information acquired by the consent information acquisition means; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the restriction means controls the extraction means not to extract the feature when the consent information indicates that the person does not consent to the use of the feature.

3. a registration means for registering the feature extracted by the extraction means and the consent information acquired by the consent information acquisition means in association with each other, 2. The information processing device according to claim 1, wherein the restriction means controls the registration means not to register the feature when the consent information indicates that the person does not consent to the use of the feature.

4. 4. The information processing device according to claim 3, wherein, when the consent information indicates that the person has not consented to the use of the feature, the registration means associates the person's biometric information with the consent information and registers them.

5. a receiving means for receiving an instruction to change the consent information, The information processing device according to claim 4, characterized in that the extraction means extracts features from registered biometric information of the person corresponding to the consent information in response to the reception means receiving a change instruction to change the consent state indicated by the consent information from non-consent to consent.

6. 6. The information processing device according to claim 5, wherein the registration means associates and registers the biometric information acquired by the biometric information acquisition means, the consent information acquired by the consent information acquisition means, and the feature extracted by the extraction means.

7. a matching means for performing a matching process using the feature extracted by the extraction means and a feature registered in advance, 2. The information processing apparatus according to claim 1, wherein the restriction means restricts use of the result of the matching by the matching means in accordance with the consent information acquired by the consent information acquisition means.

8. 8. The information processing apparatus according to claim 7, wherein the restriction means controls so that the results of the matching process relating to a person whose consent information does not indicate consent are not displayed.

9. An imaging means; an imaging control means for controlling the imaging means based on a collation result by the collation means, 8. The information processing apparatus according to claim 7, wherein the restriction means controls the image capture control means not to control the image capture means for a person for whom the consent information does not indicate consent.

10. 10. The information processing apparatus according to claim 9, wherein the control performed by said imaging control means includes at least one of focus control and exposure control.

11. a first recording means for recording the person information obtained by the matching process by the matching means and the image captured by the imaging means in association with each other; 10. The information processing apparatus according to claim 9, wherein the restriction means controls so that personal information relating to a person for whom the consent information does not indicate consent is not recorded in association with the image.

12. a second recording means for recording the image captured by the imaging means; The restriction means controls so that an image showing a person for whom the consent information does not indicate consent is not recorded by the second recording means.

10. The information processing apparatus according to claim 9,

13. a transmitting means for transmitting the person information obtained by the matching process by the matching means to another device via a communication network, 8. The information processing apparatus according to claim 7, wherein the restriction means controls so that the personal information of a person for whom the consent information does not indicate consent is not transmitted by the transmission means.

14. The information processing apparatus according to claim 1 , wherein the biometric information is a captured image of the person's face, iris, fingerprint, or veins.

15. a biometric information acquisition means for acquiring biometric information of a person; consent information acquisition means for acquiring consent information indicating the consent status of the person regarding AI processing of the biometric information; A processing means for performing AI processing on the biometric information of the person; A restriction means for restricting the AI ​​processing by the processing means based on the consent information acquired by the consent information acquisition means; 2. An information processing device further comprising:

16. a presentation means for presenting consent conditions, which are conditions for consent by the person regarding the use of the biometric information in the AI ​​processing; 16. The information processing apparatus according to claim 15, wherein the consent information is information set based on the consent conditions.

17. 17. The information processing device according to claim 16, wherein the consent conditions include any one of the content of the AI ​​processing, the data used by the AI ​​processing, the period during which the AI ​​processing is performed, information about an organization that performs the AI ​​processing, and information about an organization that manages the data used by the AI ​​processing.

18. 18. The information processing device according to claim 17, wherein the information used in the AI ​​processing includes any one of the person's name, address, email address, telephone number, ID number, and attribute information.

19. An information processing method executed by an information processing device, a biometric information acquisition step of acquiring biometric information of a person; a consent information acquisition step of acquiring consent information indicating a consent state of the person regarding use of the feature extracted from the biometric information; an extraction step of extracting the feature amount from biometric information of the person; a limiting step of limiting at least one of extraction and use of the feature amount based on the consent information acquired in the consent information acquiring step; An information processing method comprising:

20. An information processing method performed by an information processing device, a biometric information acquisition step of acquiring biometric information of a person; a consent information acquisition step of acquiring consent information indicating the consent status of the person regarding AI processing of the biometric information; A processing step of performing AI processing on the biometric information of the person; a limiting step of limiting the AI ​​processing by the processing step based on the consent information acquired by the consent information acquisition step; An information processing method comprising:

21. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 18.

Images