Information processing device, information processing method, and program

Abstract

To more effectively protect privacy of a person himself with respect to a feature amount obtained from biological information. Biological information of a person is acquired. Consent information indicating a consent state of the person related to use of a feature amount extracted from the biological information is acquired. The feature amount is extracted from the biological information of the person. Extraction and / or use of the feature amount is restricted on the basis of the acquired consent information.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] AI processing is known, which performs processes such as detection or estimation using parameters learned from data such as images. AI processing is capable of intelligent processing and judgment like humans, and has been increasingly applied in various fields in recent years. However, there are also calls for restrictions on the unauthorized use of personal information (e.g., facial images, fingerprints, irises, etc.) in AI, and certain considerations are required when using AI processing. In particular, biometric authentication, which is one type of AI processing, may violate personal privacy depending on the application, so further restrictions on its use are required. Patent Document 1 discloses a method for obtaining consent from an individual before registering a subject for biometric authentication.

[0003] Japanese Patent Application Laid-Open No. 2022-119549

[0004] Deng, Jiankang, et al. “Retinaface: Single-shot multi-level face localization in the wild.” IEEE / CVF conference on computer vision and pattern recognition. 2020.

[0005] Although Patent Document 1 describes obtaining prior consent for the acquisition of facial images, there is a risk that extraction of features from biometric information such as facial images and use of those features may be carried out without the person's consent.

[0006] An object of the present invention is to more effectively protect the privacy of an individual with respect to feature quantities obtained from biometric information.

[0007] To achieve the object of the present invention, for example, an information processing device according to one embodiment includes the following configuration: biometric information acquisition means for acquiring biometric information of a person, consent information acquisition means for acquiring consent information indicating a consent state of the person regarding use of feature amounts extracted from the biometric information, extraction means for extracting the feature amounts from the biometric information of the person, and restriction means for restricting at least one of the extraction and use of the feature amounts based on the consent information acquired by the consent information acquisition means.

[0008] This makes it possible to more effectively protect the privacy of individuals regarding features obtained from biometric information.

[0009] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0010] The accompanying drawings are included in the specification and constitute a part thereof, illustrate embodiments of the present invention, and are used together with the description to explain the principles of the present invention. A diagram for explaining a situation in which an information processing device according to an embodiment is used. A block diagram showing an example of a hardware configuration of an information processing device according to an embodiment. A block diagram showing an example of a functional configuration of an information processing device according to an embodiment. A flowchart showing an example of registration processing by an information processing device according to an embodiment. A diagram for explaining a registered dictionary in an information processing device according to an embodiment. A flowchart showing an example of authentication processing by an information processing device according to an embodiment. A diagram showing an example of an output image by an information processing device according to an embodiment. A diagram showing an example of a system configuration according to an embodiment. A flowchart showing an example of output processing by an information processing device according to an embodiment. A diagram for explaining a comparison of registered dictionaries between devices according to an embodiment. A diagram showing an example of a system configuration according to an embodiment. A block diagram showing an example of a functional configuration of an information processing device according to an embodiment. A diagram for explaining a registered dictionary in an information processing device according to an embodiment. A diagram for explaining consent conditions by an information processing device according to an embodiment. A flowchart showing an example of AI processing by an information processing device according to an embodiment.

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] [First Embodiment] [Overview] An information processing device according to this embodiment acquires biometric information of a person and acquires consent information indicating consent information from the person regarding the use of feature amounts extracted from the biometric information. The information processing device then extracts feature amounts from the person's biometric information and, based on the acquired consent information, restricts at least one of the extraction and use of the feature amounts.

[0013] The following describes a method for restricting the use of biometric information in a biometric authentication registration process for an image of a person captured by a digital camera. In this embodiment, the digital camera is described as having an information processing device and an image capture device, but the present invention is not limited to this. For example, the digital camera having the image capture device and the information processing device may be connected via a network.

[0014] 1A and 1B are diagrams for explaining situations in which an information processing apparatus according to this embodiment is used, Fig. 1A shows a situation in which a person is photographed using a digital camera.

[0015] The information processing device 1 executes processing by each functional unit shown in FIG. 3, which will be described later. The information processing device 1 according to this embodiment is part of a digital camera and controls the overall processing of the digital camera. In the following description, it is assumed that an imaging device 14, which is part of the digital camera, operates in response to instructions from the information processing device 1. Subject 2 is a first subject, a person in this case, whose image is captured by the imaging device 14 in the situation shown in FIG. 1A. Subject 3 is a second subject, a person in this case, whose image is captured by the imaging device 14 in the situation shown in FIG. 1A. In the following description, a person who is a subject in an image may be referred to as an "individual."

[0016] The button 11 is a shutter button that a user (not shown) of the imaging device 14 presses to instruct the information processing device 1 to take a photograph. The panel 12 is a display panel that displays the results of processing to the user of the imaging device 14. Here, the panel 12 displays an image to allow the user to check the composition or the state of the subject when taking a photograph. The panel 12 may be of any type as long as it can display an image, and may be, for example, a liquid crystal panel, an organic EL (Electro Luminescence) panel, or a projector.

[0017] The operation keys 13 are used to obtain input from a user. The type of the operation keys 13 is not particularly limited as long as the operation keys 13 can obtain input from a user. For example, the operation keys 13 may be buttons provided on the imaging device 14, a keyboard and a mouse connected to the information processing device 1, or the panel 12, which is a touch panel, may also serve as the operation keys 13.

[0018] 1B shows a digital camera having an imaging device 14 and the information processing device 1, viewed from the opposite side to the orientation shown in FIG. 1A. The imaging device 14 is an imaging unit consisting of a lens and a sensor, and captures an image in response to a request from the information processing device 1.

[0019] 2 is a block diagram showing an example of the hardware configuration of the information processing device 1. The information processing device 1 includes a CPU 21, a ROM 22, a RAM 23, an external memory 24, an input unit 25, a display unit 26, a communication I / F 27, an I / O 28, and a communication bus 29.

[0020] The CPU 21 is a central processing unit that controls various devices connected to the system bus 29 and executes a program that implements the processing of the present invention. The ROM 22 stores a BIOS program and a boot program. The RAM 23 is a memory used as the main storage device of the CPU 21. The external memory 24 stores various data such as programs processed by the information processing device 1 and captured images.

[0021] The input unit 25 acquires input from a user via the buttons 11 or operation keys 13 mounted on the information processing device 1. The display unit 26 outputs the calculation results of the information processing device 1 to the display panel 12 in accordance with instructions from the CPU 21. The communication I / O 27 is a communication interface and performs information communication with the outside. The communication I / O 27 may perform communication via wired communication using a USB or the like, or via wireless communication such as a local area network or serial communication, and the type of communication is not limited. The I / O 28 inputs data from the imaging device 14.

[0022] [Explanation of the Configuration Diagram] FIG. 3 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. The biometric information acquisition unit 31 acquires biometric information. Here, the biometric information acquisition unit 31 acquires a captured image showing a subject as biometric information. The consent information management unit 32 acquires and manages consent information indicating whether or not a person has given consent to the use of the acquired biometric information. Here, the consent information management unit 32 can acquire and manage consent information based on input via the operation keys 13. Note that, hereinafter, the term "consent" simply refers to consent to the use of the biometric information as described above. Furthermore, here, biometric information will be described as referring to the entire captured image showing the subject. However, for example, a rectangular area (bounding box) in which the subject is detected, or a rectangular area in which the subject's face is detected, etc. may also be used as biometric information, and is not particularly limited as long as it is information within an image that includes information about the subject.

[0023] The feature extraction unit 33 extracts feature amounts from biometric information. Here, the feature extraction unit can extract feature amounts from a captured image, which is biometric information, for a person who has agreed to the use of feature amounts based on consent information.

[0024] The output unit 34 outputs (transmits) the consent information, biometric information, and feature amounts to an external device. The registration unit 35 registers the biometric information and consent information. Specifically, the registration unit 35 associates the biometric information, feature amounts extracted from the biometric information, and consent information and stores them in the external memory 24. The registration unit 35 can also associate a person's name obtained by input via the operation keys 13 with the feature amounts and consent information and record them in the external memory 24. Note that instead of outputting the consent information, biometric information, and feature amounts to an external device, the output unit 34 may output the consent information or the biometric information and feature amounts to an external device.

[0025] The matching unit 36 ​​compares the input biometric information with the biometric information registered by the registration unit 35 to determine who the person is or is not among the registered biometric information. The matching result utilization unit 37 includes a control unit 38, a display unit 39, and a recording unit 40, and performs various processes using the matching results determined by the matching unit 36. The control unit 38 controls the imaging device 14 based on the matching results. The display unit 39 displays the matching results on the display panel 12. The recording unit 40 records the matching results in the external memory 24.

[0026] Each process according to this embodiment will be described below with reference to a flowchart, but the process procedure is not limited to that shown. For example, the order of processes may be changed as long as the same processing results are obtained, multiple processes may be integrated, a process described as a single step may be subdivided, or some processes may be omitted. Furthermore, each process may be individually extracted and function independently as a single functional element, and may be used in combination with processes other than those shown.

[0027] [Description of Data Structure] The consent information according to this embodiment is information indicating whether or not there is consent regarding the use of features, as described above, and here, consent / non-consent is expressed as a Boolean value. The initial value of the consent information is non-consent.

[0028] The registration dictionary according to this embodiment is a database in which the user of the information processing device 1 registers information about people he or she wishes to match. Here, the registration dictionary stores, for each person, the person's name (character string), consent information, facial image, and feature data in association with each other. Note that multiple pieces of facial image and feature data may be stored for each person.

[0029] [Explanation of Flowchart of Processing Procedure] The biometric authentication process performed by the information processing device 1 according to this embodiment is divided into a registration process for registering biometric information of a person to be authenticated, and an authentication process for determining which of the registered people the input biometric information corresponds to. Furthermore, the information processing device 1 according to this embodiment restricts the use of biometric information of a certain person in the registration process based on the person's consent information acquired based on the biometric information. Hereinafter, the process for restricting the use of biometric information in such registration process will be described with reference to FIG. 4 .

[0030] Fig. 4 is a flowchart showing an example of registration processing by the information processing device 1. The processing described in Fig. 4 starts when, for example, a user of the information processing device 1 instructs execution of the registration processing (for example, by pressing the operation button 13 on the display panel 12 shown in Fig. 1). The user in this embodiment is assumed to be a photographer who photographs a subject using a digital camera including the information processing device 1.

[0031] In S101, the information processing device 1 performs initialization processing for the registration processing. Here, the CPU 21 of the information processing device 1 reads a program from the external memory 24 of FIG. 2 and makes the registration processing, which will be described later, operable. As part of the initialization processing, the CPU 21 of the information processing device 1 also loads a registration dictionary from the external memory 24 and makes it available for addition. The processing described below in FIG. 4 is realized by the CPU 21 of the information processing device 1 executing the necessary programs.

[0032] In S102, the biometric information acquisition unit 31 acquires biometric information (here, an image of an individual's face captured by the imaging device 14 in FIG. 1B). At this time, the biometric information acquisition unit 31 controls the imaging device 14 using known imaging technology so that the focus and exposure are suited to the target's face, and acquires the captured image. Note that the following description will be given assuming that the captured image that becomes the biometric information contains one subject, but the captured image may contain multiple subjects, and the processing described below may be performed individually on each of them.

[0033] In S103, the consent information management unit 32 acquires and manages consent information for individuals (subjects) appearing in the captured images acquired in S102. As described above, the consent information acquired here indicates whether or not there is consent to the use of images showing the individual's face, i.e., their biometric information.

[0034] The "use of biometric information" according to this embodiment includes at least one of photographing and recording a face image, extracting features from the face image, performing a registration process to register the features, performing an authentication process based on the features (here, a process to identify a person), and performing some kind of control using the results of the authentication process. Details of these usage methods will be described later. The information processing device 1 according to this embodiment restricts the use of this biometric information based on consent information.

[0035] The consent information management unit 32 according to this embodiment can acquire consent information based on, for example, a user's operation via the display panel 12 or the operation keys 13. Specifically, the consent information management unit 32 can display, on the display panel 12, a message indicating that biometric information will be used and a message for accepting a selection as to whether or not to agree to the use, and acquire the result of the selection as consent information.

[0036] It should be noted that the user name is assumed to be set in advance (for example, based on the user's login or the user's input during operation). However, in order to prevent so-called impersonation, in which a person other than the user arbitrarily agrees to an operation, the user may be set based on a captured image different from the biometric information. For example, an imaging device (not shown) may be further provided to capture an image of the operator of the operation keys 13, and the user may be set based on the image captured by such an imaging device (by a known person recognition process). In such a case, an additional determination may be made as to whether the set user and the person in the biometric information acquired in S102 are the same person. If they are not the same person, the process of FIG. 4 may be terminated at that point.

[0037] Furthermore, although the explanation has been given assuming that consent information is acquired based on user input via the operation keys 13, the process is not limited to this, as long as it is possible to acquire whether or not the user has consented. For example, if it is detected that the user has performed a predetermined action (e.g., a gesture indicating an instruction by voice, etc.), it may be determined that the user has consented to the use of their biometric information, and image capture by the imaging device 14 and processing for using the biometric information (e.g., authentication processing) may be performed. This process allows the user to indicate their intention to consent simply by making a gesture, thereby improving convenience. The gesture may be, for example, a peace sign using the hand, or a gesture of raising or lowering the hand, and the gesture may be recognized using known motion recognition technology.

[0038] In S104, the feature extraction unit 33 determines whether or not the user has consented to the use of their biometric information based on the consent information acquired in S103. If the user has consented, the process proceeds to S105, and if not, the process proceeds to S107.

[0039] In S105, the feature extraction unit 33 extracts features from the biometric information of a user who has consented to the use of their biometric information. In this embodiment, the biometric information is a captured image of an individual's face, and the feature extraction unit 33 identifies the position of the face in the captured image and then extracts features from the face. Any known image processing technology can be used in the process of identifying the position of the face in the captured image and the process of extracting features from the person's face. For example, the technology described in Non-Patent Document 1 may be used to identify the position of the face in the captured image. Furthermore, for example, a method using Deep Learning may be used to extract features from the person's face, or a method such as Local Binary Pattern (LBP) or Histogram Oriented Gradient (HoG) may be used.

[0040] In S106, the registration unit 35 associates the extracted feature with the biometric information, consent information, and personal name, and registers them in a registration dictionary. The registration unit 35 can acquire the personal name by accepting user input from an input unit (not shown) via the operation keys 13 in FIG. 1A. The format of the personal name according to this embodiment is not particularly limited as long as it is set so as to enable distinction between registered persons. For example, the personal name may be a real name, a nickname, a number, a symbol, or the like. Furthermore, although this embodiment will be described assuming that the registration unit 35 registers the consent information, the processing may be performed by a different functional unit, such as the consent information management unit 32 registering the consent information.

[0041] In S107, the registration unit 35 records the biometric information, consent information, and person's name in association with each other. Here, feature extraction is not performed for individuals who have not given consent, and only biometric information, i.e., a face image, is registered for the consent information and person's name.

[0042] In S108, the registration unit 35 determines whether to end the registration process. Here, for example, if the user has input to end the registration process, it may be determined that the registration process is to be ended. If it is determined that the registration process is to be ended, the process in FIG. 4 ends; if not, the process returns to S102.

[0043] FIG. 5 is a diagram showing an example of the contents of a registration dictionary after a registration process. In FIG. 5 , each column displays a "No." indicating an index, a "Person's Name" character string indicating a person's name, "Consent Information" that displays a check mark if consent to the use of biometric information is indicated in the consent information, a "Facial Image" displaying an image of the subject, and a "Feature Amount" that schematically displays the extracted and recorded feature amount. Each row and column represents one person, and three people, "Mr. A," "Mr. B," and "Mr. C," are registered, respectively, in Nos. 1 to 3. In FIG. 5 , consent to the use of biometric information has been obtained for "Mr. A" and "Mr. C," and feature amounts have been extracted and registered. Consent has not been obtained for "Mr. B," and feature amounts have not been registered. This registration dictionary is used in the authentication process described below.

[0044] [Effects] This process makes it possible to restrict the use of biometric information for persons who have not given their consent. In particular, by preventing feature registration, it is possible to prevent individuals from being identified through the authentication process described below (i.e., to restrict face authentication processing). Furthermore, by registering only an image when consent has not been given, if consent is obtained later, it becomes possible to extract and register features without obtaining new biometric information, improving convenience (since it is no longer necessary to obtain an image and consent at the same time, convenience is improved for both the user and the person being registered).

[0045] [Variation 1-1] [Variations of the Consent Acquisition Method] In this embodiment, the subject's consent is acquired by displaying on the display panel 12 of the information processing device 1 a message indicating that biometric information will be used and a message (an image prompting consent) prompting the subject to choose whether or not to consent. However, the method of acquiring consent information is not particularly limited to this, as long as it is possible to confirm whether or not the user has consented. For example, the information processing device 1 may acquire consent information based on input from an application on a mobile device such as a smartphone (not shown). In such a case, the information processing device 1 may communicate with the mobile device and request the mobile device to transmit consent information when acquiring consent information (S103 in FIG. 4). Upon receiving the request, the mobile device may display an image prompting consent on its screen and accept input indicating consent from the user of the mobile device, thereby acquiring consent information and transmitting it to the information processing device 1. At this time, to detect impersonation and prevent consent by a different person, the information processing device 1 may confirm that the person being registered in the registration process and the user of the mobile device are the same person (perform identity authentication). To this end, the information processing device 1 may perform identity authentication on its own side, or may perform identity authentication using a security function of the mobile terminal. For example, the information processing device 1 may transmit a facial image of a person to be registered to the mobile terminal, and the mobile terminal may use a known facial authentication method to verify whether the user and the facial image are the same person. The facial image of the user of the mobile terminal is captured, for example, by the mobile terminal's internal camera. With this configuration, consent obtained through impersonation can be avoided and consent information can be obtained from the subject. Note that identity authentication is not limited to such facial authentication, and may be any authentication process using, for example, fingerprint authentication provided in the mobile terminal, or authentication using an electronic certificate.

[0046] [Variation 1-2] [Processing When Image Is Not Registered] In the present embodiment, a facial image is registered even when consent is not provided. However, it is also possible to prevent registration of a facial image when consent is not provided. According to such processing, by not storing a facial image when consent is not provided, it is possible to perform processing that takes user privacy into greater consideration (respects individual wishes more). Note that, compared to consent information, facial images are often relatively easy to obtain through imaging processing or acquisition processing via SNS (Social Networking Service), etc. Therefore, when priority is given to light processing speed and ease of operation in the registration processing, facial images may be registered regardless of whether consent is provided, as described in S107.

[0047] [Variation 1-3] [Images and personal names are not required in the registration dictionary] In the first embodiment, it has been described that information such as facial images and personal names are linked to consent information and registered in the registration dictionary. However, the information registered in the registration dictionary for consent information is not limited to information necessary for matching, and personal names are not required. For example, in the registration dictionary, only facial images may be linked to consent information and registered, or feature amounts may be linked to consent information and registered, or both may be linked to consent information and registered.

[0048] [Embodiment 2] The information processing device 1 according to embodiment 1 restricts the use of biometric information in the above-described registration process based on the presence or absence of consent. The information processing device 1 according to embodiment 2 acquires consent information in the same manner as embodiment 1, and restricts the use of biometric information in the authentication process based on the presence or absence of consent. The information processing device 1 according to this embodiment has the same hardware configuration and functional configuration as the information processing device 1 according to embodiment 1, and can execute the same process, so duplicated explanations will be omitted.

[0049] 6 is a flowchart showing an example of authentication processing performed by the information processing device 1 according to this embodiment. For example, the authentication processing is started when a user of the information processing device 1 instructs the execution of the authentication processing (for example, by pressing the operation button 13 on the display panel 12 shown in FIG. 1A). Note that the authentication processing according to this embodiment is performed after the registration processing described in the first embodiment is executed.

[0050] In S111, the information processing device 1 performs initialization. The initialization process is performed for the registration process. Here, the information processing device 1 reads a program from the external memory 24 of FIG. 2 and makes the authentication process, which will be described later, operable. As part of the initialization process, the information processing device 1 also loads a registration dictionary from the external memory 24 and makes it available for use.

[0051] In S112, the biometric information acquisition unit 31 acquires biometric information (here, an image of an individual's face from the imaging device 14 in FIG. 1B). Here, the biometric information acquisition unit 31 acquires, as biometric information, an image of a candidate to be recorded by the digital camera from the imaging device 14. The captured image acquired here is displayed on the display panel 12 as a live view in the processing described below. The user (photographer) can determine the composition of the image to be recorded or the timing of pressing the shutter button 11 while checking the live view. Note that the captured image acquired here may include multiple faces.

[0052] In S113, the feature extraction unit 33 extracts features from the biometric information. Here, as described in the registration process, the feature extraction unit 33 identifies the position of the face in the captured image and then extracts features from the face of the user who has agreed to the use of their biometric information. If the image includes multiple faces, features are extracted from each of the multiple faces.

[0053] In S114, the matching unit 36 ​​identifies a person by comparing the feature values ​​of the individual registered in the registration dictionary with the feature values ​​extracted in S113. Here, the matching unit 36 ​​compares the feature values ​​extracted in S113 with the biometric information registered by the registration unit 35 to determine whether the extracted feature value matches or does not match any of the registered biometric information. For example, the matching unit 36 ​​calculates the similarity between the extracted feature value and the registered feature value. If the similarity value exceeds a predetermined threshold among the registered individuals, the matching unit 36 ​​determines that the extracted person is the individual. If no individual exceeds the threshold, the matching unit 36 ​​determines that the extracted person is not a person. Any known method can be used to calculate the similarity. For example, cosine similarity or L2 distance may be used as the similarity. Since the L2 distance is a measure in which the value decreases as the distance between feature values ​​decreases, the reciprocal of the L2 distance may be converted into the similarity value. Here, as described in the description of FIG. 5 regarding the registration process, since no feature values ​​for "Mr. B" are registered, matching for Mr. B cannot be performed. In addition, as for Mr. B, since he has not consented to the use of his biometric information, no feature values ​​are extracted.

[0054] In the following steps S115 to S117, the matching result utilization unit 37 performs various processes using the matching result output by the matching unit 36. These processes will be described in detail below. In step S115, the display unit 39 displays the matching result on the display panel 12. FIG. 7 is a schematic diagram showing an example of the matching result displayed by the display unit 39. In FIG. 7, a face frame G101 of a person displaying "Unknown" and a face frame G102 of a person displaying "Mr. C" and a face frame G103 displaying "Unknown" are arranged on the display panel 12.

[0055] Here, G101 is "Mr. B," but as mentioned above, "Mr. B" has not consented to the use of his / her biometric information, so the matching process was not performed. Therefore, the result is determined to be "Unknown," meaning that the person is not a member of the registered dictionary. Also, here, G102 is "Mr. C," and as mentioned above, "Mr. C" has consented to the use of his / her biometric information, so the result is determined to be "Mr. C" by matching the feature values ​​of "Mr. C" in the registered dictionary. G103 indicates that the person was not matched with anyone registered in the registered dictionary and was determined to be "Unknown." In other words, in this example, only the name of "Mr. C," who consented, is displayed, while the matching results for other people are displayed as "Unknown," with no indication of their identity. In this way, the use of biometric authentication can be restricted by preventing matching of people without consent. The display shown in FIG. 7 is a live view display on a digital camera, and the user can determine whether to save the image by pressing the shutter button while viewing this display.

[0056] In S116, the control unit 38 controls the imaging parameters of the imaging device 14 (here, focus control and exposure control) based on the matching result. For example, the control unit 38 can control the imaging parameters so that the focus and exposure are controlled for the face of the person whose name has been identified based on the matching result. That is, the control unit 38 may control the imaging parameters so that a person whose name has not been identified (who has not consented to the use of their biometric information) is not referenced in the control of the imaging parameters. Here, the focus and exposure control for a specific person can be performed using known AF / AE technology, and a detailed description thereof will be omitted. Note that, here, if there are multiple people whose names have been identified in the captured image, it is possible to determine in advance which of the people to focus and expose (for example, a priority can be set for each person).

[0057] In S117, the recording unit 40 records the captured image using the matching result. Here, the recording unit 40 captures a captured image when the user presses the shutter button 11, and records only the matching results of persons who have given their consent as tag information for the captured image (i.e., the recording unit 40 can control so that the matching results of persons who have not given their consent are not recorded in association with the image). In the example of FIG. 7 , as described above, information about "Mr. C," who has been identified with consent, is recorded as tag information. As shown in FIG. 7 , information about the face frame of G102 for "Mr. C" may also be recorded together with the captured image. According to this processing, it is possible to obtain captured images recorded in association with the matching results only for persons who have given their consent.

[0058] In step 118, the matching result utilization unit 37 determines whether to terminate the authentication process. Here, for example, if the user has input to terminate the authentication process, it may be determined that the registration process is terminated. If it is determined that the authentication process is terminated, the process in FIG. 6 ends; if not, the process returns to S112.

[0059] [Effect] According to this processing, it is possible to restrict the use of biometric information for persons who have not given their consent. In particular, it is possible to restrict the use of biometric authentication processing or processing results for persons who have not given their consent. In the authentication processing, biometric authentication processing is performed on the input face image to determine whether or not consent has been given, but the results are not used for persons who have not given their consent, which effectively restricts the biometric authentication processing. Therefore, it is possible to restrict the biometric authentication processing while respecting the individual's will.

[0060] [Variation 2-1] [Extracting Feature Amounts Regardless of Consent] In the registration process according to this embodiment, the feature amounts of persons who have not given consent are not registered. However, the feature amounts of persons who have not given consent may be registered even if the person has not given consent. In this case, the information processing device 1 restricts the use of biometric information by not using the matching results of persons who have not given consent during authentication processing, rather than not extracting feature amounts from persons who have not given consent.

[0061] [Modification 2-2] [Variations of Consent Information] In the present embodiment, the consent information has been described as information indicating only whether or not consent has been given, but it may also be possible to specify the type of consent, such as what consent is given for. The type of consent may be, for example, consent to using acquired biometric information to control imaging parameters such as AF / AE for the subject, consent to recording images of the subject, or consent to displaying the matching results, or other consent to various processes. Such processing can achieve restrictions that respect the individual's wishes in more detail.

[0062] [Variation 2-3] [Features are extracted during authentication without being registered] In the present embodiment, the features of the registered dictionary are described as being extracted and stored during the registration process, but features extracted during the authentication process may be registered in the registered dictionary. In that case, during the authentication process, consent information in the registered dictionary can be referenced, and features can be extracted only from persons who have consented, and compared with the input features.

[0063] [Variation 2-4] [Registering a Person Without Consent if the Matching Process Does Not Match Anyone] When the information processing device 1 determines in the matching process that the extracted feature does not match anyone through matching with the registered dictionary, the information processing device 1 may additionally register the person corresponding to the feature in the registered dictionary as a person without consent. In this case, in the example of FIG. 7 , G103 is a person who has been determined not to match anyone registered in the registered dictionary, so the facial image of G103 is registered. Here, the name set for a person without consent is set in advance to use a predetermined character string. In addition, in this case, a person who has been determined not to match anyone may be registered in association with feature extracted from the facial image. According to this processing, it is possible to explicitly identify a person without consent by looking at the registered dictionary, thereby improving convenience for the user of the information processing device 1.

[0064] [Variation 2-5] [Re-acquisition if Consent Is Not Given] The information processing device 1 may re-display a display for accepting a person who has not given consent, asking them to choose whether or not to consent to the use of their biometric information, and prompt them to consent to the use of their biometric information again (re-acquisition). That is, the consent information management unit 32 of the information processing device 1 can accept an instruction to change the consent information. In this case, using a method of obtaining consent by instructing the subject to make a gesture is highly convenient, as it allows consent information to be obtained simultaneously from multiple people in the photo. For a person for whom consent information indicating consent has been re-acquired, features are extracted from the facial image, and the features and consent information are newly registered in the registration dictionary. Furthermore, when previously acquired consent information indicates non-consent, the consent information management unit 32 of the information processing device 1 can also accept an instruction to change the consent status from non-consent to consent. In response to the consent information management unit 32 accepting the above-mentioned change instruction, the feature extraction unit 33 of the information processing device 1 extracts features from the registered biometric information (facial image) of the person corresponding to the consent information. The registration unit 35 then associates the facial image with the changed consent information and registers it. According to this modified example, there is no need to obtain consent and acquire a facial image at the same time, or it becomes possible to change a consent decision once made, thereby improving convenience for users who obtain consent and for people who give consent.

[0065] [Variation 2-6] [Variations in Use of Authentication Results] Furthermore, the recording of images of persons who have not given their consent may be restricted. For example, in the example of FIG. 7 , the face frames G101 and G103 of persons who have not given their consent may be filled in (for example, with a single color, black) and output as an image. In this way, the information processing device 1 can restrict the use of biometric information by outputting an image from which areas containing biometric information have been deleted. This processing allows images to be recorded with the consent of all persons whose face areas appear in the image. Furthermore, for example, the information processing device 1 may not record an image unless the consent of all persons appearing in the image has been obtained.

[0066] Although this method of restricting the recording of images is effective in terms of respecting the will of the individual, it is a strong restriction. From this perspective, for example, images may be recorded (output as is) for people who have not given their consent, but images may not be recorded (face areas are blacked out) for people who have refused to use their biometric information, so that images can be output that, to some extent, balances convenience and respect for the will of the individual.

[0067] In addition, in the present embodiment, when displaying the matching result, the presence or absence of consent is described as being displayed on a live view image as shown in FIG. 7 . However, such display of the presence or absence of consent is not limited to the live view image, but may also be displayed on an output image. The information processing device 1 in this embodiment has a function for displaying recorded captured images, as is typically provided in digital cameras, and consent information may be superimposed on such captured images, similar to the live view image in FIG. 7 . This type of processing makes it possible to check the presence or absence of consent for people whose images have been captured in the past.

[0068] Furthermore, as described in Modification 2-2, when consent is obtained for each process, whether or not the matching results are used for each process may be switched depending on the content of the consent. For example, if a person consents to the recording of an image but does not consent to the adjustment of focus or exposure using the matching results, restrictions on the use of biometric information are controlled for each such process. In this case, for example, the focus or exposure is not adjusted for that person, and an image of that person is recorded. In this way, restrictions can be imposed that are more in line with the person's intentions.

[0069] [Variation 2-7] [Variations of Information Processing Device] While the information processing device 1 according to this embodiment has been described as part of a digital camera, the information processing device 1 is not limited to this configuration as long as it can perform similar processing. For example, the information processing device 1 may be a smartphone with a camera or a network camera with a pan-tilt-zoom function that allows the angle of view to be adjusted. In this case, one possible method for using the matching results is to control pan-tilt-zoom to capture an identified individual. When using the information processing device 1 for such purposes, restricting the use of biometric information for persons who have not consented can prevent invasion of privacy and prevent misuse. Furthermore, when the information processing device 1 is a network camera, a separate server may be provided to manage the network camera and record captured video, and the server may perform the consent information acquisition or registration process. In such a configuration, when acquiring consent information, a UI for acquiring consent information on the server, i.e., a display indicating the use of biometric information and a display accepting consent or consent, may be displayed on a display device. The user may then input consent or consent using an input device such as a mouse or keyboard.

[0070] [Embodiment 3] In this embodiment, an example will be described in which consent information, biometric information, and feature amounts registered by the information processing device 1 according to embodiment 1 are output to another device. In addition, the other device used here is a digital camera, similar to the information processing device 1.

[0071] 8 is a schematic diagram showing an example of use of the information processing device 1 according to this embodiment. The information processing device 1 is the same as that in the first embodiment, and therefore a duplicated description will be omitted.

[0072] The network 15 is a network through which the devices communicate with each other. The network 15 may be, for example, a local area network (LAN), but the form of the network is not particularly limited as long as it can connect the devices so that they can communicate with each other. For example, the network 15 may be wireless or wired.

[0073] The information processing device 16 is a second information processing device to which the information processing device 1 according to this embodiment outputs consent information, and is assumed to be a digital camera in this example. The information processing device 16 according to this embodiment has the same hardware configuration and functional configuration as the information processing device 1, and redundant explanations will be omitted. The information processing device 16 is also capable of sending and receiving requests and data from other devices.

[0074] The mobile terminal 17 is a smartphone to which the information processing device 1 according to this embodiment outputs consent information. The mobile terminal 17 is capable of sending and receiving requests and data from other devices, similar to the information processing device 16. The information processing device 1 according to this embodiment outputs, in addition to consent information, feature amounts of persons who have given consent to the mobile terminal 17. This allows the mobile terminal 17 to use the acquired feature amounts of persons who have given consent. Conversely, for persons who have not given consent, the information processing device 1 cannot output consent information or feature amounts.

[0075] The PC 18 is a personal computer to which the information processing device 1 outputs consent information. The PC 18 according to this embodiment is capable of sending and receiving requests and data from other devices, similar to the information processing device 16. Furthermore, the information processing device 1 outputs, in addition to the consent information, the feature amounts of the consenting person to the PC 18. This allows the PC 18 to use the acquired feature amounts of the consenting person.

[0076] Fig. 9 is a flowchart showing an example of output processing by the information processing device 1 according to this embodiment. The processing described in Fig. 9 starts when, for example, a user of the information processing device 1 instructs execution of the output processing (for example, by pressing the operation button 13 on the display panel 12 shown in Fig. 1A). Here, an example will be described in which the information processing device 1 described in Fig. 8 outputs data included in the registered dictionary, i.e., consent information, biometric information, and feature amounts, to the information processing device 16.

[0077] In S121, the output unit 34 acquires the registered dictionary registered by the registration unit 35. As described above, the registered dictionary includes consent information, biometric information, and feature amounts. As described in the first embodiment, this consent information is acquired by the consent information management unit 32. The output unit 34 according to this embodiment acquires the consent information stored in the registered dictionary, but may also acquire consent from an individual as in the first embodiment.

[0078] At S122, the output unit 34 establishes a connection with the destination information processing device 16, enabling data output. At S123, the output unit 34 references the registered dictionary held by the destination information processing device 16 and compares it with the registered dictionary held by the information processing device 1 to detect differences. FIGS. 10A-C are schematic diagrams illustrating the above-described difference detection and synchronization process performed by the information processing device 1 according to this embodiment. FIG. 10A shows the registered dictionary held by the information processing device 1, and FIG. 10B shows the registered dictionary held by the destination information processing device 16. The output unit 34 compares the registered dictionaries and detects differences. The process of detecting differences is a process of identifying information not stored in the external device as differences. In this example, the difference is that although "Mr. A" at No. 1 in each registered dictionary is the same person, different facial images and feature values ​​are registered in each. Furthermore, for Nos. 2 and 3 in FIG. 10A, the person not registered in FIG. 10B is also a difference.

[0079] In order to determine whether the two people are the same person, the matching unit 36 ​​can perform the above-mentioned matching process on each feature. The matching unit 36 ​​may also store unique numbers, symbols, etc. that identify individuals in a registered dictionary, and determine whether the two people are the same person by comparing such information. As described above, the "Person's Name" field contains a person's name or nickname, and is not necessarily a unique number or symbol, so it may not be used to determine whether the two people are the same person.

[0080] In S124, the output unit 34 displays to the user which information to output and then confirms whether or not it is OK to output the information. Here, the output unit 34 displays a screen such as that shown in FIGS. 10A and 10B. In this example, the output unit 34 outputs the facial image and feature amounts of No. 1 and the facial image and feature amounts of No. 3 from the information processing device 1. Along with this display, the information processing device 1 receives input from the user via an input unit (not shown) regarding whether or not to output the information.

[0081] In S125, if the user has input permitting output (Yes in S125), the output unit 34 proceeds to S126, otherwise, ends the processing in Fig. 12. In S126, the output unit 34 outputs the consent information, biometric information, and feature amounts.

[0082] 10C shows the registered dictionary of the information processing device 16 after output. The registered dictionary shown in FIG. 10C reflects information from the registered dictionary (a) of the information processing device 1, which was not held before output. In this embodiment, the information processing device 1 also acquires the difference from the information processing device 16 and synchronizes with the registered dictionary held in the information processing device 16. Therefore, the registered dictionaries of the information processing device 1 and the information processing device 16 are synchronized, and both information processing devices store the registered dictionary shown in FIG. 10C. When the registered dictionary of the information processing device 1 is output, the registered dictionaries of both information processing devices do not necessarily need to be synchronized, and only the registered dictionary held in the information processing device 16 may be updated.

[0083] In FIG. 10C , multiple facial images and feature amounts are registered for one person ("Mr. A"), and matching can be performed by matching each feature amount and then calculating a representative matching result from the multiple matching results. Any known method for matching data can be used to calculate the representative matching result. For example, matching can be performed by selecting the matching result with the highest similarity or by averaging the similarities. According to this type of processing, by registering multiple facial images and feature amounts for one person, matching can be performed based on facial images and feature amounts of that person taken under multiple shooting conditions, thereby improving matching accuracy.

[0084] [Effects] With this configuration, the consent information, biometric information, and feature amounts acquired by the information processing device 1 can be output to another device. By outputting the consent information to another device and making it available on the output destination device, it is possible to eliminate the need to make some kind of contact with the person to be processed to acquire the consent information. For example, in a group of family members or friends who want to share consent information, the consent information can be synchronized between the information processing devices, thereby reducing the number of cumbersome consent acquisitions.

[0085] Furthermore, by outputting the biometric information and feature quantities together with the consent information, the procedure of acquiring the biometric information and feature quantities again on the output destination device can be omitted. Note that by outputting the feature quantities, the feature quantities can be used for facial recognition on another device, but by also outputting the biometric information (here, a facial image), the feature quantities can be extracted again from the facial image when the facial recognition model is updated. Furthermore, with this configuration, the user of the device can visually check the registration status, such as which individuals have consented or are registered, thereby improving convenience from a management perspective.

[0086] [Variation 3-1] [Variations in Output Destination] In this embodiment, the information processing device 1 and the information processing device 16 are both digital cameras. However, other devices capable of executing similar processing may be used. For example, a smartphone, a personal computer, or other device may be used as the information processing device (1 or 16). Such information processing devices may be required to search for a specific person among a large number of captured images. Therefore, the information processing device may search only for people who have given consent based on the output consent information. That is, when a user specifies a person to search, the consent information may be referenced and the user may specify only people who have given consent. This processing allows only images of people who have given consent to be searched, thereby realizing restrictions that better respect individual consent. Furthermore, the information processing device 16, which serves as the output destination, may be a device that provides cloud services. Even in such a case, the information processing device 1 according to this embodiment can output consent information to the output destination device via a network.

[0087] [Modification 3-2] [Variations of consent information] The information processing device 1 may also be configured to allow a user to specify consent to outputting consent information or biometric information to another device. With such a configuration, it is possible to prevent the consent information or biometric information from being inadvertently output to another device while permitting use of the consent information or biometric information by the target device.

[0088] [Variation 3-3] [Selection of consent information to output] Although the information processing device 1 according to the present embodiment has been described as automatically synchronizing and outputting the output destination and consent information, the user may be able to select the information to synchronize. In this case, the items that the user can select are limited to those for which the consent information indicates consent. With this configuration, the user can select and synchronize only the consent information that is necessary.

[0089] [Modification 3-4] [Variation of Output Information] In the present embodiment, an example has been described in which a registered dictionary is output to another device and synchronized, but only a portion of the information contained in the registered dictionary may be output and synchronized. For example, only consent information and personal names may be output and synchronized, or only biometric information or feature amounts may be output and synchronized.

[0090] [Embodiment 4] [Consent to AI Processing and Use of Processing Results] In embodiments 1 to 3, examples of restricting the use of biometric information were described. However, in this embodiment, a case where the target of use restriction is not limited to biometric authentication, but the use of AI processing and its processing results is restricted will be described. In this embodiment, a method for restricting AI processing targeting customers who have not given consent by identifying customers (users) appearing in images captured by cameras installed in a commercial facility and consent information related to those customers will be described. Note that in this case, it is assumed that multiple services using AI processing are provided at the commercial facility, and consent is obtained for each of these services.

[0091] In this embodiment, the term "AI processing" refers to processing that performs intelligent processing, identification, judgment, etc. using techniques such as machine learning, and specifically, AI processing is assumed to include face detection, face recognition, age / gender estimation, etc. Furthermore, the AI ​​processing according to this embodiment is not limited to image processing, but also includes text analysis, analysis of other sensing data, etc.

[0092] The processing performed by the information processing device 1 according to this embodiment will be described below. FIG. 11 is a schematic diagram showing an example of the system configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment functions as a server that controls the system of this embodiment. The information processing device 1 according to this embodiment also includes input devices such as a mouse and a keyboard, as well as a screen display panel for operation, and can accept operations from the administrator of the system of this embodiment. The network 15 is the same as that of embodiment 3. The other hardware configurations are the same as those of the information processing device 1 according to embodiment 1, so duplicated explanations will be omitted.

[0093] Imaging device 41 is a camera for facial recognition payment, and is composed of a camera unit and a communication device. Imaging device 41 transmits the captured facial image to information processing device 1 via network 15. Surveillance camera 42 and surveillance camera 43 are a first surveillance camera and a second surveillance camera, respectively. Surveillance camera 42 and surveillance camera 43 are each composed of a camera unit and a communication device, and transmit the captured image to information processing device 1 via network 15.

[0094] 12 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. The biometric information acquisition unit 51 acquires images of an individual, in this case, a customer, from the face recognition payment camera 41, the surveillance camera 42, and the second surveillance camera 43.

[0095] The individual identification unit 52 identifies the individual appearing in the image acquired by the biometric information acquisition unit 51. The consent information management unit 53 acquires and manages consent information indicating what AI processing an individual has consented to and the use of the processing results.

[0096] The AI ​​processing unit 54 is an AI processing unit that performs AI processing on the image acquired by the biometric information acquisition unit 51. The AI ​​processing unit 54 according to this embodiment refers to the consent information and executes only the AI ​​processing to which the individual who is the target of the AI ​​processing has consented. In other words, the AI ​​processing unit 54 according to this embodiment restricts the AI ​​processing based on the consent. A specific description of the content of the AI ​​processing will be given later.

[0097] The AI ​​processing result utilization unit 55 utilizes the results processed by the AI ​​processing unit 54. The AI ​​processing result utilization unit 55 creates information that serves as the basis for services to be provided to individuals.

[0098] The consent condition storage unit 56 stores consent conditions, including conditions related to consent, for each individual, and presents them to the user. The consent conditions according to this embodiment include information such as the content of the AI ​​processing, the data used by the AI ​​processing, the purpose of the AI ​​processing, information about the organization performing the AI ​​processing, information about the organization managing the data used by the AI ​​processing, the services provided to the customer, and the period during which the AI ​​processing will be performed. These consent conditions are presented to the customer in advance and consent is obtained, and are conditions for implementing the AI ​​processing described below. The output unit 57 outputs and displays the information created by the AI ​​processing result utilization unit 55 and the consent conditions stored in the consent condition storage unit 56 to an external device.

[0099] The specific steps of the processing performed by the information processing device 1 according to this embodiment are described below. The processing according to this embodiment is divided into two processes: a pre-processing in which consent conditions are presented to a customer and consent information is registered in a registration dictionary together with a facial image or feature amount, and an AI processing in which AI processing is performed based on the consent information registered in the pre-processing.

[0100] In the pre-processing according to this embodiment, feature amounts, biometric information, consent information, and personal names are linked and registered in a registration dictionary, similar to the registration processing described in embodiment 1. Since this embodiment targets customers who use commercial facilities, this registration processing is performed, for example, using the customer's smartphone when the customer enters the commercial facility, or the registration processing is performed in advance on a website provided by the commercial facility. Another difference between the processing performed by the information processing device 1 according to this embodiment and that in embodiment 1 is that the consent information includes consent to multiple AI processes.

[0101] FIG. 13 is a diagram schematically illustrating the contents of a registered dictionary in this embodiment. The columns for No., person's name, face image, and feature value shown in FIG. 13 are the same as those in embodiment 1, and therefore will not be described here. The consent information column displays services using AI processing, such as "Facial Recognition Payment," "Crowd Analysis," and "Visitor Trend Analysis," with check boxes indicating consent for each service. In the example shown in FIG. 13, "Mr. A" consents to all services, while "Mr. B" does not consent to any of the services. On the other hand, "Mr. C" does not consent to "Facial Recognition Payment" and "Visitor Trend Analysis," but does consent to "Crowd Analysis." Thus, the consent information according to this embodiment describes the purpose (type) of AI processing and includes information indicating whether the user consents to each AI processing. Note that the AI ​​processing shown in FIG. 13 is an example; other types of AI processing, such as face detection, face recognition, or age / gender estimation, may also be described.

[0102] This consent information is acquired by the operator of the commercial facility presenting the conditions of each consent information (consent conditions) to the customer when the customer enters the commercial facility and confirming the customer's willingness to consent. For example, the address of a web page showing the consent conditions is presented to the customer, and the customer who accesses the address is asked to confirm the consent conditions on a smartphone or other device, and consent is acquired based on the user's input. The address may be presented as text on a poster or the like, or as a two-dimensional barcode or the like to reduce the customer's input effort. The form of the address is not particularly limited as long as it is accessible to the user. Furthermore, the address may be distributed to the customer's smartphone or the like at a specific location using contactless communication technology.

[0103] 14 is a schematic diagram showing an example of consent conditions presented to a customer. The consent conditions are presented in this way, and consent information is acquired by having the customer input whether or not they consent. The consent conditions shown in FIG. 14 present information about services using the three AI processes described above: "face recognition payment," "crowding analysis," and "visitor trend analysis."

[0104] "Facial recognition payment" is a service that identifies people in an image and provides a function for making payments such as purchasing goods using payment information (such as bank account or credit card information) pre-linked to that person. "Crowd analysis" is a service that counts the number of people in an image, visualizes the number of people within the camera's field of view, and presents the visualization results to customers. "Visitor trend analysis" is a service that estimates the age and gender of customers to analyze trends among visitors and provide advertisements based on their age and gender on signage in commercial facilities, etc. All of these services using AI processing can be realized by using known AI processing such as face detection, face recognition, or age and gender estimation.

[0105] In the example of the service item "1. Facial Recognition Payment" in FIG. 14 , the information presented includes AI processing details 201, data processed by AI 202, services provided to the customer by AI processing 203, and consent period 204. The consent period indicates the period during which the consent setting is valid. In the example of FIG. 14 , the customer can enter the details, but a specific date or period may be presented and the user's selection accepted. After confirming these conditions, the customer can indicate their consent by selecting a check box 205. In this case, an unselected check box means that the user has not consented. For other service items, the same information as "1. Facial Recognition Payment" is presented. Note that if a length is set as the consent period, the period for executing the AI ​​processing can be set by obtaining the current time when the AI ​​processing is executed.

[0106] When the customer has finished inputting whether or not they agree to all services, they press the agreement button 206 to determine their intention to agree to these agreement conditions, i.e., their agreement information, and the agreement information is acquired by the information processing device 1.

[0107] With this configuration, it is possible to obtain consent information after having the customer confirm the services they receive and the data they provide (data processed by AI). Therefore, it is possible to provide only AI processing services that the customer has consented to, and conversely, AI processing that the customer has not consented to is not performed, i.e., the implementation of AI processing can be restricted. Therefore, it is possible to provide services that are in line with the customer's wishes, improving convenience for both the service provider and the customer.

[0108] [AI Processing] Next, the AI ​​processing according to this embodiment will be described. Fig. 15 is a flowchart showing an example of the AI ​​processing executed in this embodiment. Note that the processing shown in Fig. 15 is executed after the above-mentioned preliminary processing is completed. If the processing shown in Fig. 15 is executed before the preliminary processing is completed, there is no person who is said to have consented to the AI ​​processing, and the AI ​​processing with consent described below is not executed.

[0109] In S131, the information processing device 1 performs initialization processing for AI processing. Here, the information processing device 1 loads the registered dictionary registered in the pre-processing and makes it readable.

[0110] In S132, the biometric information acquisition unit 51 acquires an image of an individual from a camera. In this example, the biometric information acquisition unit 51 acquires images from the monitoring cameras 42 and 43.

[0111] In S133, the individual identification unit 52 identifies the identity of the person in the acquired image. Here, the individual identification unit 52 identifies, for all people in the image, whether they correspond to any person registered in the registration dictionary or whether they do not match any person. This identification method can be performed in the same way as in S114 of the second embodiment. The information processing device 1 according to this embodiment extracts features from the captured image regardless of whether consent is obtained, to identify an individual, and restricts the use of subsequent biometric information for that individual based on the consent information.

[0112] In S134, the consent information management unit 53 acquires the consent information of the identified person. In the example of Fig. 13, the feature amount and the consent information are registered in association with each other, so the consent information management unit 53 refers to the consent information of the identified person. Note that, again, for a person who is determined not to match anyone in the comparison process, no consent information exists, and therefore the person is considered to have not consented to any services.

[0113] In S135, the AI ​​processing unit 54 restricts the use of the biometric information based on the consent information, and then executes AI processing on the acquired image (including the face). Here, the AI ​​processing unit 54 executes AI processing for which consent has been obtained, while not executing AI processing for which consent has not been obtained, based on the consent information.

[0114] It should be noted that, here, the (type of) imaging device and the type of AI processing to be executed are associated and set in advance, and only AI processing for which consent has been obtained is executed from among the types of AI processing thus set in advance. Here, images are acquired from surveillance camera 42 and surveillance camera 43, and the AI ​​processing to be executed is "congestion analysis" and "visitor trend analysis" associated with those surveillance cameras (to the type "surveillance camera"), and "face recognition payment" is not executed.

[0115] For example, if a person identified as "Mr. C" shown in FIG. 13 is present in the captured image being processed, age and gender estimation for the "Visitor Trend Analysis," the only AI process for which consent is granted, is executed based on Mr. C's consent information. On the other hand, if a person detected in the captured image does not consent to the "Visitor Trend Analysis," age and gender estimation is not executed for that person. For example, if a person identified as "Mr. A" shown in FIG. 13 is present in the captured image, all executable AI processes are executed because Mr. A's consent information indicates consent for all AI processes. This processing is performed in accordance with the consent conditions stored in the consent condition storage unit 56 and presented to the customer, as shown in FIG. 14. The information processing device 1 may be configured to allow the operator of the system to confirm whether the processing content, data to be processed, or purpose described in FIG. 14 is in accordance with the processing content, data to be processed, or purpose.

[0116] In S136, the AI ​​processing result utilization unit 55 performs various processes using the results of the AI ​​processing. For example, the AI ​​processing result utilization unit 55 may convert and process the results of AI processing, such as face detection or age / gender estimation, into a form that can be used for services and output it to another device as needed. For example, when the AI ​​processing result utilization unit 55 executes a "visitor trend analysis," it may output data counting the number of people for each estimated age and gender. Here, the age and gender are estimated only for people who have consented to the "visitor trend analysis" in the captured image, and only those who consent are counted. This count may be accumulated for a predetermined period while the system is operating, and the number of visitors for each age and gender during that period may be calculated and output to another device as needed. This information can be used as data to improve the operation of the commercial facility, and is therefore beneficial to commercial facility operators.

[0117] Furthermore, the AI ​​processing result utilization unit 55 may provide a service to customers by displaying advertisements based on the estimated age and gender on a signage terminal near the camera that captured the image. This type of processing is beneficial because it can provide advertisements that are likely to be suitable for the customer. In this way, AI processing and the use of processing results that are beneficial to both the operator of the commercial facility and the customer can be carried out with consent.

[0118] Furthermore, when the AI ​​processing result utilization unit 55 executes the "crowding analysis," it can count the number of people in the same way as when executing the "visitor trend analysis," and output the count results to another device. This allows the congestion status to be presented to customers, improving customer convenience. Such a method of presenting the congestion status may be displayed on a display in the store, or may be distributed via an application to the customer's smartphone, etc.

[0119] In S137, the information processing device 1 determines whether to terminate the AI ​​processing. Here, the information processing device 1 determines to terminate the AI ​​processing if an instruction to terminate has been received from the administrator of the system, and determines not to terminate the AI ​​processing if not. If it is determined in S137 that the AI ​​processing should be terminated, the processing of FIG. 15 ends, and if not, the processing returns to S132.

[0120] According to this type of processing, it is possible to limit the processing performed on customer data to only those that have been consented to. In particular, since it is possible to limit AI processing after obtaining the customer's consent to the execution of AI processing for each AI processing service, it is possible to perform processing that better respects the individual's will. Therefore, from the customer's perspective, it is possible to indicate whether or not to consent to AI processing for each AI processing service, and from the service provider's perspective, it is possible to provide only the AI ​​processing and services that the customer desires, thereby improving convenience.

[0121] It is also possible to offer more dynamic conditions for limiting and allowing AI processing, as well as rewards, such as offering customers a time-limited online coupon if they allow a business to use their anonymized in-store purchasing history to improve their service.

[0122] Furthermore, by presenting the consent conditions to the customer in advance, it is possible to present what information about the customer will be used in AI processing and what benefits the customer will receive as a result. This allows the customer to choose whether or not to consent to AI processing for each service. This allows the customer to choose whether or not to receive a service based on AI processing, taking into account the advantages and disadvantages according to their own will, thereby improving convenience.

[0123] [Variation 4-1] In the fourth embodiment, an example was described in which an image from a surveillance camera was acquired and AI processing was performed using the surveillance camera, but each process can be performed in the same way even when a different type of imaging device is used. Below, we will explain a case in which each process is performed using an image acquired from the facial recognition payment camera 41 instead of the surveillance cameras 42 and 43. Here, it is assumed that "facial recognition payment" is associated with the facial recognition payment camera as the AI ​​processing to be performed.

[0124] Here, in S132, an image is acquired from the face recognition payment camera 41, and in the subsequent processing, the AI ​​processing of "face recognition payment" is executed. In the following, the description of the processing that overlaps with that described with reference to FIG. 15 will be omitted.

[0125] In S135, the AI ​​processing unit 54 executes AI processing for the user who has consented to "face authentication payment" based on the consent information. Here, the user is identified using face authentication, but the result of identifying the user performed in S133 may be used.

[0126] In S136, the AI ​​processing result utilization unit 55 performs payment based on the facial recognition result, referring to payment information such as a pre-registered credit card or bank account, and transmits the payment result to the payment terminal of the commercial facility. Note that prior to this process, there is assumed to be processing such as a store clerk at the commercial facility determining the payment amount or obtaining the customer's intention to use facial recognition payment. Note that for persons who have not consented, payment cannot be made, and therefore information indicating that the payment has failed is output to the payment terminal.

[0127] This type of processing also makes it possible to restrict use based on consent for services that use relatively confidential information, such as payment information. Some customers may feel uneasy about using such information, so by making it possible to restrict use based on consent, it becomes possible to process services that take into consideration the customer's wishes.

[0128] [Variation 4-2] In this embodiment, it has been described that all consent information of persons not in the registered dictionary is processed as if they have not consented, but this processing is not particularly limited to this. For example, the information processing device 1 may perform each process assuming that persons not listed in the registered dictionary have consented to all AI processing. In this case, however, it is desirable to notify customers by displaying a message such as "If a person enters a commercial facility according to operating regulations, they will be considered to have consented to AI processing" so that customers do not feel uncomfortable that AI processing has been performed without their consent.

[0129] [Variation 4-3] In this embodiment, biometric information, i.e., an image of a face, is used as the data input to AI processing, but it does not have to be biometric information. For example, personal information such as an individual's name, address, email address, telephone number, or attribute information (race, gender, age, occupation) may be registered in advance in a registration dictionary as text or numerical values ​​and used for AI processing. This personal information may be an ID number assigned a unique number to each individual. In this case, an example of AI processing and services may include behavioral and purchasing predictions using personal text data. Even if the input data is text or numerical values, because it is personal information, it can be provided with a service that is highly satisfactory to customers by restricting its use in accordance with the individual's consent.

[0130] Furthermore, in the present embodiment, biometric information is used to identify the individual, but biometric information does not necessarily have to be used as long as the individual can be identified. For example, the individual may be identified by information such as a membership number or a user name, and consent information in a registered dictionary may be referenced. Furthermore, for example, the individual may be identified by an RFID tag carried by the user or by device-to-device communication with a smartphone.

[0131] [Variation 4-4] In this embodiment, information such as the content of the AI ​​processing, the data to be used, or the purpose is presented as a consent condition, but the information presented here is not limited to this. For example, as described above, the name of the organization or company that manages the AI ​​processing and the data to be used may be presented as part of the consent condition. Such processing allows the customer to decide whether to consent by taking into account whether the organization or company is trustworthy. In this case, the organization or company may be assigned a score in advance indicating whether it is trustworthy, and this score may be presented to the customer.

[0132] Information about the location of use may also be presented as a condition of consent. For example, information about whether the user's information will be used only at that commercial facility or at commercial facilities in other locations may be presented as part of the condition of consent. Furthermore, the screen presenting the condition of consent may allow the customer to select the location where their information will be used. This type of processing makes it possible to configure the system so that customers can collectively indicate their consent for all commercial facilities operated by the same operator.

[0133] [Variation 4-5] In the present embodiment, an example has been described in which the information processing device 1 is used for AI processing and services in a commercial facility as shown in Figure 14, but the location, AI processing, and services are not limited to those described here. For example, when using biometric information as data for improving the operation of the commercial facility, a process of restricting the use of biometric information in the information processing device 1 may be performed. In this case, the information processing device 1 according to the present embodiment can be used, for example, when performing AI processing to constantly detect and track customers and visualize their movement paths within the commercial facility.

[0134] Furthermore, by using facial recognition to manage entry to specific locations or rooms, it is possible to eliminate the need for membership cards or keys. From this perspective, the information processing device 1 may be used in commercial facilities such as sports gyms, schools, hospitals, and other facilities. When the information processing device 1 is used in a school or hospital, in addition to the above-mentioned AI processing for entry management, it is expected that the information processing device 1 will be used for AI processing and services such as operating an AI to detect suspicious individuals for use in a security service, or operating an AI to detect abnormal behavior such as falls for use in a monitoring service. Even in such cases, by obtaining the user's consent for each AI processing and service, it is possible to provide services that reflect the customer's wishes.

[0135] However, some services that are of high public interest or urgency, such as security services, monitoring services, or guidance services based on an analysis of the number or location of passersby during a disaster, may be implemented without consent. In such cases, a separate status notification module (not shown) may be provided to control the services provided by the information processing device 1 based on the notification results so that the services will operate only under appropriate conditions in accordance with current or future laws and regulations. This configuration allows customers to always receive urgent services without having to wait for consent each time, thereby improving convenience.

[0136] For example, in a group setting such as a school, when a service is provided in which automatic photographing of faces using face detection is performed and the photographs are classified by person and sold, consent may be obtained for both automatic photographing using face detection and consent for person identification processing for classifying the photographs by person. Such processing allows AI processing to be restricted according to individual wishes. In particular, it becomes possible to perform processing that respects the detailed wishes of individuals, such as allowing only face detection but not allowing classification to identify the individual.

[0137] Although the present embodiment has been described assuming that consent information is obtained from the user, the consent information does not have to be information entered by the user himself / herself, as long as it indicates whether or not the user has given consent. For example, a parent or guardian of a user (especially a child) may submit consent information on behalf of the user. Such processing can also accommodate cases where, for example, when a child gets lost in a commercial facility and AI processing (face detection and face recognition) is used on surveillance camera images, the parent or guardian enters consent information on the child's behalf. Furthermore, the present embodiment has been described assuming that consent is obtained upon entering a commercial facility. However, it is also possible for a parent or guardian to access the system and consent to the AI ​​processing for finding the lost child when the child becomes lost. In this way, the person entering the consent information does not have to be the user himself / herself; the consent information may be entered by a person deemed to have the right to provide consent. Furthermore, the time or place at which consent is given is not limited and may be any time before AI processing is performed. For example, the consent information may be entered when the service is needed.

[0138] [Variation 4-6] In this embodiment, consent information for the use of AI processing is acquired for each commercial facility. However, for example, by sharing consent information between facilities, it is possible to eliminate the need for users to enter consent information one by one at each facility. Furthermore, for example, similar to setting security items for each trust level in Internet web browsing, multiple trust levels and corresponding consent conditions may be associated in advance, and processing may be performed based on these settings. In this case, facility users can set consent simply by specifying which trust level the facility they are using corresponds to. Furthermore, (default) consent conditions may be set in the absence of user input. Furthermore, in this embodiment, consent information is described as being entered in advance, but it may also be possible to edit the information, for example, while using the facility. This configuration allows users to change the simultaneous conditions at a certain timing, improving convenience.

[0139] [Other Embodiments] In the above-described first to fourth embodiments, examples of acquiring a face image as biometric information have been described. However, the biometric information is not limited to images containing a face, as long as it includes information about the user. For example, the biometric information may be an image containing information that can identify the user, such as an individual's iris, fingerprint, or veins. Even when using such biometric information, it is possible to similarly extract features from the biometric information. Note that the device for acquiring the captured image may be an imaging device that includes an appropriate sensor corresponding to each type of biometric information.

[0140] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0141] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0142] This application claims priority based on Japanese Patent Application No. 2024-030639, filed February 29, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device comprising: a biometric information acquisition means for acquiring biometric information of a person; a consent information acquisition means for acquiring consent information indicating the person's consent status regarding the use of features extracted from the biometric information; an extraction means for extracting the features from the person's biometric information; and a restriction means for restricting at least one of the extraction and use of the features based on the consent information acquired by the consent information acquisition means.

2. The information processing device according to claim 1, characterized in that the restriction means controls the extraction means not to extract the feature when the consent information indicates that the person does not consent to the use of the feature.

3. An information processing device as described in claim 1 or 2, characterized in that it has a registration means for registering the feature extracted by the extraction means and the consent information acquired by the consent information acquisition means in association with each other, and the restriction means controls the registration means to prevent the registration of the feature if the consent information indicates that the person does not consent to the use of the feature.

4. The information processing device according to claim 3, characterized in that, when the consent information indicates that the person does not consent to the use of the features, the registration means associates the person's biometric information with the consent information and registers them.

5. An information processing device as described in claim 4, further comprising a reception means for receiving an instruction to change the consent information, wherein the extraction means extracts features from registered biometric information of the person corresponding to the consent information in response to the reception means receiving a change instruction to change the consent status indicated by the consent information from non-consent to consent.

6. The information processing device according to claim 5, characterized in that the registration means associates and registers the biometric information acquired by the biometric information acquisition means, the consent information acquired by the consent information acquisition means, and the features extracted by the extraction means.

7. An information processing device as described in any one of claims 1 to 6, characterized in that it has a matching means that performs matching processing using the features extracted by the extraction means and pre-registered features, and the restriction means restricts the use of the matching results by the matching means depending on the consent information acquired by the consent information acquisition means.

8. The information processing device according to claim 7, wherein said restriction means controls so that the results of said matching process relating to persons whose consent information does not indicate consent are not displayed.

9. An information processing device as described in claim 7, comprising an imaging means and an imaging control means for controlling said imaging means based on the result of the matching by said matching means, wherein said restriction means controls said imaging control means not to control said imaging means for a person whose consent information does not indicate consent.

10. An information processing apparatus according to claim 9, wherein the control performed by said imaging control means includes at least one of focus control and exposure control.

11. An information processing device as described in claim 9 or 10, characterized in that it has a first recording means for recording personal information obtained by the matching process by the matching means in association with an image captured by the imaging means, and the restriction means controls so that personal information regarding a person for whom the consent information does not indicate consent is not recorded in association with the image.

12. An information processing device according to claim 9, further comprising a second recording means for recording images captured by said imaging means, and wherein said restriction means controls so that images showing persons for whom said consent information does not indicate consent are not recorded by said second recording means.

13. An information processing device as described in claim 7, further comprising a transmission means for transmitting personal information obtained by the matching process by said matching means to another device via a communication network, and said restriction means controls so that personal information of persons for whom the consent information does not indicate consent is not transmitted by said transmission means.

14. An information processing device according to any one of claims 1 to 13, wherein the biometric information is an image of the person's face, iris, fingerprint, or veins.

15. An information processing device further comprising: a biometric information acquisition means for acquiring biometric information of a person; a consent information acquisition means for acquiring consent information indicating the consent status of the person regarding AI processing of the biometric information; a processing means for performing AI processing on the biometric information of the person; and a restriction means for restricting AI processing by the processing means based on the consent information acquired by the consent information acquisition means.

16. An information processing device as described in claim 15, further comprising a presentation means for presenting consent conditions, which are conditions for the person's consent to the use of the biometric information in the AI ​​processing, and the consent information is information set based on the consent conditions.

17. The information processing device according to claim 16, wherein the consent conditions include any one of the content of the AI ​​processing, the data used by the AI ​​processing, the period during which the AI ​​processing is performed, information about the organization performing the AI ​​processing, and information about the organization managing the data used by the AI ​​processing.

18. The information processing device according to claim 17, characterized in that the information used in the AI ​​processing includes the person's name, address, email address, telephone number, ID number, or attribute information.

19. An information processing method executed by an information processing device, comprising: a biometric information acquisition step of acquiring biometric information of a person; a consent information acquisition step of acquiring consent information indicating the person's consent status regarding the use of features extracted from the biometric information; an extraction step of extracting the features from the person's biometric information; and a restriction step of restricting at least one of the extraction and use of the features based on the consent information acquired by the consent information acquisition step.

20. An information processing method performed by an information processing device, comprising: a biometric information acquisition step of acquiring a person's biometric information; a consent information acquisition step of acquiring consent information indicating the person's consent status regarding AI processing of the biometric information; a processing step of performing AI processing on the person's biometric information; and a restriction step of restricting the AI ​​processing by the processing step based on the consent information acquired by the consent information acquisition step.

21. A program for causing a computer to function as each means of the information processing device according to any one of claims 1 to 18.

Images