Biological information processing device, information processing system, biological information processing method, and program

Abstract

To provide a biometric information processing device that utilizes an individual's biometric information and its characteristic features in an ethically sound manner, without requiring any effort or obtaining the individual's consent, while respecting the individual's privacy. [Solution] The biometric information processing device comprises a biometric information acquisition unit that acquires biometric information, a biometric information utilization unit that utilizes biometric information or features extracted from biometric information, an explicit consent acquisition unit that acquires consent for the use of the biometric information of the person to whom the biometric information pertains, a biometric information utilization restriction unit that restricts the use of biometric information based on the consent value, and a consent state estimation unit that estimates a consent state indicating consent for the use of the biometric information of the person to whom the biometric information pertains based on the acquisition status at the time of acquisition of the biometric information. The biometric information utilization restriction unit then relaxes the restrictions on the use of the biometric information of the person to whom the biometric information pertains or features extracted from biometric information based on the consent state estimation unit.

Description

Technical Field

[0001] The present invention relates to a biological information processing device, and particularly relates to a biometric authentication technology suitable for using an individual's biological information without ethical problems even without obtaining the individual's consent.

Background Art

[0002] The use of biological information (biometrics) has been significantly expanding in recent years. As typical examples, biometric authentication technologies in security and authentication, and AI processing that performs processes such as detection and estimation using parameters learned from data such as images are known. Since such AI processing can perform intelligent processing and judgment like humans, its applications in various fields have been progressing in recent years.

[0003] On the other hand, there is a fear of violating an individual's privacy and ethical problems when a third party uses an individual's information (for example, face images, fingerprints, irises, etc.) without obtaining the individual's consent, and certain considerations are required. In particular, biometric authentication such as face images and fingerprints may violate an individual's privacy depending on the application, and it has been pointed out that the collection and use of biological information may contribute to inappropriate surveillance and discrimination. Therefore, further restrictions are required for the use of biological information.

[0004] Regarding the technology using face authentication, for example, it is described in Patent Document 1. Patent Document 1 discloses a method of obtaining an individual's consent before registering a face authentication subject.

[0005] Also, related to this, as a technology for specifying the position when acquiring a face image, for example, it is described in Non-Patent Document 1.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Non-Patent Documents

[0007] [Non-Patent Document 1] Deng, Jiankang, et al. “Retinaface: Single-shot multi-level face localization in the wild.” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020. [Overview of the project] [Problems that the invention aims to solve]

[0008] Although Patent Document 1 states that prior consent should be obtained for the acquisition of facial images, there was a risk that features would be extracted from biometric information, such as facial images, and used without the individual's consent.

[0009] Furthermore, even if the system is controlled to allow the use of only the biometric information of those who have given their consent, obtaining consent from individuals is a cumbersome process for both users and operators of biometric authentication, reducing the convenience of operating the system.

[0010] The objective of the present invention is to provide a biometric information processing device that allows an individual to utilize their biometric information and biometric information characteristics in an ethically sound manner, without requiring the operator to expend time or obtain the individual's consent, while respecting the individual's privacy. [Means for solving the problem]

[0011] The configuration of the biometric information processing apparatus of the present invention preferably comprises a biometric information processing apparatus that processes using a person's biometric information, and further comprises a biometric information acquisition unit that acquires biometric information, a biometric information utilization unit that utilizes biometric information or features extracted from biometric information, an explicit consent acquisition unit that acquires consent for the use of the biometric information of the person to which the biometric information pertains, and a biometric information utilization restriction unit that restricts the use of biometric information based on the consent value, and further comprises a consent state estimation unit that estimates a consent state indicating consent for the use of the biometric information of the person to which the biometric information pertains based on the acquisition status at the time of acquisition of biometric information, and the biometric information utilization restriction unit relaxes the restrictions on the use of the biometric information of the person to which the biometric information pertains or features extracted from biometric information based on the consent state estimation unit. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a biometric information processing device that allows an individual to utilize their biometric information and biometric information characteristics in an ethically sound manner, without requiring the operator to expend time or obtain the individual's consent, while respecting the individual's privacy. [Brief explanation of the drawing]

[0013] [Figure 1A] This is a diagram (part 1) illustrating the image acquisition process when using a bio-information processing device. [Figure 1B] This is a diagram (part two) illustrating the image acquisition process when using a biometric information processing device. [Figure 2] This is a hardware configuration diagram of the imaging device. [Figure 3] This is a block diagram showing an example of the functional configuration of a biological information processing device according to Embodiment 1. [Figure 4] This figure shows an example of a personal registration table. [Figure 5] This figure shows an example of an estimated consent information table. [Figure 6] This flowchart shows the process of registering information about individuals whose biometric data will be used. [Figure 7] It is a flowchart showing a process of using the registered personal biometric information. [Figure 8] It is a diagram showing an example of displaying AF / AE information on a display panel. [Figure 9] It is a diagram showing an example of a rule regarding the relationship between the use of biometric information and consent information. [Figure 10] It is a configuration diagram of an information processing system that uses biometric information. [Figure 11] In the information processing system according to Embodiment 2, it is a flowchart showing a series of processes of using the biometric information obtained by the imaging device 1 in external processing. [Figure 12] It is a flowchart showing the details of the biometric information use process.

Mode for Carrying Out the Invention

[0014] Hereinafter, each embodiment of the present invention will be described with reference to FIGS. 1A to 12.

[0015] 〔Embodiment 1〕 Hereinafter, Embodiment 1 according to the present invention will be described with reference to FIGS. 1A to 9. The biometric information processing apparatus according to this embodiment acquires information for specifying biometric information, and acquires an explicit consent indicating whether a person has consented or not to the use of information for specifying a person. Then, based on the situation at the time of acquiring the information for specifying a person, it estimates whether it was a situation where consent was presumed, and restricts the use of the information for specifying a person based on the acquired explicit consent and the presumed consent.

[0016] In the following, we will describe how the biometric information processing device in this embodiment uses or restricts the use of biometric information when performing biometric authentication using an image of a person captured by an imaging device (digital camera). While the biometric information processing device in this embodiment is described as a device built into the imaging device, it may also be a general information processing device such as a PC (personal computer) that processes images from the imaging device, or a server connected via a network.

[0017] First, using Figures 1A and 1B, we will explain the image acquisition process when using the bio-information processing device according to Embodiment 1. Figure 1A is a diagram illustrating the image acquisition process when using a biomedical information processing device (Part 1). Figure 1B is a diagram illustrating the image acquisition process when using a biomedical information processing device (part two).

[0018] The bio-information processing device according to this embodiment is incorporated into a part of the imaging device 1, whose external appearance is shown in Figure 1A, and processes bio-information contained in the image. The imaging device 1 is, for example, a digital camera. Figure 1B shows the imaging device 1 as viewed from the opposite side to the orientation in Figure 1A, i.e., from the user (photographer) side.

[0019] Here, subject 2 shown in Figure 1B is the first subject captured in Figure 1A, and is here defined as a person. Similarly, subject 3 is the second subject captured by imaging device 1, and is also defined as a person. In the following, a person who is a subject in an image may be referred to as "individual."

[0020] The external configuration related to the user interface of imaging device 1 will be explained below using Figure 1B.

[0021] The shutter button 11 is a button that, when pressed by the user (photographer) operating the imaging device 1, opens and closes the shutter and instructs the imaging device 1 to take a picture. The display panel 12 is a panel that displays the processing results to the user of the imaging device 1. The display panel 12 displays an image to check the composition or the appearance of the subject when taking a picture. The display panel 12 is not limited in type as long as it can display an image; for example, it may be an LCD panel, an OLED (Electro-Luminescence) panel, or a projector.

[0022] Operation key 13 is an operation key for obtaining input from the user. The type of operation key 13 is not particularly limited as long as it can obtain user input. For example, operation key 13 may be a hardware button provided by the imaging device 1, or if the display panel 12 is a touch panel, it may be a key that operates the key displayed on the display panel 12.

[0023] Next, we will explain the hardware configuration of the imaging device using Figure 2. Figure 2 is a hardware configuration diagram of the imaging device.

[0024] As shown in Figure 2, the imaging device 1 consists of an imaging unit 20, a biomedical information processing device 100, an operating device 14, a display device 15, and a communication interface 16.

[0025] The imaging unit 20 is a mechanism for capturing light from the outside and includes an image sensor 21 and an optical system 22. The optical system 22 is a mechanism that uses physical phenomena such as refraction, reflection, and diffraction of light to create an image of an object or to focus light, and has the function of forming an image of the subject on the image sensor 21. The optical system 22 is composed of multiple lens groups (not shown) and an aperture (not shown), etc.

[0026] The operating device 14 includes the shutter button 11 and operation keys 13 described earlier, and is a device installed for the user to operate the imaging device 1. The display device 15 is a device that displays information when the user takes a picture with the imaging device 1 or makes necessary settings, and the display panel 12 described earlier is one example of this.

[0027] The bio-information processing device 100 is a device that performs information processing for processing captured images containing bio-information. The bio-information processing device 100 is configured such that a CPU 111, a DSP 112, a main memory 113, and a non-volatile memory 114 are connected by a bus, as shown in Figure 2, for example. The CPU (Central Processing Unit) 111 is a processor that references data on the main memory 113 and executes programs. The main memory 113 is a high-speed volatile semiconductor memory device that holds data and programs accessed by the CPU 111. The DSP 112 is a processor that converts image signals into digital image data. The non-volatile memory 114 is a non-volatile semiconductor memory device such as flash memory that stores data and programs.

[0028] In this embodiment, the non-volatile memory 114 stores image data 60, image data 61 describing data related to the image data, a personal registration table 50, and an estimated consent information table 51. Details of the image data 61, personal registration table 50, and estimated consent information table 51 will be described later.

[0029] Furthermore, as shown in Figure 2, the non-volatile memory 114 has programs installed that implement various functions (biometric information acquisition program 41 to device authentication program 48). Each of these functions will be explained later.

[0030] Communication I / F16 is an interface for communicating with external devices. Communication I / F16 may communicate via wired communication using USB or similar methods, or via wireless communication using a local area network or as defined in the IEEE 802.11 standard.

[0031] Next, the functional configuration of the biological information processing device according to Embodiment 1 will be described using Figure 3. Figure 3 is a block diagram showing an example of the functional configuration of a biological information processing device according to Embodiment 1.

[0032] As shown in Figure 3, the biometric information processing device consists of the following functional units: a biometric information acquisition unit 31, a consent status estimation unit 32, a person identification unit 33, an explicit consent acquisition unit 34, a consent information generation unit 35, a biometric information utilization unit 36, a biometric information utilization restriction unit 37, and a device authentication unit 38.

[0033] The biological information acquisition unit 31 is a functional unit that acquires biological information. In this case, the biological information acquisition unit 31 acquires biological information from captured images in which the subject is shown.

[0034] The consent status estimation unit 32 estimates the consent status of the person identified by the person identification unit 33, based on the biometric information and the circumstances under which the biometric information was acquired, and outputs estimated consent information as the estimation result. Specific examples of how the consent status is estimated will be explained in detail later.

[0035] The person identification unit 33 extracts features from the input biometric information to identify the person. The extracted features are linked to the biometric information and consent information and stored in the non-volatile memory 114. The person identification unit 33 can also record the person's name, entered from the operating device 14, in the non-volatile memory 114, linked to the features and consent information. The person identification unit 33 may also have a function to identify who the person is by comparing it with pre-registered features. Furthermore, the process of extracting features from biometric information performed by the person identification unit 33 may also have a function to control whether or not to extract features based on consent to the use of biometric information, i.e., to restrict the use of the imaging device.

[0036] The explicit consent acquisition unit 34 obtains explicit consent information from the person possessing the biometric information, indicating explicit consent regarding the use of the acquired biometric information. Here, the explicit consent acquisition unit 34 can obtain explicit consent information based on input via the operation key 13. In the following, when simply referred to as "consent," it refers to consent regarding the use of the biometric information and the feature quantities extracted from the biometric information as described above. Furthermore, here, biometric information is explained as the entire captured image containing the subject, but biometric information is not particularly limited as long as it is information within the image that includes information about the subject. For example, the image of the bounding box in which the subject was detected, or the image of the bounding box in which the face of the subject was detected, may be considered biometric information.

[0037] The consent information generation unit 35 combines the explicit consent information acquired by the explicit consent acquisition unit 34 and the estimated consent information output by the consent state estimation unit 32 to generate consent information indicating the degree of explicit consent of the subject regarding the use of the imaging device. The generated consent information is linked to the corresponding biometric information and stored in the main memory 113 or non-volatile memory 114. In this embodiment, the biometric information is image information of a person's face, and the consent information is linked to and stored in the rectangular coordinate information indicating the position of the face.

[0038] The biometric information utilization unit 36 ​​utilizes biometric information and feature quantities based on the consent information generated by the consent information generation unit 35. The specific method of utilization will be described later.

[0039] The biometric information usage restriction unit 37 restricts the use of biometric information by the biometric information usage unit 36 ​​based on the consent information generated by the consent information generation unit 35. The specific restriction method will be described later.

[0040] The device authentication unit 38 performs authentication with external devices that utilize biometric information. Upon successful authentication, the biometric information processing device 100 can send and receive data and processing requests with the external device.

[0041] Next, the data structure used in the bio-information processing device of this embodiment will be described using Figures 4 and 5. Figure 4 shows an example of a personal registration table. Figure 5 shows an example of an estimated consent information table.

[0042] As shown in Figure 4, the personal registration table 50 consists of the following fields: personal ID 50a, personal name 50b, explicit consent 50c, facial image 50d, and feature quantity 50e.

[0043] The personal ID 50a stores an ID that uniquely identifies the individual. The personal name 50b stores a string representing the individual's name, if the individual's name is known. The explicit consent 50c stores a flag indicating whether or not explicit consent has been given by the corresponding individual to use their biometric information. For example, if the value of explicit consent 50c is "yes," it means that consent has been given, and if the value is "no," it means that consent has not been obtained. The default value is "no." The face image 50d stores the face image data of the corresponding individual (for example, it may be the image ID and rectangular coordinate values, or it may be specific image data). The feature vector 50e stores the biometric information feature vector calculated by a predetermined algorithm.

[0044] In the example shown in Figure 4, three individuals are registered with the names "Tanaka ○△o," "●yama □ko," and "Suzu◎▽o." Figure 4 shows that explicit consent for the use of biometric information has been obtained for "Suzu◎▽o," and that his features have been extracted and registered. On the other hand, explicit consent for the use of biometric information has not been obtained for either "Tanaka ○△o" or "●yama □ko," and therefore, their features have not been registered.

[0045] The estimated consent information table 51 is a table that stores the results of whether or not an estimated consent was given for an individual registered in the individual registration table 50. As shown in Figure 5, the estimated consent information table 51 consists of the fields for individual ID 51a and estimated consent 51b.

[0046] Personal ID 51a stores an ID that uniquely identifies the individual, similar to Personal ID 50a in Figure 2. Presumed consent 51b stores a flag indicating whether or not presumed consent was given by the corresponding individual for the use of their biometric information. For example, if the value of presumed consent 51b is "yes," it means that presumed consent was given, and if the value is "no," it means that presumed consent was denied. Here, presumed consent means that, based on the circumstances of the shooting and the captured image, it is presumed that the individual in question gave consent for the use of their biometric information. The specific process for determining whether or not presumed consent was given will be explained later.

[0047] Next, we will explain the process of using biological information by a biological information processing device with reference to Figures 6 to 9. Figure 6 is a flowchart showing the process of registering information about an individual whose biometric data is used. Figure 7 is a flowchart showing the process of using the biometric information of a registered individual.

[0048] Hereinafter, the processing using biometric information by the biometric information processing device 100 according to this embodiment can be divided into two parts: the process of registering information about the individual whose biometric information will be used, and the process of using the biometric information of the registered individual.

[0049] Furthermore, in the biometric information processing device 100 according to this embodiment, when registering information about an individual, it extracts features related to biometric information only when explicit consent has been obtained from the individual. On the other hand, when using the registered biometric information of an individual, it determines whether or not there is presumed consent from the individual based on the shooting circumstances and images, and if there is presumed consent, it relaxes the restrictions on the use of biometric information even if there is no explicit consent from the individual.

[0050] First, we will explain the process of registering information about individuals whose biometric data will be used, using Figure 6.

[0051] The process of registering information about an individual whose biometric data is used is initiated, for example, when a user of the imaging device 1 presses an operation key 13 as shown in Figure 1B, thereby instructing the device to perform the registration process. In this embodiment, the user is defined as a user who takes a photograph of a subject using the imaging device 1, such as a digital camera that includes the biometric information processing device 100.

[0052] First, the biometric information processing device 100 performs initialization processing for the registration process of personal information (S101). Here, the CPU 111 of the biometric information processing device 100 reads a program from the non-volatile memory 114 in Figure 2 and makes the following registration process operational. The process shown in the following flowchart is realized by the CPU 111 of the biometric information processing device 100 executing the necessary program shown in Figure 2.

[0053] Next, the biological information acquisition unit 31 of the biological information processing device 100 acquires biological information (in this case, an image of a person's face taken from the imaging device 1 in Figures 1A and 1B) (S102). At this time, the biological information acquisition unit 31 controls the imaging device 1 using known imaging techniques so that the focus and exposure are set to match the target face, and acquires the captured image. Hereinafter, the following explanation assumes that there is only one subject in the captured image which will become the biological information, but there may be multiple subjects in the captured image, and the processing described below may be performed individually for each of them.

[0054] Next, the explicit consent acquisition unit 34 of the biometric information processing device 100 acquires explicit consent information for the individual (subject) captured in the image acquired in S102 (S103). As mentioned above, the explicit consent information acquired here represents the image of the individual's face, that is, whether or not consent has been given to the use of their biometric information.

[0055] "Use of biometric information" as used here includes at least one of the following: capturing and recording a facial image, extracting features from the facial image, registering the features, performing authentication based on the features (in this case, the process of identifying a person), and performing control using the results of the authentication process. Details of these "uses of biometric information" will be described later. The biometric information processing device 100 according to this embodiment restricts the use of this biometric information based on explicit consent information and presumptive consent information, which will be described later.

[0056] The explicit consent acquisition unit 34 according to this embodiment can acquire explicit consent information based, for example, on user operations via the display panel 12 or operation keys 13. Specifically, the explicit consent acquisition unit 34 can display on the display panel 12 a statement indicating that biometric information will be used and a statement accepting a choice of whether or not to consent to this, and acquire the result of the choice as explicit consent information.

[0057] Here, the user's name is assumed to be pre-configured and associated with a unique ID (for example, based on user login or user input during operation). However, in order to prevent so-called "impersonation," where someone other than the user performs an operation with unauthorized consent, the user may be configured based on an image captured by a different source than biometric information. For example, an imaging device (not shown) that captures the operator of operation key 13 may be provided, and the user may be configured based on the image captured by such an imaging device (by a known person identification process). In such a case, an additional determination may be made as to whether the configured user and the person in the biometric information acquired in S102 are the same person, and if they are not the same person, the process in Figure 6 may be terminated at that point.

[0058] Furthermore, although this explanation assumes that explicit consent information is obtained based on user input via the operation key 13, the process is not limited to this specific method as long as it is possible to obtain whether or not the user has given consent. For example, if it is detected that the user has performed a predetermined action (e.g., uttering a consent voice or making a consent gesture), the system may determine that the user has consented to the use of biometric information and proceed with imaging by the imaging device 1 and processing the use of biometric information (e.g., authentication processing). Such processing allows the user to indicate their consent simply by making a gesture, thereby improving convenience. The gesture can be, for example, a peace sign made with the hand, or raising or lowering the hand, and the gesture can be recognized using known motion recognition technology.

[0059] Next, the explicit consent acquisition unit 34 of the biometric information processing device 100 determines whether the user has consented to the use of biometric information based on the explicit consent information acquired in S103 (S104). If consent has been given (S104: Yes), the process proceeds to S105; otherwise (S104: No), the process proceeds to S107.

[0060] Next, the person identification unit 33 of the biometric information processing device 100 extracts feature quantities from the biometric information of a user who has consented to the use of their biometric information (S105). As mentioned above, the biometric information is an image of an individual's face, and the person identification unit 33 identifies the position of the face in the image and then extracts feature quantities from that face. In the process of identifying the position of the face in the image and the process of extracting feature quantities from a person's face, any known image recognition technology can be used. For example, to identify the position of the face in the image, the technology described in Non-Patent Literature 1 may be used. Also, for example, to extract feature quantities from a person's face, a method using Deep Learning may be used, or methods such as LBP (Local Binary Pattern) or HoG (Histogram Oriented Gradient) may be used.

[0061] If explicit consent is given in S104, the person identification unit 33 of the biometric information processing device 100 registers the biometric information, extracted feature quantities, and unique identifier as "explicit consent given" and registers values ​​in each field of the personal registration table 50 shown in Figure 4 (S106). The person identification unit 33 can obtain the person's name by receiving input from the user via the operating device 14 in Figure 2. The format of the person's name in this embodiment is not particularly limited; for example, the name may be a real name, a nickname, a number, or a symbol. In this embodiment, the person identification unit 33 is described as registering the explicit consent information, but the processing may be performed by a different functional unit, such as the explicit consent acquisition unit 34 registering the explicit consent information.

[0062] If explicit consent is not given in S104, the person identification unit 33 of the biometric information processing device 100 records "no explicit consent" and links the biometric information with a unique identifier, registering values ​​in each field of the personal registration table 50 shown in Figure 4 (S107).

[0063] Next, the person identification unit 33 of the biometric information processing device 100 determines whether or not to terminate the registration process (S108). For example, if the user has entered an input indicating that they want to terminate the registration process, it may be determined that the registration process should be terminated. If it is determined that the registration process should be terminated (S108:Yes), the process is terminated; otherwise (S108:No), the process returns to S102.

[0064] Next, we will explain the process of using the registered personal biometric information with reference to Figure 7. The procedure shown in Figure 7 illustrates how a user of an imaging device 1, such as a digital camera, can capture an image of a subject using a technique to identify a person. For example, the user of the imaging device 1 can start the procedure shown in Figure 7 by half-pressing the shutter button 11 shown in Figure 1B. The actual shooting (recording of the captured image), which will be described later, is performed by pressing the shutter button further after it has been half-pressed. Furthermore, the biometric information-based processing (authentication processing) according to this embodiment is performed at least once after the registration processing for registering personal information, as shown in Figure 6 above, has been executed.

[0065] First, the biometric information processing device 100 performs initialization for processing that utilizes biometric information (S111). Here, the biometric information processing device 100 reads a program from the non-volatile memory 114 shown in Figure 2 and makes the program that executes the following procedure ready to run. Also, as part of the initialization process, the biometric information processing device 100 loads the registration dictionary from the non-volatile memory 114 and makes it available for use.

[0066] Next, the biometric information acquisition unit 31 of the biometric information processing device 100 acquires biometric information (in this case, an image of an individual's face from the imaging device 1 in Figure 1B). Here, the biometric information acquisition unit 31 acquires candidate images to be recorded as biometric information from the image sensor 21. The captured images acquired here are displayed on the display panel 12 as a Live View in the processing described later. The user can determine the composition of the image to be recorded or the timing of pressing the shutter button 11 while checking the Live View. Note that the captured images acquired here may contain multiple faces.

[0067] Next, the person identification unit 33 of the biometric information processing device 100 extracts features from the input image (biometric information) and identifies who the subject of the biometric information is by comparing them with the features stored in the personal registration table 50 (S113). Since it is necessary to identify the person in order to confirm whether or not explicit consent has been given, features are temporarily extracted from the biometric information that has been input without explicit consent and deleted when use is complete. If the image contains multiple faces, features are extracted from each of the faces. Specifically, the comparison of features involves calculating the similarity between the features, and if the similarity exceeds a predetermined threshold among the registered individuals, it is determined that the extracted person is that individual; if no person exceeds the threshold, it is determined that it is no one. Any known method can be arbitrarily used to calculate the similarity, and for example, cosine similarity may be used, or L2 distance may be used. Since L2 distance is a measure in which the value decreases as the distance between features becomes closer, the reciprocal of L2 distance may be converted to similarity and used. As explained in Figure 4, since "●yama□ko" does not have any registered features, matching cannot be performed for "●yama□ko".

[0068] Next, the consent status estimation unit 32 of the biometric information processing device 100 obtains the value of explicit consent 50c of the corresponding record for the identified person among the input biometric information and confirms whether or not consent has been given (S114). As mentioned above, the possible values ​​of explicit consent 50c are binary: consent given / no consent.

[0069] Next, the consent status estimation unit 32 of the biometric information processing device 100 refers to the explicit consent information obtained in S114 and estimates the consent status for subjects without explicit consent (S115). By doing so, the target of consent status estimation can be narrowed down to subjects without explicit consent, which leads to a reduction in processing load. Alternatively, consent status may be estimated for all subjects without narrowing down the target. The method for estimating the consent status will be described in detail below.

[0070] To estimate the subject's consent status, the acquisition status of biometric information is referred to, and if the acquisition status meets predetermined conditions, it is considered that there is presumed consent. Here, the acquisition status refers to the conditions of the device, the subject, and their surroundings at the time of biometric information acquisition. Specifically, in this embodiment, for example, if the position of the imaging device 1 is close to the subject, and the subject is facing forward with a smiling or expressionless face, it is determined that there is a state of consent that can be presumed, i.e., presumed consent.

[0071] The following details the conditions under which this consent is presumed.

[0072] Regarding the position of the imaging device, if the imaging device is close to the subject, it can be assumed that the subject is aware of being photographed, rather than being secretly photographed from a distance. Therefore, this can be used as a basis for determining presumed consent. In other words, the positional relationship between the imaging device and the subject is the condition. Specifically, the distance between the subject and the imaging device is measured, and if it is less than a predetermined value, the condition is deemed to be met. The distance between the imaging device and the subject is measured using a distance sensor mounted on the imaging device. Alternatively, it may be calculated from the relationship between known lens and sensor information and the size of the subject in the captured image.

[0073] Regarding the subject's state, if the subject is facing the imaging device or making a specific facial expression or gesture, it can be presumed that the subject is aware of being photographed, and this can serve as material for determining presumed consent. Specifically, existing image recognition technology is used to detect the subject's face orientation and facial expression from the image, and it is determined that the conditions are met if the face is facing forward (i.e., facing the imaging device 1) and the facial expression is either a smile or a neutral expression. Here, the conditions are a combination of face orientation and facial expression, but these can be used individually, or combined with other conditions such as gestures. Furthermore, if a predetermined gesture (such as an OK sign) is set in the biometric information processing device 100 in advance, and presumed consent is determined when that gesture is detected, although the conditions under which presumed consent can be determined are limited, the possibility of incorrectly determining presumed consent can be reduced.

[0074] If both of the above two conditions are met, the consent status estimation unit 32 determines that consent has been estimated and stores the value "yes" in the estimated consent 51b of the corresponding record in the estimated consent information table 51 shown in Figure 5.

[0075] Next, the consent information generation unit 35 of the biometric information processing device 100 generates consent information (S116) based on the value of explicit consent obtained in S114 and the value of estimated consent estimated in S115. The consent information can take three values: "no consent", "estimated consent", and "explicit consent". The value of the consent information is set to "explicit consent" if the value of explicit consent is "yes" (explicit consent from the subject is given). Also, if the value of explicit consent is "no" (there is no explicit consent from the subject) and the value of estimated consent is "yes" (the subject's consent is estimated), the value is set to "estimated consent". Also, if the value of explicit consent is "no" (there is no explicit consent from the subject) and the value of estimated consent is "no" (the subject's consent is not estimated), the value is set to "no consent". Setting it in this way means that the degree of explicit consent regarding the subject's biometric information and the use of biometric information is increasing in the order of "no consent", "estimated consent", and "explicit consent" (i.e., "no consent" < "estimated consent" < "explicit consent").

[0076] Next, the biometric information utilization unit 36 ​​of the biometric information processing device 100 utilizes the biometric information under the restrictions imposed by the biometric information utilization restriction unit 37 (S117). Here, utilizing the biometric information means, for example, utilizing the feature quantities extracted in S113, and using them to control the imaging parameters of the imaging device 1 (here, autofocus (AF) control and auto exposure (AE) control). For example, the biometric information utilization restriction unit 37 can control the imaging parameters so that focus and exposure are controlled for the faces of subjects whose names have been identified. That is, the biometric information utilization restriction unit 37 controls the system so that individuals whose names have not been identified (who have not explicitly consented to the use of their biometric information) are not referenced in the control of the imaging parameters (they are not focused on).

[0077] However, in this case, for individuals whose names could not be identified but who were determined to have given "presumed consent," the imaging parameters that control focus and exposure will be controlled (focusing) in the same way as when explicit consent is given.

[0078] Controlling focus and exposure for specific individuals can be achieved using known AF / AE techniques. Furthermore, if multiple individuals with identified names are present in the captured image, it is possible to pre-specify which of them should be focused on and exposed (for example, by setting a priority for each individual).

[0079] Next, the biological information utilization unit 36 ​​of the biological information processing device 100 displays AF / AE information on the display panel 12 as a result of utilizing the biological information (S117).

[0080] The following example of displaying AF / AE information on the display panel will be explained using Figure 8. Figure 8 shows an example of displaying AF / AE information on the display panel. In Figure 8, the face frames G101 of a person labeled "Unknown", G102 of a person labeled "Suzu◎▽o", and G103 of a person labeled "●yama□ko" are placed. In addition, for G102 and G103, the consent information values ​​"Explicit Consent" and "Presumed Consent" are displayed together.

[0081] Face frame G101 relates to "Tada ○△o," but since explicit consent for the use of biometric information was not given, and it was determined that there was no presumed consent, the person could not be identified. Face frame G102 relates to "Suzu ◎▽o," and as mentioned above, "Suzu ◎▽o" has explicit consent for the use of biometric information, so it was determined to be "Suzu ◎▽o" by matching it with the features of "Suzu ◎▽o" in personal registration table 50. Face frame G103 is "●yama □ko," and although there was no explicit consent, it was determined that there was presumed consent, which means that person identification using features was performed, and the results were used for the aforementioned imaging parameter control and the display shown in Figure 8.

[0082] In this way, the use of biometric information can be restricted by preventing the identification of individuals without explicit consent and the use of the results thereof. Furthermore, even in cases where explicit consent is absent, biometric information can be used based on the determination of presumed consent. In other words, by presuming consent regarding the subject, the restrictions on the use of biometric information that are imposed when explicit consent is absent are mitigated. By doing so, biometric information can be used without the need to obtain cumbersome explicit consent when explicit consent is absent, thereby improving user convenience.

[0083] The display shown in Figure 8 is the live view display in a digital camera, and the user can decide whether or not to record an image by pressing the shutter button 11 while looking at this display.

[0084] Next, the biometric information utilization unit 36 ​​records biometric information under the restrictions of the biometric information utilization restriction unit 37 (S118). Here, the biometric information is the facial image in which the image appears, and the facial image is recorded in the non-volatile memory 114 of the biometric information processing device 100 so that the user can refer to it later. Here, the biometric information utilization unit 36 ​​captures an image when the user presses the shutter button 11, and records the consent information as image data 61 of the person's image to the captured image. At this time, it is controlled so that it can be recorded regardless of the consent information (i.e., even without consent). By doing so, situations where a photo opportunity is missed due to a lack of consent can be avoided, and convenience can be improved. Also, since the recording is done within the device and not output to other devices, it is easier to obtain understanding from the subject.

[0085] For example, when recording the image shown in Figure 8, as shown in the figure, the coordinate values ​​of the rectangular frame of face frame G102, the person's name "Suzu◎▽o", and the consent information "Explicit consent" are recorded as image data 61 of the image. Similarly, information regarding face frame G103 is also recorded as image data 61. For face frame G101, "Unknown" is recorded as image data 61 to indicate that the person's name has not been identified.

[0086] Finally, the biometric information utilization unit 36 ​​of the biometric information processing device 100 determines whether or not to terminate the process of utilizing biometric information (S119). For example, if the user has given input to terminate the process of utilizing biometric information, it may be determined to terminate that process. If it is determined in S119 to terminate the process of utilizing biometric information (S119:Yes), the process is terminated; otherwise (S119:No), the process returns to S112.

[0087] Next, using Figure 9, we will explain an example of a rule regarding the relationship between the use of biometric information and consent information. Figure 9 shows an example of rules regarding the relationship between the use of biometric information and consent information.

[0088] The rules shown in Figure 9 indicate whether the use of biometric information is permitted when the consent information value is "no consent," "presumed consent," or "explicit consent," respectively, for the content shown in the description of the manner in which biometric information is used. In other words, "OK" indicates permission to use biometric information, and "NG" indicates restriction (cannot be performed) on the use of biometric information.

[0089] Numbers 1-3 illustrate the limitations described in this embodiment, and all are processes performed within the imaging device 1. Number 1 represents the "shooting (in-camera recording of captured images)" process, and as mentioned above, recording is permitted regardless of the consent information value. Number 2 is the registration of feature quantities, and as mentioned above, it is restricted if the consent information value is not "explicit consent" (when it is "no consent" or "presumed consent"). Number 3 is permission to use person identification, i.e., face recognition processing, for AF / AE during shooting, and as mentioned above, it is permitted only for consent information values ​​of "presumed consent" and "explicit consent". In this way, the degree of relaxation of restrictions is increased depending on the explicitness of consent indicated by the consent information. That is, in the conventional technology, all processing was restricted if there was no explicit consent, whereas in this embodiment, if there is presumed consent, it is permitted to use the results of person identification during shooting, and the restrictions are relaxed.

[0090] No. 4 and beyond will be explained in Embodiment 2.

[0091] As explained above, the processing of the biometric information processing device of this embodiment allows for the use of biometric information by restricting its use for individuals who have not given explicit consent, while relaxing the restrictions based on presumed consent estimated from the circumstances under which the biometric information was acquired. In particular, by controlling permission and restriction for each process according to the degree of explicit consent stated in the consent information, it is possible to achieve a certain degree of balance between privacy protection and convenience.

[0092] [Experimental Analysis 1-1] In the above embodiment 1, an example was described in which two biological information acquisition conditions, the position of the imaging device and the state of the subject, were used as determination conditions to determine presumed consent. However, other determination conditions may be used as long as the subject's consent can be presumed.

[0093] For example, other criteria for determination may include consent information from people other than the subject, such as people appearing in the same image as the subject. If a person nearby has given explicit consent, it can be determined that the person next to them is likely a friend or acquaintance and has given similar consent, thus providing material for determining presumed consent. Whether or not someone is nearby can be determined, for example, by detecting a person's face using existing face detection technology, and if the distance of the detected face in the image is shorter than a predetermined distance, it can be determined that the person is nearby. As mentioned above, consent information is recorded as metadata linked to the position of each face in the image, so this can be referenced. Alternatively, known scene estimation techniques can be used to determine whether or not a person nearby is an acquaintance based on the arrangement of people in the captured image, and their consent information can be used. Furthermore, not only a single captured image but also previously captured images can be used to determine whether or not the person is an acquaintance.

[0094] Other criteria for determining consent include the fact that if there was presumed consent in previously captured images of the subject, the likelihood of presumed consent now increases, and this can serve as material for determining presumed consent. If there is even one image of presumed consent from the past, the condition may be considered met. Alternatively, if there is more than one image of a predetermined number, the condition may be considered met.

[0095] Other criteria for determination include the type of imaging device, i.e., the hardware configuration of the device incorporating the means for acquiring biometric information. If the lens equipped on the imaging device is not a telephoto lens, but rather a wide-angle lens or similar lens that requires close-up shooting to capture a large image of the face, then the subject is more likely to be aware of being photographed, and this can be used as material for determining presumed consent. A method for determining the type of lens is to record information identifying the lens type on the lens, and when connected to a camera, the lens and camera communicate, allowing the camera (i.e., the imaging device) to acquire the lens type information from the lens. The information identifying the lens type may be a specific product-specific number or information about the lens type, such as wide-angle or telephoto.

[0096] Furthermore, other criteria for judgment may include shooting parameters, i.e., the parameters acquired when obtaining biometric information. For example, if the ISO sensitivity during shooting is low, it is likely that the shooting took place in a bright place, increasing the likelihood that the subject was aware of being photographed. When the subject is aware of being photographed, it is presumed that the subject has given consent for the use of their biometric information. Also, if the camera's focal length is short, the subject must be physically close to be photographed, which increases the likelihood that the subject was aware of being photographed. In addition, if the shutter speed is slow, the time it takes to take a picture is longer (compared to when the shutter speed is fast), making it easier for the subject to recognize that they are being photographed. These shooting parameters can be set as criteria for judgment by specifying conditions such as being above or below a certain value.

[0097] Other criteria for determining consent may include the location where imaging (the device is used). If the location is a specific event venue, it can be assumed that the subject is aware to some extent that they are being photographed, and this can be used as a basis for determining presumed consent. Information on the shooting location (location where the imaging device is operated) can be specified by equipping the biometric information processing device with a GPS (Global Positioning System) sensor, recording map information, and designating a predetermined location as an event venue, or by having the user of the imaging device 1, which incorporates the biometric information processing device 100, register their home address. This allows for the determination of whether or not the location is an event venue using GPS at the time of shooting. Furthermore, in the case of use like a surveillance camera with a fixed position, there is a possibility that the subject is not being photographed as they wish, so if the location of the imaging device moves over a certain period of time, this can also be used as a basis for determining presumed consent. In other words, the method of fixing (installation method) of the imaging device can also be used as a basis for determination.

[0098] In addition to the subject's orientation, facial expression, and gestures mentioned in this embodiment, other factors such as whether the subject has previously given explicit consent may also be used. If the subject has given explicit consent in the past, there is a high probability that they will continue to give consent, so this can be used as material for determining presumed consent. To do this, a time limit for using the explicit consent can be set, and if information on expired explicit consent is linked to the subject in question, past consent records can be checked and referenced.

[0099] Furthermore, another condition regarding the subject's state may be whether or not the subject is taking a selfie. If it is a selfie, it is clear that the subject has given their consent, and therefore it may be used as material for determining presumed consent. As a method for determining whether or not it is a selfie, the biometric information processing device 100 may be equipped with a gyro sensor and the determination may be made based on the tilt of the imaging device, or the selfie may be determined from the composition of the captured image, such as the size of the face area in the captured image, or both may be combined.

[0100] In addition, in Embodiment 1, presumed consent was determined when both conditions of the imaging device's position and the subject's state were met, but other conditions may also be used. It may be determined when only one of the conditions is met, or multiple conditions for other acquisition situations may be prepared, and presumed consent may be determined when a predetermined number of conditions are met.

[0101] You could set a numerical importance level for each condition and add to that level if the condition is met. This would allow for an estimation of the consent state that prioritizes the conditions that are important for determining presumed consent.

[0102] Furthermore, for subjects who are currently being photographed or have been photographed and whose images are determined to have presumed consent, the system may retroactively apply this determination to previously photographed images as well. By doing so, previously taken images can be processed under the presumed consent condition without obtaining explicit consent, thereby improving convenience.

[0103] [Variation 1-2] In this embodiment, when registering an individual's biometric information, only explicit consent was used to restrict the extraction of biometric feature quantities. However, estimated consent may also be used during registration. Specifically, even if explicit consent is not given during the registration process of an individual's biometric information, the consent status may be estimated and biometric feature quantities may be registered. By doing so, even if explicit consent is not given during the registration of an individual's biometric information, the feature quantities can be used, thereby improving convenience.

[0104] [Examples 1-3] In Embodiment 1, the estimated consent information was shown as a binary value indicating whether or not there was estimated consent from the biometric information provider, but multiple values ​​may be used. For example, the number of conditions that are met among the conditions considered to be estimated consent may be used as the estimated consent value. This allows for an indication of the accuracy of the estimated consent. Depending on the accuracy of the estimate, processing such as switching between restricting or permitting processes may be implemented. Alternatively, a predetermined calculation formula may be used to calculate a real value for multiple conditions.

[0105] Furthermore, in this embodiment, the consent information takes three values, but other values ​​may also be used, for example, real numbers. To calculate real numbers, a predetermined calculation formula can be prepared that takes the explicit consent information and the inferred consent information as inputs, and the calculation can be performed using that formula.

[0106] [Variations 1-4] In this embodiment, the consent of the subject is obtained by displaying a message on the display panel 12 of the imaging device 1 indicating that biometric information will be used and a message (an image prompting consent) allowing the user to choose whether or not to consent. However, the method of obtaining consent information is not particularly limited as long as it is possible to confirm whether or not the user has given consent. For example, the biometric information processing device 100 may obtain consent information based on input on an application on another mobile terminal such as a smartphone. In such a case, the biometric information processing device 100 can communicate with the mobile terminal and, when obtaining consent information (S103 in Figure 6), request the mobile terminal to send the consent information. The mobile terminal that receives the request displays an image prompting consent on its screen, receives input from the user of the mobile terminal indicating consent, obtains the consent information, and transmits it to the biometric information processing device 100. At this time, in order to detect impersonation and prevent consent from being given by another person, the biometric information processing device may confirm that the person being registered and the user of the mobile terminal are the same person (perform identity authentication). For this purpose, the user of the mobile terminal may perform identity authentication on the information processing device 1 side, or may perform identity authentication using the security function of the mobile terminal. For example, the biometric information processing device 100 transmits a facial image of a registered person to a mobile terminal, and the mobile terminal can verify whether the user and the facial image belong to the same person using known facial recognition technology. The facial image of the mobile terminal user is captured, for example, by the mobile terminal's front camera. With this configuration, consent obtained through impersonation can be avoided, and consent information can be obtained from the subject. Note that the authentication of the person is not limited to such facial recognition technology, and may also be any authentication process using, for example, fingerprint authentication provided by the mobile terminal, or authentication process using an electronic certificate.

[0107] [Embodiment 2] Embodiment 2 will be described below with reference to Figures 10 and 11.

[0108] In Embodiment 1, an example was described in which biological information in an image taken by an imaging device is used and processed internally by the biological information processing device 1 based on consent information. In this embodiment, an example is described in which biological information is output to an external device and processed by the external device using the biological information.

[0109] First, we will explain the configuration of an information processing system that utilizes biological information using Figure 10. Figure 10 is a diagram illustrating the configuration of an information processing system that utilizes biological information.

[0110] As shown in Figure 10, the information processing system of this embodiment is configured such that an imaging device 1, an imaging device 1a, a mobile terminal 7, and an image management server 8 are connected by a network 5. Each device can receive and send / receive requests and data from other devices.

[0111] The imaging device 1 has the configuration shown in Figure 2 of Embodiment 1 and incorporates a bio-information processing device 100 inside, and is an imaging device like a digital camera.

[0112] Network 5 is a network for each device to communicate with one another. Network 5 may be a local area network (LAN), for example, or a global network, such as the Internet. Furthermore, the connection method for Network 5 may be wireless communication or wired communication.

[0113] The imaging device 1a is a second imaging device in the information processing system, and is a device such as a digital camera, having the same hardware and functional configuration as imaging device 1.

[0114] Mobile device 7 is a portable information processing device, such as a smartphone or tablet.

[0115] The image management server 8 is a server that stores and manages images from the imaging device 1 and provides image-related data and information to other devices. The imaging device 1a, the mobile terminal 7, and the image management server 8 function as external devices in this embodiment.

[0116] In this embodiment, we will first describe an example in which the imaging device 1 outputs biometric information, feature quantities, and consent information to the image management server 8 based on the consent information, and the image management server 8 processes them.

[0117] Image management server 8 provides image management services, including storing images sent from other devices for each user, and providing various services such as sharing and managing images in response to requests from other devices.

[0118] Next, using Figures 11 and 12, we will explain a series of processes in which the biological information obtained by the imaging device 1 is used in external processing in the information processing system shown in Figure 10. Figure 11 is a flowchart showing a series of processes in which biological information obtained by the imaging device 1 is used in external processing in the information processing system according to Embodiment 2. Figure 12 is a flowchart detailing the processing of biometric information.

[0119] In the process shown in Figure 11, the process begins when the user of the imaging device 1 issues an instruction to output biological information, feature quantities, and consent information to an external source by operating the imaging device 1.

[0120] First, the device authentication unit 38 of the biometric information processing device 100 shown in Figure 3 of the embodiment identifies the device that outputs biometric information and feature quantities (S151). Specifically, the biometric information processing device 100 built into the imaging device 1 records in advance the services on which the user or seller of the imaging device 1 has authorized the processing of biometric information and feature quantities, and the devices that provide those services, and the user is allowed to select from among them. This identifies the external device that outputs biometric information and feature quantities. By allowing the user to select in this way, it is possible to specify what kind of processing the external device will perform on the biometric information and feature quantities. In this embodiment, it is assumed that the image classification process using feature quantities, which will be described later, is specified, and the image management server 8 is selected as the external device that will perform that processing.

[0121] Next, the device authentication unit 38 of the biometric information processing device 100 performs authentication processing with the external device identified in step S151, namely the image management server 8, via the network 5 (S152). Specifically, the device authentication unit 38 sends a request to the image management server 8 for permission to communicate data such as biometric information and feature quantities. At this time, the device authentication unit 38 outputs a device identifier that can identify the imaging device 1 to the image management server 8, enabling the image management server 8 to authenticate. The device identifier is an identifier used to identify the device, and it may be prepared in-house or an existing one may be used. For example, information such as the device's serial number or MAC address may be used.

[0122] Next, the image management server 8 performs device authentication based on the acquired device identifier (S153). Specifically, the image management server 8 authorizes the communication of data and requests between the biometric information processing device 1 built into the imaging device 1 and the image management server 8. The authorization mechanism involves the image management server 8 pre-maintaining a list of authorized device identifiers, and granting authorization if the acquired device identifier is included in the list.

[0123] By performing device authentication in this manner, the biometric information processing device 100 built into the imaging device 1 can output biometric information and feature quantities to the services and devices it anticipates and specifies. In other words, it is possible to prevent output to unintended services and devices, thereby preventing the unintended use of biometric information.

[0124] Next, the image management server 8 determines in step S153 whether device authentication was successful (S154). If device authentication is successful (S154:Yes), it proceeds to S155. If device authentication is unsuccessful (S154:No), it terminates the process.

[0125] When device authentication is successful, the image management server 8 creates authentication data and transmits it via the network 5 to the device authentication unit 38 of the biometric information processing device 100 built into the imaging device 1 (S155). The authentication data is data indicating permission to receive biometric information and its feature quantities from the biometric information processing device 100 built into the imaging device 1.

[0126] Next, the biometric information utilization unit 36 ​​of the biometric information processing device 100 transmits the values ​​of the personal registration table 50 shown in Figure 4 to the image management server 8 (S156), assuming that the device authentication unit 38 has received authentication data. The personal registration table 50 is a table that associates a person's name, face image and feature quantities, and explicit consent, as described in Embodiment 1. Here, the biometric information consists of numerous face images, including the face of the subject. When transmitting, the biometric information utilization unit 36 ​​also transmits the authentication data obtained from the device authentication unit 38 to the image management server 8. By checking the authentication data, the image management server 8 authorizes the receipt of the explicit consent value in the personal registration table 50 and the captured images.

[0127] Next, the image management server 8 receives the values ​​from the personal registration table 50 and stores them in an external storage device (not shown) such as an HDD or SSD of the image management server 8 (S157).

[0128] Next, the biometric information utilization unit 36 ​​of the biometric information processing device 100 requests utilization processing for the biometric information received by the image management server 8 in step S157 (S158). As mentioned above, the biometric information processing device 100 has available services recorded in advance and requests the execution of those services. Multiple services may be provided, and one or more of them may be selected.

[0129] Next, the image management server 8 performs the processing of biometric information requested by the biometric information processing device 100 in step S158 (S200). In this embodiment, for example, a feature-based image classification service (identification of an individual using feature quantities) is performed. At this time, the imaging device 1 transmits the image to be classified. Details of the biometric information processing will be explained later with reference to Figure 12.

[0130] Next, the biometric information utilization unit 36 ​​of the biometric information processing device 100 acquires and displays the processing result of S200, i.e., metadata identifying a person, from the image management server 8 (S159). Here, a list of all people present in the metadata is displayed along with their facial images, and the user is allowed to select which person's image they want to display. The image of the selected person is then displayed. This method enhances convenience by allowing users to display images of only specific people, or to perform tasks such as processing, editing, or deleting images.

[0131] Next, we will explain the details of the processing of biological information using Figure 12.

[0132] This corresponds to process S200 in Figure 11.

[0133] In the biometric information utilization process, the image management server 8 searches for a person who matches the biometric information (face image) stored in S157 from the image transmitted in S158 (S201). By using face detection technology or face recognition technology as described in Embodiment 1, it identifies who is in which image. The information of the identified person is recorded as metadata associated with the image.

[0134] Next, the image management server 8 estimates the consent status of the subjects in the target image (S115a). This process is the same as the process in S115 in Figure 7 of Embodiment 1.

[0135] Next, the image management server 8 generates consent information based on the consent status estimated in S115a (S116a). This process is the same as the process in S116 in Figure 7 of Embodiment 1.

[0136] The consent information can take on three values: "no consent," "presumed consent," and "explicit consent," and the method for setting these values ​​is the same as in Embodiment 1. Next, the image management server 8 extracts the individuals to be identified based on the consent information in S116a (S202). At this time, the image management server 8 limits the processing targets to only those individuals with explicit consent in the personal registration table 50 and those whose consent information from the captured image is either "estimated consent" or "explicit consent". In this case, since only the features of individuals with explicit consent can be used from the features of the personal registration table 50 (rule No. 2 in Figure 9), image classification is performed only for individuals with explicit consent. Individuals with estimated consent are not in the registration dictionary, so their images are classified together as "other individuals". In this way, processing by authenticated external devices can be restricted according to consent information. Many people feel uneasy about the processing of biometric information by external devices, so by excluding individuals without explicit consent from processing, it is possible to balance privacy protection and convenience to a certain extent.

[0137] Next, referring again to Figure 9, we will describe an example of a rule regarding the relationship between the use of biometric information and consent information related to Embodiment 2.

[0138] Here, we will explain the relationship between the handling of biological information by external devices and consent information, as in this embodiment, using sections No. 4 to No. 6.

[0139] No. 5 is the processing of output to a designated external device, i.e., an external device that has performed authentication processing, and is permitted regardless of the value of the consent information (even if it is "no consent"). No. 6 is the processing of biometric information by a designated external device, and is permitted when the value of the consent information is "presumed consent" or "explicit consent". In this way, the degree of relaxation of restrictions is increased depending on the degree of explicit consent indicated by the consent information. In other words, in the conventional technology, all processing was restricted unless there was explicit consent from the provider of biometric information, whereas in this embodiment, if consent can be presumed from the provider of biometric information, processing by the external device is permitted, and restrictions are relaxed.

[0140] Furthermore, No. 4 represents output to an external device or service not specified by the information processing device, i.e., an unexpected one, and only "explicit consent" is permitted for the consent information value. This restricts the use of biometric information in unexpected ways.

[0141] According to the processing of this embodiment, the processing load on the biometric information processing device 100 can be reduced by having an external device (image management server) perform processing on the subject's consented biometric information based on the consent information. At this time, by using the estimated consent information estimated from the biometric information and limiting the processing to be performed, users of the biometric information processing device 100 can also use biometric information without explicit consent, thereby increasing convenience. Furthermore, it is also convenient for the subject of the biometric information, as they do not have to go through the troublesome process of giving explicit consent.

[0142] [Variation 2-1] In this embodiment, the external device that outputs biological information is described as the image management server 8 in Figure 10, but a different device may be used as long as it can perform similar processing. For example, it may be the imaging device 1a or the mobile terminal 7.

[0143] [Variation 2-2] In this embodiment, biometric information was output to an authenticated external device regardless of consent information, but the biometric information that can be output may be restricted according to the consent information.

[0144] Furthermore, in this embodiment, the image management server 8 controls the processing target to biometric information where the consent information is either "presumed consent" or "explicit consent," but other controls may also be implemented. For example, the processing target may not be restricted by consent information (i.e., all biometric information is processed), and the processing may be switched according to the consent information. For example, for biometric information without consent, masking may be performed, and for others, the person in the image may be identified and the image classified. By doing so, the complex processing of managing biometric information without consent can be automated, thereby increasing convenience. Also, for people who have given consent, convenient processing can be performed accordingly, thus protecting privacy while increasing convenience.

[0145] [Modified example 2-3] In this embodiment, the external device, i.e., the image management server 8, processes the biometric information and feature quantities obtained from the biometric information processing device that has been authenticated. However, permission information for processing the biometric information may be linked to the biometric information, and the image management server 8 may process the information according to that permission information. Specifically, the biometric information processing device 100 may further include a permission information generation unit as a functional component, and the permission information creation unit may create permission information for the external device to process the biometric information based on the consent information. The created permission information is then output from the biometric information utilization unit 36 ​​to the external device, linked to the biometric information. The image management server 8 processes the biometric information based on the acquired permission information.

[0146] By linking permission information to biometric information in this way, the image management server 8 can determine whether it is permissible to process the biometric information and features without having to query the biometric information processing device 100, and the flexibility of processing timing is increased, thereby improving convenience.

[0147] Furthermore, this permission information may also include information about the services that the image management server 8 is permitted to perform. By doing so, if multiple services are available on the image management server 8, the image management server 8 alone can know which processing is permitted for which biometric information, thus increasing the flexibility of processing timing and improving convenience.

[0148] [Variation 2-4] Furthermore, in this embodiment, unintended processing of biometric information is prevented by controlling the output to authenticated external devices only. However, technologies such as encryption may be introduced to more reliably restrict use. Specifically, the biometric information processing device 100 is equipped with an encryption unit, and biometric information and feature quantities are encrypted by the encryption unit before being output to the external device. In addition, the biometric information processing device 100 is equipped with a decryption key generation unit as a functional component, which generates a decryption key for decrypting the encrypted biometric information and feature quantities. The decryption key generated by this decryption key generation unit is output from the information processing device 1 to the external device, and the external device decrypts the biometric information and feature quantities using the decryption key before using them. In this way, it is possible to prevent data from being extracted from the biometric information processing device 100 and used unintentionally. In the above embodiment, the imaging device 1 is a digital camera with a built-in biometric information processing device 100, and biometric information is often recorded on a small portable medium (a memory card such as an SD card). In such cases, even if the portable medium is removed, the biometric information cannot be used if the decryption key cannot be obtained, thus preventing unintentional use.

[0149] [Variation 2-5] In this embodiment, image classification is given as an example of a service provided by an external device, i.e., the image management server 8, but other services using other biometric information and features may also be used.

[0150] For example, a person may be identified from a large number of images using registered features, and then processed to make the person's face unrecognizable based on consent information. Known methods for making a person's face unrecognizable include masking the face, applying a mosaic effect, applying a blur filter, or cropping the image so that the face is not visible.

[0151] As shown in Modification 1-3, when the consent information value is expressed as a real number, the processing parameters may be controlled according to the consent information value. For example, the intensity of the blur filter may be adjusted proportionally to the reciprocal of the consent information value. In this way, the higher the consent information value, the less blur there will be, and the lower the explicitness, the greater the blur there will be, thus automatically protecting the subject's privacy in accordance with their intent, and thus controlling the use of biometric information in accordance with their intent. For example, a Gaussian filter can typically be used as the blur filter, and its processing parameter is the filter radius.

[0152] Furthermore, in examples of identifying individuals and classifying images, the threshold for considering two images to be the same person may be adjusted proportionally, for example, to the reciprocal of the consent information value. Here, the value for determining whether two images are the same person is, for example, the distance between each pixel in the image, and the value increases for different images. In this way, the smaller the consent information value, the higher the likelihood of being mistakenly classified as someone else, and the lower the likelihood of their own image being correctly classified. This reduces the proportion of their image that appears in the classification results, thus protecting the privacy of people with small consent information values. Conversely, the larger the consent information value, the lower the likelihood of being mistakenly classified as someone else, and the higher the likelihood of their own image being correctly classified. Although this increases privacy concerns, it can be used for processing that prioritizes the classification of one's own image.

[0153] [Other Embodiments] Embodiments 1 and 2 described examples of acquiring facial images as biometric information, but the biometric information is not limited to images containing faces, as long as it includes user information. For example, the biometric information may be an image containing information that can identify the user, such as an individual's iris, fingerprints, or veins. Even when using such biometric information, it is possible to extract features from the biometric information in the same way. The device that acquires the captured image may be an imaging device that includes appropriate sensors corresponding to each type of biometric information. For example, the subject of the captured image of the user of the biometric information may be linked to other biometric information of the individual, such as their iris, fingerprints, or veins, and other biometric information may be made available when consent can be inferred from the image context.

[0154] [Variations of bio-information processing devices] Although the biometric information processing device 100 according to this embodiment has been described as being incorporated into the imaging device 1 (digital camera), it is not limited to this configuration as long as similar processing can be performed. For example, the biometric information processing device 100 may be a smartphone with a camera, or it may be incorporated into a network camera equipped with pan, tilt, and zoom functions that allow adjustment of the imaging angle. In that case, one possible use of the results of identifying a person is to control the pan, tilt, and zoom to image the identified individual. When using the biometric information processing device 100 for such purposes, it is possible to suppress the infringement of privacy and prevent misuse by restricting the use of biometric information for persons who have not given their consent. Furthermore, when the biometric information processing device 100 is incorporated into a network camera, a separate server may be prepared to manage the network camera and record the captured video, and the server may perform the process of acquiring or registering consent information. In such a configuration, when acquiring the user's consent, a UI for acquiring consent information is displayed on the server, that is, a display showing that biometric information will be used and a display accepting the option of whether or not to consent, and consent can be obtained by user input using an input device such as a mouse or keyboard.

[0155] The biological information processing device of this embodiment has been described using an example in which the CPU reads and executes a program installed in non-volatile memory. However, it can also be realized by a hardware circuit (for example, an ASIC) that implements the functions described in Figure 3 of Embodiment 1.

[0156] (Composition 1) A biometric information processing device that processes information using a person's biometric information, A biological information acquisition unit that acquires the aforementioned biological information, A bio-information utilization unit that utilizes the aforementioned bio-information or features extracted from the aforementioned bio-information, An explicit consent acquisition unit for obtaining consent regarding the use of the biometric information of a person relating to the aforementioned biometric information, The system includes a biometric information usage restriction unit that restricts the use of the biometric information based on the consent value, The system further includes a consent status estimation unit that estimates a consent status indicating consent to the use of the biometric information of a person related to the biometric information, based on the acquisition status at the time the biometric information was acquired. The biometric information usage restriction unit is characterized in that it relaxes the restrictions on the use of the biometric information of the person concerned or the feature quantities extracted from the biometric information, based on the consent state estimated by the consent state estimation unit.

[0157] (Configuration 2) The agreement status estimation unit, The positional relationship between the device into which the biometric information processing device is incorporated and the person when acquiring the biometric information, The state of the person at the time the aforementioned biometric information was acquired, The biometric information of persons other than the aforementioned person included in the biometric information, and consent information, Hardware configuration of the device into which the aforementioned bio-information processing device is incorporated, The acquisition parameters when the biological information acquisition unit acquires biological information, The person’s past consent information at the time of acquiring the biometric information, The biometric information processing device according to configuration 1, characterized in that it estimates the consent status of the person based on either the operating location of the device into which the biometric information processing device is incorporated, the installation method of the device into which the biometric information processing device is incorporated, or a combination thereof.

[0158] (Composition 3) The biometric information processing device according to either configuration 1 or configuration 2, characterized in that the biometric information is an image including a person's face.

[0159] (Composition 4) A biological information processing device according to any one of configurations 1 to 3, characterized in that it is incorporated into an imaging device and takes an image captured by the imaging device as input.

[0160] (Composition 5) The system further includes a consent information generation unit that generates consent information indicating the degree of explicit consent based on the consent value and the estimated consent status. The aforementioned limiting unit is characterized in that the degree of relaxation increases as the degree of explicit consent increases.

[0161] (Composition 6) The biometric information processing apparatus according to configuration 5, characterized in that the biometric information usage restriction unit permits the display of the person's biometric information or feature quantities extracted from the biometric information by the biometric information usage unit based on the degree of explicitness of consent indicated in the consent information.

[0162] (Composition 7) The biometric information processing apparatus according to configuration 5 is characterized in that the biometric information usage restriction unit permits the output of the person's biometric information and the feature quantities extracted from the biometric information to an external device by the biometric information usage unit based on the degree of explicitness of consent indicated in the consent information.

[0163] (Composition 8) The system further comprises a permission information generation unit that generates permission information for the use of the aforementioned biological information or feature quantities extracted from the biological information by the external device, The biometric information processing device according to configuration 7, characterized in that the biometric information utilization unit outputs the permission information to the external device.

[0164] (Composition 9) Authentication unit that performs authentication processing with the external device. Furthermore, The biometric information processing apparatus according to configuration 7, characterized in that the biometric information utilization unit outputs the permission information, the consent information, and the biometric information or feature quantities extracted from the biometric information to an external device that has undergone authentication processing by the authentication unit.

[0165] (Composition 10) The biometric information processing device according to configuration 9, characterized in that the biometric information usage restriction unit permits output to the external device authenticated by the biometric information usage unit when the degree of explicitness of consent indicated by the consent information is greater than the consent state in which consent is presumed.

[0166] (Composition 11) An encryption unit that encrypts the aforementioned biological information or feature quantities extracted from the aforementioned biological information, The system further comprises a decryption key generation unit that generates a decryption key for decrypting the encrypted biometric information or the feature quantities extracted from the encrypted biometric information, The biometric information processing apparatus according to configuration 10, characterized in that the biometric information utilization unit outputs the decryption key and the encrypted biometric information or feature quantities extracted from the encrypted biometric information to the external device.

[0167] (Composition 12) An information processing system comprising a biometric information processing device that processes a person's biometric information, and an external device connected to the biometric information processing device that receives and utilizes the biometric information generated by the biometric information processing device, The aforementioned biological information processing device is A biological information acquisition unit that acquires the aforementioned biological information, The system includes an explicit consent acquisition unit that obtains consent for the use of the biometric information of a person relating to the aforementioned biometric information, The external device estimates the consent status indicating consent for the use of the biometric information of the person concerned, based on the acquisition status at the time the biometric information was acquired. The external device is an information processing system characterized by relaxing restrictions on the use of the biometric information of the person concerned or features extracted from the biometric information, based on the estimated consent status.

[0168] (Method 1) A method for processing biometric information using a biometric information processing device that processes biometric information of a person, The biometric information processing device performs a biometric information acquisition step of acquiring a person's biometric information, The biological information processing device performs a biological information utilization step that utilizes the biological information or features extracted from the biological information, The biometric information processing device includes an explicit consent acquisition step in which it acquires the consent status of the person for the use of the biometric information in the biometric information utilization step, The biometric information processing device includes a biometric information usage restriction step that restricts the use of the biometric information in the biometric information usage step based on the consent value, The biometric information processing device includes a consent state estimation step that estimates a consent state indicating consent to the use of the biometric information of a person related to the biometric information, based on the acquisition status at the time the biometric information was acquired. The biometric information processing method is characterized in that the biometric information usage restriction step relaxes the restrictions on the use of the biometric information of the person whose estimated consent status indicates consent, or the features extracted from the biometric information.

[0169] (Program 1) A program that causes a computer to perform each of the steps described in Method 1. [Explanation of symbols]

[0170] 1,1a…Imaging device, 2,3…Subject, 5…Network, 7…Mobile terminal, 8…Image management server, 11…Shutter button, 12…Display panel, 13…Operation keys, 14…Display panel, 20…Imaging unit, 21…Image sensor, 22…Optical system, 14...Operating device, 15...Display device, 16...Communication I / F, 100...Biometric information processing device 1, 111...CPU, 112...DSP, 113...Main memory, 114...Non-volatile memory, 31...Biometric information acquisition unit, 32...Consent status estimation unit, 33...Person identification unit, 34...Explicit consent acquisition unit, 35...Consent information generation unit, 36...Biometric information utilization unit, 37...Biometric information utilization restriction unit, 38...Device authentication unit, 50...Personal registration table, 51...Estimated consent information table, 60...Image data, 61...Metadata

Claims

1. A biometric information processing device that processes information using a person's biometric information, A biological information acquisition unit that acquires the aforementioned biological information, A bio-information utilization unit that utilizes the aforementioned bio-information or features extracted from the aforementioned bio-information, An explicit consent acquisition unit for obtaining consent regarding the use of the biometric information of a person relating to the aforementioned biometric information, The system includes a biometric information usage restriction unit that restricts the use of the biometric information based on the consent value, The system further includes a consent status estimation unit that estimates a consent status indicating consent to the use of the biometric information of a person related to the biometric information, based on the acquisition status at the time the biometric information was acquired. The biometric information usage restriction unit is characterized in that it relaxes the restrictions on the use of the biometric information of the person concerned or the feature quantities extracted from the biometric information, based on the consent state estimated by the consent state estimation unit.

2. The agreement status estimation unit, The positional relationship between the device into which the biometric information processing device is incorporated and the person when acquiring the biometric information, The state of the person at the time the aforementioned biometric information was acquired, The biometric information of persons other than the aforementioned person included in the biometric information, and consent information, Hardware configuration of the device into which the aforementioned bio-information processing device is incorporated, The acquisition parameters when the biological information acquisition unit acquires biological information, The person’s past consent information at the time of acquiring the biometric information, The biometric information processing device according to claim 1, characterized in that it estimates the consent status of the person based on either the operating location of the device into which the biometric information processing device is incorporated, the installation method of the device into which the biometric information processing device is incorporated, or a combination thereof.

3. The biological information processing apparatus according to claim 1, characterized in that the biological information is an image including a person's face.

4. The biological information processing device according to claim 1, which is incorporated into an imaging device and takes an image captured by the imaging device as input.

5. The system further includes a consent information generation unit that generates consent information indicating the degree of explicit consent based on the consent value and the estimated consent status. The biological information processing apparatus according to claim 1, characterized in that the degree of relaxation of the restriction increases as the degree of explicit consent increases.

6. The biometric information processing apparatus according to claim 5, wherein the biometric information usage restriction unit permits the display of the person's biometric information or feature quantities extracted from the biometric information by the biometric information usage unit based on the degree of explicitness of consent indicated in the consent information.

7. The biometric information usage restriction unit permits the output of the biometric information of the person and the feature quantities extracted from the biometric information to an external device by the biometric information usage unit, based on the degree of explicitness of consent indicated in the consent information, as described in claim 5.

8. The system further comprises a permission information generation unit that generates permission information for the use of the aforementioned biological information or feature quantities extracted from the biological information by the external device, The biometric information processing device according to claim 7, characterized in that the biometric information utilization unit outputs the permission information to the external device.

9. Authentication unit that performs authentication processing with the external device. Furthermore, The biometric information processing apparatus according to claim 7, characterized in that the biometric information utilization unit outputs the permission information, the consent information, and the biometric information or feature quantities extracted from the biometric information to an external device that has been authenticated by the authentication unit.

10. The biometric information processing device according to claim 9, characterized in that the biometric information usage restriction unit permits output to the external device authenticated by the biometric information usage unit when the degree of explicitness of consent indicated by the consent information is greater than the consent state in which consent is presumed.

11. An encryption unit that encrypts the aforementioned biological information or feature quantities extracted from the aforementioned biological information, The system further comprises a decryption key generation unit that generates a decryption key for decrypting the encrypted biometric information or the feature quantities extracted from the encrypted biometric information, The biometric information processing apparatus according to claim 10, characterized in that the biometric information utilization unit outputs the decryption key and the encrypted biometric information or feature quantities extracted from the encrypted biometric information to the external device.

12. An information processing system comprising a biometric information processing device that processes a person's biometric information, and an external device connected to the biometric information processing device that receives and utilizes the biometric information generated by the biometric information processing device, The aforementioned biological information processing device is A biological information acquisition unit that acquires the aforementioned biological information, The system includes an explicit consent acquisition unit that obtains consent for the use of the biometric information of a person relating to the aforementioned biometric information, The external device estimates the consent status indicating consent for the use of the biometric information of the person concerned, based on the acquisition status at the time the biometric information was acquired. The external device is an information processing system characterized by relaxing restrictions on the use of the biometric information of the person concerned or features extracted from the biometric information, based on the estimated consent status.

13. A method for processing biometric information using a biometric information processing device that processes biometric information of a person, The biometric information processing device performs a biometric information acquisition step of acquiring a person's biometric information, The biological information processing device performs a biological information utilization step that utilizes the biological information or features extracted from the biological information, The biometric information processing device includes an explicit consent acquisition step in which it acquires the consent status of the person for the use of the biometric information in the biometric information utilization step, The biometric information processing device includes a biometric information usage restriction step that restricts the use of the biometric information in the biometric information usage step based on the consent value, The biometric information processing device includes a consent state estimation step that estimates a consent state indicating consent to the use of the biometric information of a person related to the biometric information, based on the acquisition status at the time the biometric information was acquired. The biometric information processing method is characterized in that the biometric information usage restriction step relaxes the restrictions on the use of the biometric information of the person whose estimated consent status indicates consent, or the features extracted from the biometric information.

14. A program for causing a computer to perform each of the steps described in claim 13.

Images