Biological information processing apparatus, information processing system, biological information processing method, storage medium, and program product

Abstract

The invention provides a biological information processing apparatus, an information processing system, a biological information processing method, a storage medium, and a program product. The biological information processing apparatus includes: a biological information acquisition unit that acquires biological information; a biological information using unit that uses the biological information and the features extracted from the biological information; an explicit agreement acquisition unit that acquires agreement of the use of the biological information of the person; a biological information use restriction unit that restricts the use of the biological information on the basis of the agreed value; and an agreement state estimation unit that estimates an agreement state indicating agreement of the use of the biological information of the person on the basis of the acquisition state when the biological information is acquired. Further, a biological information use restriction unit relaxes restrictions on the use of the biological information of the person and features extracted from the biological information on the basis of the agreement state estimated by the agreement state estimation unit.

Description

TECHNICAL FIELD

[0001] The present application relates to a biological information processing apparatus, and particularly to a biological authentication technology that is applicable to the use of biological information of a person without causing any ethical problems even if the consent of the person has not been obtained. BACKGROUND

[0002] In recent years, the use of biological information (biometrics) has been greatly expanded. As a representative example, in a biological authentication technology for security and authentication, an AI technology is known that uses parameters that have been learned from data such as images and the like to perform processing such as detection and estimation. Such AI processing is capable of intelligent processing and judgment in the same manner as a person, and therefore in recent years, the application of this AI processing in various fields has been progressing.

[0003] On the contrary, the use of personal information (for example, face images, fingerprints, and irises, and the like) by a third party without the permission of the parties concerned can infringe on personal privacy and cause ethical problems, and requires certain considerations. In particular, biological authentication of face images and fingerprints and the like can infringe on personal privacy depending on its purpose, and therefore, it has been indicated that the collection and use of biological information will foster the possibility of improper monitoring and discrimination. Therefore, in the use of biological information, greater restrictions are expected.

[0004] For example, Japanese Patent Application Publication No. 2022-119549 (unexamined patent application) discloses matters related to a technology that uses authentication using face recognition. Japanese Patent Application Publication No. 2022-119549 (unexamined patent application) discloses a method of obtaining the consent of an individual before registering a subject that performs authentication using face recognition.

[0005] Furthermore, in this regard, for example, Non-Patent Literature 1 (Deng, Jiankang, et al. “Retinaface: Single-shot multi-level face localization in the wild.” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020.) contains a disclosure of a technology that specifies the position at the time of acquiring a face image.

[0006] Although Japanese Patent Application Publication No. 2022-119549 (unexamined patent application) discloses obtaining consent before face image acquisition, there is a possibility that feature extraction from the biological information represented by a face image and the use of these features and the like will be performed without the consent of the parties concerned having been given.

[0007] Furthermore, even if control is performed so that only biological information of a person who has given consent thereto can be used, it is a time- and labor-consuming task for both the user and the operator of the biometric authentication to obtain the consent of the person, and convenience in operating the system decreases.

[0008] An object of the present application is to provide a biological information processing apparatus that can use biological information of a person and feature points from the biological information in a mode that takes into account personal privacy without causing moral problems even if consent of the person has not been obtained, and without increasing time and labor required for an operator. SUMMARY

[0009] The biological information processing apparatus of the present application is preferably a biological information processing apparatus that performs processing by using biological information from a person, the biological information processing apparatus including: a biological information acquisition unit that acquires biological information; a biological information use unit that uses the biological information or a feature extracted from the biological information; an explicit consent acquisition unit that acquires consent relating to use of the biological information from a person associated with the biological information; and a biological information use restriction unit that restricts use of the biological information based on a value of the consent; further including a consent state presumption unit that presumes a consent state indicating the consent relating to use of the biological information from the person associated with the biological information based on an acquisition state at the time of acquisition of the biological information, wherein the biological information use restriction unit has been caused to relax a restriction relating to use of the biological information from the person associated with the biological information and the feature that has been extracted from the biological information based on the consent state that the consent state presumption unit has presumed.

[0010] Features of the present disclosure will become apparent from the following description of embodiments with reference to the drawings. The following description of embodiments is described by way of example with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1A is a diagram (one of two diagrams) that illustrates a camera state when the biological information processing apparatus is used.

[0012] Figure 1B is a diagram (one of two diagrams) that illustrates a camera state when the biological information processing apparatus is used. Figure Two ) is a diagram (one of two diagrams) that illustrates a camera state when the biological information processing apparatus is used.

[0013] Figure 2 is a hardware structure diagram of a camera apparatus.

[0014] Figure 3 is a block diagram showing one example of a functional structure of the biological information processing apparatus according to the first embodiment. is a diagram (one of two diagrams) that illustrates a camera state when the biological information processing apparatus is used.

[0015] Figure 4 Fig. 1 is a diagram showing one example of an individual registration table.

[0016] Figure 5 Fig. 2 is a diagram showing one example of an inferred consent information table.

[0017] Figure 6 Fig. 3 is a flowchart showing a process for registering information about an individual in relation to which biological information is to be used.

[0018] Figure 7 Fig. 4 is a flowchart showing a process for using registered biological information of an individual.

[0019] Figure 8 Fig. 5 is a diagram showing one example of displaying information for AF / AE on a display panel.

[0020] Figure 9 Fig. 6 is a diagram showing one example of a rule for the relationship between the use of biological information and consent information.

[0021] Figure 10 Fig. 7 is a structural diagram of an information processing system using biological information.

[0022] Figure 11 Fig. 8 is a flowchart showing a series of processes in which biological information that has been obtained by the imaging device 1 is used in an external process in the information processing system according to the second embodiment.

[0023] Figure 12 Fig. 9 is a flowchart showing details of the use process of biological information. DETAILED DESCRIPTION

[0024] While the present disclosure has been described with reference to embodiments, it is to be understood that the disclosure is not limited to the disclosed embodiments. The scope of the appended claims encompasses all such modifications and equivalents of the subject matter recited in the claims.

[0025] Hereinafter, various embodiments of the present disclosure will be described using Figures 1A-12 Hereinafter, various embodiments of the present disclosure will be described using

[0026] [First Embodiment]

[0027] Hereinafter, various embodiments of the present disclosure will be described using Figures 1A-9 Hereinafter, various embodiments of the present disclosure will be described using

[0028] The bioinformatics processing device according to this embodiment acquires designated bioinformatics information and obtains explicit consent, which indicates whether the person has already consented or has not yet consented to the use of the designated person's information. Furthermore, the bioinformatics processing device presumes the existence of presumed consent based on the state at the time of acquiring the designated person's information, and restricts the use of the designated person's information based on the acquired explicit consent and presumed consent.

[0029] The following method will be described: In an image of a person captured by a camera device (digital camera), the biometric processing device according to this embodiment uses biometric information for biometric authentication and restricts the use of biometric information. Note that although the biometric processing device described in this embodiment is a device built into the camera device, it may also be a general information processing device (such as a PC (personal computer)) that processes images from the camera device, or it may be a server connected via a network.

[0030] First, we will use Figure 1A and Figure 1B To illustrate the camera state when the bio-information processing device according to the first embodiment is used.

[0031] Figure 1A This is a diagram illustrating the camera status when the bioinformatics processing device is in use (Figure 1 of the two diagrams).

[0032] Figure 1B These are diagrams illustrating the camera status when the bioinformatics processing device is in use (two diagrams). Figure Two ).

[0033] The bioinformatics processing device according to this embodiment is incorporated into... Figure 1A The image is shown in the portion of the camera device 1, which is an external view, and the biological information included in the image is processed. The camera device 1 is, for example, a digital camera. Figure 1B As shown from Figure 1A The image shows the camera device 1 as seen from the opposite side of the orientation (i.e., as seen from the user's (photographer's) side).

[0034] In this context, Figure 1B The subject 2 shown is Figure 1A The first subject being photographed is a person, and in this context, subject 2 is a person. Additionally, subject 3 is the second subject being photographed by camera device 1, and also subject 3 is a person. Note that below, the person who is a subject in the image is referred to as an "individual".

[0035] Below, we will use Figure 1B Describe the external structure of the camera device 1 according to the user interface.

[0036] The shutter button 11 is a button that is pressed by the user (photographer) of the image pickup apparatus 1 to open and close the shutter and instructs the image pickup apparatus 1 to perform image pickup. The display panel 12 is a panel for displaying the processing result to the user of the image pickup apparatus 1. The display panel 12 displays an image for confirming the composition and the state of the subject at the time of image pickup. The display panel 12 can be any type of display as long as it can display an image, and for example, can also be a liquid crystal panel, an organic EL (electroluminescence) panel, and a projector.

[0037] The operation key 13 is an operation key for obtaining input from the user. The operation key 13 is not limited to a specific type of key as long as it can input information from the user. For example, the operation key 13 can be a hardware button provided to the image pickup apparatus 1, or the operation key 13 can be a key that operates a key displayed on the display panel 12 when the display panel 12 is in the form of a touch panel.

[0038] Next, the hardware structure of the image pickup apparatus will be described using Figure 2

[0039] Figure 2 is a hardware structure diagram of the image pickup apparatus.

[0040] As shown in Figure 2 , the image pickup apparatus 1 includes an image pickup unit 20, a biological information processing apparatus 100, an operation apparatus 14, a display apparatus 15, and a communication I / F 16.

[0041] The image pickup unit 20 is a structure that photographs external light, and has an image pickup element 21 and an optical system 22. The optical system 22 is a structure that creates an image of a subject and converges light using physical phenomena such as refraction, reflection, and diffraction of light, and has a function of forming an image of a subject on the image pickup element 21. The optical system 22 is configured by a plurality of lens groups (not shown) and an aperture (not shown), and the like.

[0042] The operation apparatus 14 includes the shutter button 11 and the operation key 13 that have been described, and is an apparatus attached to the image pickup apparatus 1 for the user to operate the image pickup apparatus 1. The display apparatus 15 is an example of the display panel 12 that has been described, and is an apparatus that displays information when the user is using the image pickup apparatus to perform image pickup and when the user is performing necessary settings.

[0043] The biological information processing apparatus 100 is an apparatus that performs information processing for processing an image that has been photographed including biological information. The biological information processing apparatus 100 is, for example, as described in Figure 2 ​The CPU 111, the DSP 112, the main memory 113, and the nonvolatile memory 114 are connected via a bus as shown below. The CPU (Central Processing Unit) 111 is a processor that refers to data on the main memory 113 and executes a program. The main memory 113 is a high-speed nonvolatile semiconductor storage device that holds data and a program accessed by the CPU 111. The DSP 112 is a processor that converts an image signal into digital data for an image. The nonvolatile memory 114 is a nonvolatile semiconductor storage device such as a flash memory that stores data and a program.

[0044] The nonvolatile memory 114 of the present embodiment stores the image data 60, the image data 61 in which data related to the image data is written, the individual registration table 50, and the presumed consent information table 51. Details of the image data 61, the individual registration table 50, and the presumed consent information table 51 will be described below.

[0045] In addition, as shown in Figure 2 The nonvolatile memory 114 has programs (the biological information acquisition program 41 to the device authentication program 48) that execute each type of function installed thereon. Note that each function will be described below.

[0046] The communication I / F 16 is an interface that communicates with an external device. The communication I / F 16 can communicate by using wired communication such as USB, or the communication I / F 16 can also communicate through a local area network and wireless communication as defined by the IEEE 802.11 standard.

[0047] Next, the functional structure of the biological information processing device according to the first embodiment will be described using Figure 3

[0048] Figure 3 is a block diagram showing an example of the functional structure of the biological information processing device according to the first embodiment.

[0049] As shown in Figure 3 The biological information processing device includes each functional unit of the biological information acquisition unit 31, the consent state presumption unit 32, the person specification unit 33, the explicit consent acquisition unit 34, the consent information generation unit 35, the biological information use unit 36, the biological information use restriction unit 37, and the device authentication unit 38.

[0050] The biological information acquisition unit 31 is a functional unit that acquires biological information. In this context, the biological information acquisition unit 31 acquires biological information from a captured image in which a subject appears.

[0051] ​The consent state inferring unit 32 infers whether the person designated by the person designating unit 33 has given consent to the use of the biological information based on the biological information and the acquisition state at the time of acquisition of the biological information, and outputs inferred consent information as a result of the inference. A specific example of inferring the consent state will be described below.

[0052] The person designating unit 33 extracts a feature from the biological information that has been input, and designates a person. The extracted feature is associated with the biological information and the consent information, and stored in the nonvolatile memory 114. In addition, the person designating unit 33 can also associate a person name that has been input from the operation device 14 with the feature and the consent information, and store it in the nonvolatile memory 114. In addition, the person designating unit 33 can also have a function of designating a specific person by referring to the features that have been registered. In addition, the process of extracting a feature from biological information performed by the person designating unit 33 also has a function of controlling whether to extract a feature based on consent to the use of the biological information, i.e., restricting the use of the imaging device.

[0053] The explicit consent acquisition unit 34 acquires explicit consent information that shows explicit consent to the use of the biological information from the person to whom the biological information belongs. In this context, the explicit consent acquisition unit 34 can acquire the explicit consent information based on input via the operation key 13. Note that, hereinafter, a case where "consent" is simply written indicates consent to the use of the biological information and the feature that has been extracted from the biological information as described above. Furthermore, although in this context, it is described that the biological information indicates the entire captured image in which the subject appears, the biological information is not particularly limited as long as it is information within an image that includes subject information. For example, an image of a square region (bounding box) in which a subject has been detected, an image of a square region in which a subject face has been detected, and the like can also be made as biological information.

[0054] The consent information generating unit 35 combines the explicit consent information acquired by the explicit consent acquisition unit 34 and the inferred consent information output by the consent state inferring unit 32, and generates consent information that shows the degree of explicit consent to the use of the imaging device with respect to the subject. The generated consent information is associated with the corresponding biological information, and stored on the main memory 113 and the nonvolatile memory 114. In the present embodiment, the biological information is face image information in which a person appears in an image, and the biological information is stored by associating coordinate information of a square that shows the position of the face with the consent information.

[0055] The biological information use unit 36 uses the biological information and the feature based on the consent information that has been generated by the consent information generating unit 35. A specific method of use will be described below.

[0056] The biological information use restriction unit 37 restricts the use of biological information by the biological information use unit 36 based on the consent information generated by the consent information generation unit 35. The specific restriction method will be described below.

[0057] The device authentication unit 38 performs authentication with an external device that uses biological information. Therefore, in a case where the authentication of the external device that uses biological information by the device authentication unit 38 has been authenticated, the biological information processing device 100 is able to perform transmission and reception of information such as requests for data and processing, and the like with the external device.

[0058] Next, the data structure used by the biological information processing device of the present embodiment will be described using Figure 4 and Figure 5

[0059] Figure 4 is a diagram showing one example of an individual registration table.

[0060] Figure 5 is a diagram showing one example of presumed consent information table.

[0061] As shown in Figure 4 , the individual registration table 50 includes fields of individual ID 50a, name 50b, explicit consent 50c, face image 50d, and feature 50e.

[0062] An ID that uniquely identifies an individual is stored in the individual ID 50a. In a case where the name of the individual is clear, a character string showing the name of the person is stored in the name 50b. A flag showing whether or not there is an explicit consent for the use of biological information from the corresponding individual is stored in the explicit consent 50c. For example, when the value of the explicit consent 50c is "Yes", it is set that there is consent, and when the value of the explicit consent 50c is "No", it is set that it shows that consent has not been obtained, and a default value is set to "No". Data of a face image of the corresponding individual is stored in the face image 50d (for example, the data can be a value of an image ID and a square coordinate, or the data can also be specific image data). A feature of biological information that has been calculated using a specific algorithm is stored in the feature 50e.

[0063] In the example shown in Figure 4 , the names "Ta○ △shi", "●yama □ko", and "Suzu◎▽fu" are registered in the name column. In addition, Figure 4 ​It is shown that explicit consent regarding the use of the biological information has been obtained for "Suzu◎▽fu", and the feature thereof has been extracted and registered. Further, it is shown that explicit consent regarding the use of the biological information has not been obtained for either of "Ta○△shi" and "●yama□ko", and the feature of none of them has been stored.

[0064] The presumed consent information table 51 is a table that stores the result of whether there is presumed consent for the individual who has been registered in the individual registration table 50. As shown in Figure 5 The presumed consent information table 51 includes fields of an individual ID 51a and a presumed consent 51b.

[0065] The ID that uniquely identifies the individual is stored in the individual ID 51a in the same manner as the individual ID 50a of Figure 2 The flag that shows whether there is presumed consent for the use of the biological information from the corresponding individual is stored in the presumed consent 51b. For example, when the value of the presumed consent 51b is "Yes", this shows that there is presumed consent, and when the value of the presumed consent 51b is "No", this shows that there is no presumed consent. In this context, the presumed consent means that consent for the use of the biological data from the corresponding person is presumed to exist based on the imaging state and the captured image. Note that the specific process for judging whether there is presumed consent will be described below.

[0066] Next, the process in which the biological information processing apparatus uses the biological information will be explained using Figures 6-9

[0067] Figure 6 is a flowchart that shows the process for registering information about the individual for whom the biological information will be used.

[0068] Figure 7 is a flowchart that shows the process of using the biological information of the individual who has been registered.

[0069] Next, the process in which the biological information processing apparatus 100 according to the present embodiment uses the biological information is divided into the process of registering information about the individual for whom the biological information will be used and the process of using the biological information of the individual who has been registered.

[0070] In addition, in the process of registering information about the individual, the biological information processing apparatus 100 according to the present embodiment extracts the feature about the biological information only when there is explicit consent from the individual himself or herself. In contrast, in the process of using the biological information of the individual who has been registered, it is judged whether there is presumed consent from the individual himself or herself based on the imaging state and the image, and when there is presumed consent, the restriction on the use of the biological information is relaxed even if there is no explicit consent from the individual himself or herself. ​

[0071] First, the use of Figure 6 The process for registering information about the individual whose biological information will be used will be explained.

[0072] For example, the process for registering information about the individual whose biological information will be used is started by the user of the imaging device 1 making a command for executing the registration process by pressing the operation key 13 or the like as shown in Figure 1B The user according to the present embodiment is a user who takes an image of a subject by using the imaging device 1 (such as a digital camera or the like) including the biological information processing device 100.

[0073] First, the biological information processing device 100 performs an initialization process for the registration process of information about the individual (S101). In this context, the CPU 111 of the biological information processing device 100 reads a program from the non-volatile memory 114 of Figure 2 and makes the program a state in which the registration process explained below can be performed. The process shown in the following flowchart is realized by the CPU 111 of the biological information processing device 100 executing the necessary program. Figure 2

[0074] Next, the biological information acquisition unit 31 of the biological information processing device 100 acquires biological information (in this context, an image showing a person's face from the imaging device 1 of Figure 1A and Figure 1B (S102). At this time, the biological information acquisition unit 31 controls the imaging device 1 to adjust the focus and exposure with respect to the face using a known technique, and acquires a captured image. Note that although in this context, the following explanation is given that there is one person as a subject in the captured image that will become biological information, there can be a plurality of subjects present in the captured image, and the process explained below can be performed individually with respect to each of these plurality of subjects.

[0075] Next, the explicit consent acquisition unit 34 of the biological information processing device 100 acquires explicit consent information about the individual (subject) appearing in the captured image acquired during S102 (S103). As explained previously, the explicit consent acquired in this context shows the presence or absence of consent with respect to the use of the image in which the face of the appearing individual appears (that is, the use of biological information).

[0076] ​In this context, "use of the biometric information" includes at least one of a feature registration process of taking and recording a face image and extracting a feature from the face image, a feature-based authentication process (in this context, a process of specifying a person), and a control using a result of the authentication process. Note that details of "use of the biometric information" will be described below. The biometric information processing device 100 according to the present embodiment restricts the use of the biometric information based on explicit consent information and presumed consent information to be described below.

[0077] The explicit consent acquisition unit 34 according to the present embodiment can acquire the explicit consent information based on an operation of the user via, for example, the display panel 12 and the operation key 13. Specifically, the explicit consent acquisition unit 34 can display, on the display panel 12, a display indicating the use of the biometric information and receive a selection of whether or not to give consent to the use of the biometric information, and acquire a result of the selection to be used as the explicit consent information.

[0078] Note that in this context, the name of the user is associated with a unique ID (for example, based on an input of the user during login) and is set in advance. However, the user can also be set based on a captured image different from the biometric information to prohibit a consent operation by a person other than the user, that is, so-called "identity spoofing". For example, a camera device (not shown) that captures an image of an operator of the operation key 13 can also be provided, and the user can be set based on a captured image from such a camera device (using a known person specifying process). In such a case, a judgment as to whether or not the person who has acquired the biometric information during S102 and the set user are the same person can also be additionally made, and in the case where they are not the same person, the process of S104 is completed at this point. Figure 6

[0079] In addition, although in this context, it has been described that the explicit consent information is acquired based on an input of the user via the operation key 13, the process is not particularly limited as long as the process can acquire the presence or absence of consent from the user. For example, in the case where it has been detected that the user has made a certain movement (for example, a sound of consent and a gesture of consent have been made), the user can be judged to consent to the use of the biometric information, and the capturing of the camera device 1 and the use process for the biometric information (for example, authentication information) can be performed. According to such a process, approval of consent can be shown using only a gesture, and thus convenience is improved. Note that a movement such as a peace sign made with a hand, a hand raised, and a hand lowered can be used as a gesture, and a known movement recognition technology can be used to perform gesture recognition.

[0080] ​Next, the explicit consent acquisition unit 34 of the biological information processing apparatus 100 performs a judgment on whether or not the user has consented to the use of his or her biological information based on the explicit consent information acquired during S103 (S104). In the case where the user has consented (S104: Yes), the processing proceeds to S105, and in the case where the user has not consented (S104: No), the processing proceeds to S107.

[0081] Next, the person specifying unit 33 of the biological information processing apparatus 100 extracts a feature from the biological information of the user who has consented to the use of biological information (S105). As described above, the biological information is a captured image in which the face of an individual appears, and the person specifying unit 33 extracts a feature from the face after having specified the face position in the captured image. During the processing of specifying the face position in the captured image and the processing of extracting a feature from the face of a person, any publicly known image recognition technique can be used. For example, the technique disclosed in Non-Patent Literature 1 or the like can be used to specify the face position in the captured image. Also, for example, a method using deep learning can be used to extract a feature from the face of a person, and a method such as LBP (Local Binary Pattern) and HoG (Histogram Oriented Gradient) or the like can be used to extract a feature from the face of a person.

[0082] When there is explicit consent in S104, the person specifying unit 33 of the biological information processing apparatus 100 makes the explicit consent "Yes", associates the biological information and the extracted feature with the unique identifier, and registers the values in the respective fields of the individual registration table 50 illustrated in Figure 4 (S106). The person specifying unit 33 can acquire the name by receiving the input of the user using the operation apparatus 14. Figure 2 The name format according to the present embodiment is not particularly limited, and for example, the name can be the real name of a person, the name can also be a nickname, and the name can also be a number and a symbol or the like. Also, although it has been explained that the person specifying unit 33 of the present embodiment performs the registration of the explicit consent information, this processing can also be performed by a different functional unit, such as causing the explicit consent acquisition unit 34 to register the explicit consent information or the like.

[0083] When there is no explicit consent during S104, the person specifying unit 33 of the biological information processing apparatus 100 makes the explicit consent "No", associates the biological information with the unique identifier, and registers the values in the respective fields of the individual registration table 50 illustrated in Figure 4 (S107).

[0084] Next, the human designation unit 33 of the biometric processing device 100 determines whether the registration process is complete (S108). In this context, for example, it can be determined that the registration process will be completed if the user has already made input to complete the registration process. If it has been determined that the registration process will be completed (S108: Yes), the process is completed; if it has been determined that the registration process will not be completed (S108: No), the process returns to S102.

[0085] Next, we will use Figure 7 This is to illustrate the processing of biometric information from registered individuals.

[0086] Figure 7 The process shown illustrates the process of capturing an image of a subject using techniques specified by the user of a camera device 1, such as a digital camera. For example, the image is created by the user of camera device 1. Figure 1B The shutter button 11 shown is in a half-pressed state to start. Figure 7 The process is shown below. Note that the video recording (image capture) is performed by pressing the shutter button further after half-pressing it. Alternatively, it can be... Figure 6 After the registration process of the information related to the individual shown has been completed at least once, the processing of using biometric information (authentication processing) according to this embodiment is carried out.

[0087] First, the bioinformatics processing device 100 initializes the processing using bioinformatics (S111). In this context, the bioinformatics processing device 100... Figure 2 The non-volatile memory 114 is read from the program, enabling the program that will perform the following processes to operate. Additionally, as an initialization process, the bioinformatics processing device 100 loads the registration dictionary from the non-volatile memory 114 and puts it into an available state.

[0088] Next, the bioinformatics acquisition unit 31 of the bioinformatics processing device 100 acquires bioinformatics (in this context, from...) Figure 1B The image captured by the camera device 1 is an image of an individual's face. In this context, the biometric acquisition unit 31 acquires images of registered candidates from the imaging element 21 for use as biometric information. During the processing described below, the captured images in this context are displayed on the display panel 12 as a live view. The user can confirm the live view, determine the composition of the image to be recorded, and the timing of pressing the shutter button 11. Note that the captured images acquired in this context may also include multiple faces.

[0089] Next, the person specifying unit 33 of the biological information processing apparatus 100 extracts features from the input image (biological information), and specifies who the subject of the biological information is by comparing the extracted features with the features stored in the individual registration table 50 (S113). In order to confirm the presence or absence of explicit consent, it is necessary to specify the person, and therefore, in this context, features are temporarily extracted from biological information that has been input without explicit consent, and are deleted at the time of completion of use of the features. In the case where a plurality of faces is included in the image, features are extracted from each of the plurality of faces. Specifically, the comparison of features calculates the similarity between the features, and in the registered individuals, in the case where there is an individual whose similarity exceeds a predetermined threshold, it can be judged that the extracted person is that individual, and in the case where there is no person whose threshold has been exceeded, it can be judged that the extracted person is not any of the individuals. In this context, an arbitrary publicly known technique can be used as the method of calculating the similarity, and for example, a Cosine similarity can be used as the similarity, and an L2 distance or the like can also be used. The L2 distance is a criterion in which the closer the distance between the features, the smaller the value becomes, and therefore, it can also be used by reversing the multiplication of the L2 distance to the similarity. In this context, as described in the description of Figure 4

[0090] Next, the consent state inferring unit 32 of the biological information processing apparatus 100 acquires the value of the explicit consent 50c in the corresponding record with respect to the person who has been specified in the input biological information, and confirms the presence or absence of consent (S114). As described above, the two values possible for the explicit consent 50c are yes / no with respect to consent.

[0091] Next, the consent state inferring unit 32 of the biological information processing apparatus 100 refers to the explicit consent information acquired during S114, and infers the consent state with respect to the subject for which there is no explicit consent (S115). In this way, it is possible to refine the object of the inference of the consent state to the subject for which there is no explicit consent, and this is related to reducing the processing burden. Furthermore, it is also possible to infer the consent state with respect to all people without performing refinement. Below, a specific explanation of the consent state inference method will be given.

[0092] ​To infer the consent state of the subject, it is set to determine that there is an inferred consent in a case where the acquisition state satisfies a predetermined condition after having referred to the acquisition state of the biometric information. In this context, the acquisition state is the state of the device, the subject, and the surroundings thereof at the time of acquisition of the biometric information. Specifically, in the present embodiment, for example, in a case where the following two conditions are satisfied, it is judged that this is a consent state for which consent is inferred, that is, there is an inferred consent, the two conditions being that the position of the imaging device 1 is close to the subject, and that the subject state is that the subject is facing forward and is smiling or is making a neutral facial expression.

[0093] Next, details of the condition for the inferred consent will be described.

[0094] Regarding the condition for the position of the imaging device, if the position of the imaging device is close to the subject, it can be inferred that this is not a state in which the image has been secretly taken from a distance and that the subject is able to know that their image is being taken, and thus this can become a basis for judging the inferred consent. That is, the positional relationship between the imaging device and the subject is made a condition. As a specific condition, the distance between the subject and the imaging device is calculated, and if the distance is less than a predetermined value, it is judged that the condition has been satisfied. The distance between the imaging device and the subject is measured by using a distance sensor built into the imaging device. The distance can also be calculated from the relationship between information from a known lens and sensor and the size of the imaged subject in the image.

[0095] Regarding the condition for the subject state, in a case where the subject is facing in the direction of the imaging device and the subject is making a certain expression or posture, it can be inferred that the subject is able to know that their image is being taken, and thus this can become a basis for judging the inferred consent. As a specific condition, the facial orientation and facial expression of the subject are detected from the image using a known image recognition technique, and in a case where the facial orientation is facing forward (that is, facing in the direction of the imaging device 1) and the facial expression is a smile or a neutral expression, it is judged that the condition has been satisfied. In this context, although the facial orientation and expression are combined to form one condition, they can also be used individually, and can also be combined with other conditions such as a posture. In addition, a predetermined posture (OK gesture, etc.) can also be set in advance in the biometric information processing device 100, and if it is judged that there is an inferred consent in a case where the posture can be detected, although the conditions for the inferred consent are limited, it is possible to reduce the possibility of erroneously judging the inferred consent.

[0096] In a case where both of the above two conditions are satisfied, the consent state inferring unit 32 infers that there is an inferred consent, and stores the value "Yes" in the inferred consent 51b of the corresponding record in the inferred consent information table 51 shown in FIG. 6. Figure 5 ​

[0097] Next, the consent information generating unit 35 of the biological information processing apparatus 100 generates consent information based on the value of the explicit consent acquired during S114 and the value of the presumed consent presumed during S115 (S116). The consent information is one of the three values of "disagreement", "presumed consent", and "explicit consent". In the case where the value of the explicit consent is the value "Yes" (there is explicit consent from the subject), the value of the consent information is made to be "explicit consent". Further, in the case where the value of the explicit consent is "No" (there is no explicit consent from the subject) and the value of the presumed consent is "Yes" (it is presumed that there is consent from the subject), the value of the consent information is made to be "presumed consent". Further, in the case where the value of the explicit consent is "No" (there is no explicit consent from the subject) and the value of the presumed consent is "No" (it is not presumed that there is consent from the subject), the value of the consent information is made to be "disagreement". The setting in this way means that the degree of explicit consent of the consent information regarding the biological information of the subject and the use of the biological information increases in the order of "disagreement", "presumed consent", and "explicit consent" (i.e., "disagreement" < "presumed consent" < "explicit consent").

[0098] Next, the biological information using unit 36 of the biological information processing apparatus 100 uses the biological information based on the restriction from the biological information use restricting unit 37 (S117). In this context, the use of the biological information is, for example, the use of the features that have been extracted during S113 and the use in the control of the imaging parameters of the imaging apparatus 1 (the automatic focus (AF) control and the automatic exposure (AE) control in this context). For example, the biological information use restricting unit 37 can perform the control of the imaging parameters so that the focus and the exposure are controlled with respect to the faces of the subjects for which the names have been specified. That is, the biological information use restricting unit 37 performs the control so that a person for whom the name has not been specified (a person for whom the use of the biological information has not been explicitly consented) is not made a reference object during the control of the imaging parameters (focus cannot be adjusted).

[0099] However, at this time, the imaging parameters are controlled so that, with respect to a person who has been judged to be "presumed consent" among the persons for whom the names have not been specified, the focus and the exposure are controlled in the same manner as in the case of explicit consent (focus can be adjusted).

[0100] The control of the focus and the exposure with respect to a specific person can be performed using known AF / AE technology. Note that, in this context, in the case where there are a plurality of persons for whom the names have been specified within the captured image, it is set that it is possible to instruct in advance with respect to which of these subjects the focus and the exposure are to be adjusted (for example, the priority of each person can be set).

[0101] Next, the biological information use unit 36 of the biological information processing apparatus displays the information of the AF / AE on the display panel 12 to be used as a result of using the biological information (S117).

[0102] Next, an example of displaying the AF / AE information on the display panel will be described using Figure 8

[0103] Figure 8 is a diagram showing an example of displaying the AF / AE information on the display panel.

[0104] In Figure 8 , the face frame G101 of the person who has been displayed as "unknown", the face frame G102 of the person who has been displayed as "Suzu◎ ▽fu", and the face frame G103 of the person who has been displayed as "●yama □ko" are arranged. Further, the "explicit consent" is combined and displayed as the value of the consent information of G102, and the "presumed consent" is combined and displayed as the value of the consent information of G103.

[0105] Although the face frame G101 is related to "Ta○ △shi", the use of the biological information for this person has not been given the explicit consent, and further it has been judged that there is no presumed consent, and therefore, this means that this person has not been designated. The face frame G102 is related to "Suzu◎ ▽fu", and as described above, "Suzu◎ ▽fu" has given the explicit consent to the use of the biological information, and therefore, by referring to the characteristics of "Suzu◎ ▽fu" from the individual registration table 50, it has been judged that this is "Suzu◎ ▽fu", and "Suzu◎ ▽fu" is displayed. Further, the face frame G103 is for "●yama□ko", and although they have not given the explicit consent, it has been judged that there is the presumed consent, and therefore, this means that the designation of the person using the characteristics has been performed, and the result thereof has been used for the control of the above-described photographing parameters and Figure 8 the display shown.

[0106] In this way, it is possible to restrict the use of the biological information by making it impossible to perform and use the designation of the person for whom the explicit consent has not been given. Further, it is possible to use the biological information by judging the presumed consent even in the absence of the explicit consent. That is, by the presumed consent with respect to the subject, it is possible to relax the restriction on the use of the biological information in the absence of the explicit consent. By doing so, in the absence of the explicit consent, it becomes possible to use the biological information without the need for a complicated explicit consent acquisition, and it is possible to improve the convenience for the user.

[0107] Note that, Figure 8 ​The display shown is a live view display on a digital camera, and the user can determine whether or not to record an image while looking at the display by pressing the shutter button 11.

[0108] Next, the biological information use unit 36 records the biological information based on the restriction of the biological information use restriction unit 37 (S118). The biological information in this context is the face image shown in the image, and is recorded on the nonvolatile memory 114 of the biological information processing device 100 so that the user can refer to the face image later. In this context, the biological information use unit 36 captures an image when the user presses the shutter button 11, associates the captured image with the consent information to use as the image data 61 of the image of the person, and stores it. At this time, control is performed so that the image can be recorded regardless of the consent information (that is, even if the consent does not exist). By doing so, it is possible to avoid a situation where a shutter opportunity is missed due to the lack of consent, and it is possible to improve the convenience for the user. Furthermore, this is recorded inside the device, and output to other devices and the like are not performed, so it is also easy to obtain the understanding of the subject.

[0109] For example, in the case of storing Figure 8 the image shown, as shown in the figure, the coordinate values of the square frame of the face frame G102 and the information of the name "Suzu◎ ▽fu" and the "explicit consent" for the consent information are also recorded as the image data 61 of the image. The information of the face frame G103 is also recorded as the image data 61 in the same manner. For the face frame G101, "unknown" showing that the name is not specified is recorded as the image data 61.

[0110] Thereafter, the biological information use unit 36 of the biological information processing device 100 determines whether or not the processing of using the biological information is completed (S119). In this context, for example, in the case where the user makes an input to end the processing of using the biological information, it can be determined that the processing will be completed. In the case where it has been determined that the processing of using the biological information will be completed during S119 (S119: Yes), the processing is completed, and in the case where it has not been determined that the processing of using the biological information will be completed during S119 (S119: No), the processing returns to S112.

[0111] Next, an example of a rule regarding the relationship between the use of the biological information and the consent information will be described using Figure 9

[0112] Figure 9 is a figure showing an example of a rule regarding the relationship between the use of the biological information and the consent information.

[0113] In Figure 9 ​In the illustrated rule, as to the content showing the use processing mode of the biological information, it is shown whether the use of the processing is permitted or not when the value of the consent information is each of "disagree", "presumed consent", and "explicit consent". That is, "OK" shows that the use of the biological information is permitted, and "NG" shows that the use of the biological information is restricted (cannot be executed).

[0114] The number 1-3 shows the restrictions that have been explained in this embodiment, and is all the processing that is executed within the imaging device 1. The number 1 shows the processing for "imaging" (recording inside the camera at the time of imaging), and as explained above, the recording is permitted regardless of the value of the consent information. The number 2 is the registration feature, and as explained above, this is restricted in the case where the value of the consent information is not "explicit consent" (when the value of the consent information is "disagree" and "presumed consent"). The number 3 is the processing that permits the use of AF / AE at the time of imaging to use the feature for specifying a person (that is, authentication using face recognition), and as explained above, it is shown that this is permitted only when the value of the consent information is "presumed consent" and "explicit consent". In this way, the degree of restriction relaxation is increased in accordance with the degree of explicit consent shown in the consent information. That is, in the related art, all the processing is restricted unless there is explicit consent, and in this embodiment, if there is presumed consent, the use of the result of specifying a person at the time of imaging is permitted, and the restriction is relaxed.

[0115] Note that the number 4 and thereafter will be explained in the second embodiment.

[0116] As explained above, the processing of the biological information processing device according to this embodiment can restrict the use of biological information for a person for which there is no explicit consent, while also relaxing the restriction by the presumed consent that has been presumed based on the acquisition state of the biological information to use the biological information. In particular, it is possible to realize both privacy protection and convenience to some extent by controlling the permission and restriction of each processing in accordance with the degree of explicit consent written in the consent information.

[0117] [Variant 1-1]

[0118] In the above-described first embodiment, although an example has been explained in which the imaging device position and the subject state, which are two biological information acquisition states, are used as the judgment condition to judge the presumed consent, other judgment conditions can also be used as long as they are conditions from which the consent of the subject can be presumed.

[0119] For example, as these other judgment conditions, the consent information of a person other than the person who is the subject (for example, a person who is in the image at the same time as the subject) can also be used. If the person in the image is close to another person who has given explicit consent, it can be judged that the subject next to that person is their friend or acquaintance, and it can be judged that they will have a high probability of giving the same consent, and thus this can be a basis for presuming consent. Whether the subjects in the image are close to each other is judged by detecting the faces of the people using a known face detection technique, and if the distance between the faces that have been detected in the image is closer than a predetermined distance, it is judged that they are close to the other subject in the image. As described above, the consent information is associated with the position of each face in the image and is stored as metadata, and thus it is sufficient to refer to this consent information. Another known scene estimation technique can also be used to judge whether people who are close together in the image know each other according to their positions in the image, and the consent information of these people is used. In addition, not only one captured image is used, but it is also possible to judge whether these people know each other by using captured images that have been taken in the past.

[0120] As another judgment condition, if the subject who is the object has presumed consent in a captured image that has been taken in the past, there is a high probability that there will also be presumed consent now, and thus this can also be a basis for judging presumed consent. If there is one captured image in which there was presumed consent in the past, it can be judged that the condition is satisfied. Instead of one image, if there are a plurality of images that are more than or equal to a predetermined number of images, it can also be judged that the condition is satisfied.

[0121] As another judgment condition, there is the type of the imaging device (that is, the hardware structure of the device in which the biological information acquisition unit has been incorporated). If the lens equipped in the imaging device is not a telephoto lens, but a lens that cannot image a face in a large size unless it is imaged at a close distance (such as a wide-angle lens, etc.), the subjects have a high probability of being able to know that they are being imaged, and this can become a basis for judging presumed consent. As a method of judging the type of the lens, it is sufficient to make it so that information for specifying the type of the lens is specified in advance in the lens, communication between the lens and the camera is performed when the lens has been connected to the camera, and the camera (that is, the imaging device) acquires the type of the lens from the lens. The information for specifying the type of the lens can be number information that is specific to a particular product and lens classification (such as wide-angle and telephoto, etc.) information.

[0122] Further, as another judgment condition, a camera parameter, that is, an acquisition parameter at the time of acquisition of the biological information, can also be used. For example, in a case where the ISO sensitivity at the time of imaging is low, it can be considered that the imaging is performed at a bright place, and the person can know with a high probability that their image is being taken. When the subject can know that their image is being taken, it is presumed that there is consent from the subject to the use of the biological information. Further, in a case where the focal length of the camera is short, it can be considered that the imaging must have been performed from a physically close distance to the subject, and the subject can know with a high probability that their image is being taken. Further, if the shutter speed is slow (compared to a case where the shutter speed is fast), the length of time associated with the imaging becomes long, and thus the subject can know that their image is being taken. If the condition is set such that the camera parameter is a value that is greater than or equal to a predetermined value and less than a predetermined value, such a camera parameter can be used as a judgment condition.

[0123] As another judgment condition, the imaging location (device usage location) can also be used. If the location of the device is a place of a specific event or the like, it can be presumed that the subject knows to some extent that their image is being taken, and thus this can be used as a basis for judging the presumed consent. Information of the imaging location (operation location of the imaging device) can be indicated as a place where the presumed consent can be assumed by installing a GPS (Global Positioning System) or the like in the biological information processing device and further recording map information, then indicating a predetermined location as an event place that has been indicated in advance, or causing a user of the imaging device 1 in which the biological information processing device 100 is built-in to register a home location. By doing so, it is possible to judge whether the location is an event place using the GPS at the time of imaging. Further, in a usage method such as a surveillance camera of a fixed location or the like, there is a possibility that this is not imaging that the subject desires, and thus if the location of the imaging device is in a state where it moves within a predetermined period of time, this can also be used as a basis for judging the presumed consent. That is, it is also possible to use the fixing method (mounting method) of the imaging device as a basis for the judgment.

[0124] Further, as the subject state, examples of the orientation of the subject, the expression of the subject, or the posture made by the subject are given in the present embodiment. However, in addition to these, it is also possible to use the result of whether there is past explicit consent for the subject. If the past explicit consent is used, the probability that the consent will continue to be shown is high, and thus this can also be used as a basis for judging the presumed consent. Therefore, it is possible to set a time usage limit for the explicit consent, and if the subject is associated with information of an expired explicit consent for the subject, it is possible to confirm and refer to the previous consent result.

[0125] In addition, as another state of the subject, whether or not the subject is photographing an own image can be made a condition. If a person is photographing an own image, it is clear that they themselves agree, and thus, this can also be used as a basis for judging the presumed agreement. As a method for judging whether or not the subject is photographing an own image, a gyro sensor can be built into the biological information processing apparatus 100, and this can be judged based on a tilt condition of the imaging device, it can be judged from the composition of the photographed image, such as the size of the face region of the subject in the photographed image, and it is also possible to combine these two methods.

[0126] In addition, in the first embodiment, although the presumed agreement is judged in a case where both the position of the imaging device and the subject state are satisfied, other conditions can also be used. It is also possible to judge the presumed agreement in a case where one of these conditions is satisfied, and a plurality of conditions for other acquisition states are also prepared, and the presumed agreement is judged in a case where a predetermined number of these conditions are satisfied.

[0127] An importance degree (a numerical value) can also be set for each condition, and the importance degrees of the conditions are added together in a case where the conditions are satisfied. By doing so, it becomes possible to presume an agreement state in which an important condition for deciding the presumed agreement has been emphasized.

[0128] Furthermore, for a subject who is currently attempting to photograph and for whom the presumed agreement is judged in an already photographed image, it is also possible to judge the presumed agreement by tracing back previously photographed images. By doing so, in a case where explicit agreement is not acquired with respect to an image that has been photographed in the past, it is possible to handle using the conditions for the presumed agreement, and it is possible to improve convenience.

[0129] [Variation 1-2]

[0130] Although in the present embodiment, only explicit agreement is used to restrict the extraction of features from biological information at the time of registering the individual's biological information, the presumed agreement can also be used at the time of registration. Specifically, even in a case where explicit agreement does not exist during the registration process of the individual's biological information, the features of the biological information can be registered by the presumed agreement state. By doing so, even in a case where explicit agreement does not exist at the time of registering the individual's biological information, the features can be used, and thus, it is possible to improve convenience.

[0131] [Variation 1-3]

[0132] Although in the first embodiment, the two values regarding whether or not there is the presumed consent from the provider of the biometric information are used to show the presumed consent information, there can be a plurality of values. For example, the number of conditions that have been satisfied in the conditions for deciding the presumed consent can also be made as the presumed consent value. By doing so, how high the accuracy of the presumed consent can be shown. The processing such as switching the processing for restriction and permission can also be performed according to the presumed accuracy. Further, as the plurality of conditions, a real value can also be calculated by using a predetermined calculation formula.

[0133] Further, although in the present embodiment, there are three values for the consent information, a different number of values can be used, for example, a real value can be used. In order to calculate the real value, it is enough to prepare a predetermined calculation formula that takes the explicit consent information and the presumed consent information as inputs and calculate the real value using the formula.

[0134] [Variant 1-4]

[0135] In the present embodiment, it has been explained that the consent of the object is acquired by displaying on the display panel 12 of the imaging device 1 a display indicating whether or not the object agrees with the display and the reception of the selection of the use of the biometric information (image prompting the consent). However, the present disclosure is not particularly limited to the method of acquiring the consent information as long as the presence or absence of the consent from the user can be confirmed. For example, the biometric information processing device 100 can also acquire the consent information based on the input in the application on the portable terminal such as a smartphone alone. In this case, when the biometric information processing device 100 communicates with the portable terminal and acquires the consent information (the explicit consent information and the presumed consent information) from the user, the biometric information processing device 100 can also acquire the consent information from the user by using the display panel 12 of the imaging device 1. Figure 6In step S103, a request for transmitting consent information can be sent to the portable terminal. Upon receiving the request, the portable terminal displays an image prompting consent on its screen, obtains consent information by receiving input indicating consent from the user of the portable terminal, and transmits the consent information to the biometric processing device 100. At this time, to detect fraud and prevent consent from being given by someone different from the user, the biometric processing device can also verify whether the person being registered is the same person as the user of the portable terminal (perform identity authentication). Therefore, user authentication can be performed on the portable terminal side, and the portable terminal's security functions can also be used for authentication. For example, a facial image of the person being registered by the biometric processing device 100 can be transmitted to the portable terminal, and the portable terminal can use known facial recognition technology to verify whether the person in the facial image is the same person as the user of the portable terminal. For example, the user's facial image can be captured by the portable terminal's front-facing camera. With this structure, fraudulent consent can be avoided, and consent information can be obtained from the recipient. Note that identity authentication is not limited to authentication using facial recognition technology, and it can also use any authentication technology already equipped on portable terminals (such as fingerprint authentication), and it can also be authentication processing using electronic certificates, etc.

[0136] [Second Embodiment]

[0137] Below, we will use Figure 10 and Figure 11 The second embodiment will now be described.

[0138] Although the first embodiment illustrates an example of processing biometric information from an image in a camera device using an internal biometric processing device 100 based on consent information, this embodiment will illustrate an example of outputting biometric information to an external device and using and processing the biometric information within the external device.

[0139] First, we will use Figure 10 This will illustrate the structure of an information processing system that uses biological information.

[0140] Figure 10 This is a structural diagram of an information processing system that uses biological information.

[0141] like Figure 10 As shown, the information processing system in this embodiment has the following structure: camera device 1, camera device 1a, portable terminal 7, and image management server 8 are connected via network 5. Each device is capable of requesting and transmitting and receiving data with other devices.

[0142] Camera device 1 has the features of the first embodiment Figure 2The illustrated structure, and is a device such as, for example, a digital camera, etc. that internally incorporates the biological information processing device 100.

[0143] The network 5 is a network for performing mutual communication between the devices. The network 5 can be, for example, a dedicated network such as a LAN (Local Area Network), and it can also be a global network such as the Internet. In addition, as a connection form of the network 5, wireless communication can be used, and wired communication can also be used.

[0144] The imaging device 1a is a second imaging device in the information processing system, and is a device such as a digital camera, etc. that is equipped with the same hardware structure and functional structure as the imaging device 1.

[0145] The portable terminal 7 is a portable information processing device, and is, for example, a smartphone and a tablet.

[0146] The image management server 8 is a server that stores images from the imaging device 1, manages the images, and provides data and information related to the images to other devices. In the present embodiment, the imaging device 1a, the portable terminal 7, and the image management server 8 are used as external devices.

[0147] In the present embodiment, first, an example in which biological information, features, and consent information are output from the imaging device 1 to the image management server 8 based on the consent information and the image management server 8 processes the biological information, the features, and the consent information will be described.

[0148] The image management server 8 provides an image management service, and the service provided is various services that store images that each user has transferred from other devices and share and manage the images according to a request from other devices.

[0149] Next, using Figure 11 and Figure 12 , a series of processes in which external processing utilizes biological information that has been obtained in the imaging device 1 in the information processing system illustrated in Figure 10 will be described.

[0150] Figure 11 is a flowchart illustrating a series of processes in which external processing utilizes biological information that has been obtained in the imaging device 1 in the information processing system of the second embodiment.

[0151] Figure 12 is a flowchart illustrating details of the use process of the biological information.

[0152] Figure 11 The process illustrated in is started at a time point at which the user of the imaging device 1 performs a command to output biological information, features, and consent information to the outside by operating the imaging device 1.

[0153] First, the Figure 3 The device authentication unit 38 of the biological information processing apparatus 100 shown in the drawing designates a device to which biological information and features are output (S151). Specifically, a user or a vendor of the imaging device 1 associates services that process biological information and features with devices that provide these services in advance and records them in the biological information processing apparatus 100 that has been built into the imaging device 1, and causes the user to select from among these services and devices. Thereby, an external device to which biological information and features are transmitted is designated. By causing the user to select in this way, the type of processing that the external device will perform on biological information and features can be designated. In the present embodiment, an image classification process using features that will be described below is instructed, and the image management server 8 that has been selected as the external device that will perform this process.

[0154] Next, the device authentication unit 38 of the biological information processing apparatus 100 performs an authentication process on the external device designated during step S151, that is, the image management server 8, via a network (S152). Specifically, the device authentication unit 38 transmits a request to permit data communication for biological information and features to the image management server 8. At this time, the image management server 8 can be authenticated by outputting a device identifier that can designate the imaging device 1 to the image management server 8. The device identifier is an identifier for designating a device, and the identifier can be prepared individually, and a pre-existing identifier can also be used. For example, information such as a manufacturing number and a MAC address of the device can also be used.

[0155] Next, the image management server 8 performs device authentication based on the device identifier that has been acquired (S153). Specifically, the image management server 8 permits data transmission and requests between the biological information processing apparatus 100 that has been built into the imaging device 1 and the image management server 8. As a structure for permitting, the image management server 8 stores a list of device identifiers that have been permitted in advance, and permits in the case where the device identifier that has been acquired is included in this list.

[0156] By performing device recognition in this way, the biological information processing apparatus 100 that has been built into the imaging device 1 can output biological information and features to the intended service and device that has been designated. That is, output to a non-intended service and device can be prevented, and therefore, unintended use of biological information can be prevented.

[0157] Next, the image management server 8 determines whether or not the device authentication during step S153 was successful (S154), and when the device authentication was successful (S154: Yes), the processing advances to S155. When the device authentication was not successful (S154: No), the processing is completed.

[0158] When the device authentication has succeeded, the image management server 8 creates authentication data, and transmits it to the device authentication unit 38 of the biometric information processing device 100 that has been built into the image pickup device 1 via the network 5 (S155). The authentication data is data that shows permission to receive data such as biometric information and features thereof from the biometric information processing device 100 that has been built into the image pickup device 1.

[0159] Next, the biometric information use unit 36 of the biometric information processing device 100 transmits the values of the individual registration table 50 shown in FIG. 8 to the image management server 8 on the condition that the device authentication unit 38 has received the authentication data (S156). The individual registration table 50 is a table in which names, face images, and features are associated with explicit consent information as explained in the first embodiment. In this context, the biometric information is a plurality of face images including the subject face. When the transmission is made, the biometric information use unit 36 makes the transmission to the image management server 8 together with the authentication data acquired from the device authentication unit 38. The image management server 8 permits the reception of the values of the explicit consent from the individual registration table 50 and the photographed images by checking the authentication data. Figure 4

[0160] Next, the image management server 8 receives the values from the individual registration table 50, and stores these values on an external device (not shown) such as an HDD and an SSD of the image management server 8 (S157).

[0161] Next, the biometric information use unit 36 of the biometric information processing device 100 requests the use processing of the biometric information that has been received from the image management server 8 during step S157 (S158). As described above, the available services are recorded in advance in the biometric information processing device 100, and the processing is requested to be executed by these services. The present disclosure can also take the form in which a plurality of services are prepared, and one or more of these services are selected.

[0162] Next, the image management server 8 executes the use processing for the biometric information for which the request has been received from the biometric information processing device 100 during step S158 (S200). In the present embodiment, for example, the present disclosure executes an image classification service using features (individual designation using features). At this time, the image pickup device 1 transmits the images that will be the targets of the image classification. Details of the use processing of the biometric information will be explained below using FIG. 9. Figure 12

[0163] ​​Next, the biological information use unit 36 of the biological information processing apparatus 100 acquires the processing result from S200, i.e., the metadata in which the individual has been specified, from the image management server 8, and displays these results (S160). In this context, a list of all people present in the metadata is displayed together with the face image, and the user is caused to select which person they want to display the image for, and the image of the selected person is displayed. By doing so, it is possible to perform an operation to display, process, edit, or delete the image in which only the specific person appears, and thus, it is possible to improve convenience.

[0164] Next, details of the biological information use processing will be explained using Figure 12 .

[0165] This is processing corresponding to S200 in the first embodiment. Figure 11

[0166] During the biological information use processing, the image management server 8 searches for a person matching the biological information (face image) stored in S157 from the same image transmitted during S158 later (S201). A face detection technique such as those explained in the first embodiment and authentication using a face recognition technique are used to specify who appears in which photo. The information of the person who has been specified is related to the image and stored as metadata.

[0167] Next, the image management server 8 performs the inference of the consent state of the subject appearing in the target image (S115a). This processing is the same as the processing of S115 in the first embodiment. Figure 7

[0168] Next, the image management server 8 performs the generation of the consent information based on the consent state that has been inferred during S115a (S116a). This processing is the same as the processing of S116 in the first embodiment. Figure 7

[0169] The values of the three types of "disagreement", "inferred consent", and "express consent", and the method for setting these values are the same as in the first embodiment.

[0170] Next, the image management server 8 extracts the specified person based on the consent information from S116a (S202). At this time, only the person for whom express consent exists in the individual registration table 50 and the person for whom the consent information from the photographed image is "inferred consent" and "express consent" are made the processing targets of the image management server 8. In this case, among the features in the individual registration table 50, only the features of the person for whom express consent exists can be used (S203). Figure 9 ​​​Thus, image classification is performed only on persons for which there is explicit consent. Persons for which there is presumed consent are not in the registration dictionary, and during image classification, they are grouped together and classified as other persons. By doing so, processing in the authenticated external device can be limited according to the consent information. Many people are anxious about the processing of biometric information by external devices, and thus, by excluding persons for which there is no explicit consent from the processing target, it is possible to achieve both privacy protection and convenience to some extent.

[0171] Next, referring back to Figure 9 Examples of rules regarding the relationship between the use of biometric information and consent information relating to the second embodiment will be described.

[0172] In this context, in the present embodiment, Nos. 4 to 6 will be described regarding the relationship between the use of biometric information and consent information in the external device.

[0173] No. 5 is a process that outputs to the indicated external device (i.e., an external device for which the authentication process has been completed), and is permitted regardless of the value of the consent information (even if it is "disagree"). No. 6 is a process of the indicated external device on biometric information, and is permitted when the value of the consent information is "presumed consent" and "explicit consent". In this way, the degree to which the restriction is relaxed increases in accordance with the degree of explicit consent indicated by the consent information. That is, although in the related art, if there is no explicit consent from the person providing the biometric information, all processing is restricted, in the present embodiment, when consent from the person providing the biometric information is presumed, processing in the external device is permitted, and the restriction is relaxed.

[0174] Further, in No. 4, there is no indication information processing device, that is, No. 4 is an output to an unintended external device / service, and is permitted only in the case where the value of the consent information is "explicit consent". Thereby, unintended use of biometric information is restricted.

[0175] According to the processing of the present embodiment, based on the consent information, by performing a process relating to biometric information for which there is consent from the subject in the external device (the image management server), it is possible to reduce the processing load of the biometric information processing device 100. At this time, information of presumed consent that has been presumed from the biometric information is also used, and the user of the biometric information processing device 100 can also use biometric information for which there is no explicit consent by limiting the processing performed, and it is possible to improve convenience. Further, the subject of the biometric information also does not need to perform complicated explicit consent, and thus, it is possible to improve convenience.

[0176] [Variant 2-1]

[0177] In the present embodiment, although an example in which the external device to which the biometric information is output is the image management server 8 has been explained, a different device can also be used as long as it can perform the same processing. For example, the imaging device la and the portable terminal 7 can also be used. Figure 10

[0178] [Variation 2-2]

[0179] In the present embodiment, although the biometric information is output to the external device for which the authentication has been completed regardless of the consent information, the biometric information that can be output can also be limited according to the consent information.

[0180] In addition, although in the present embodiment, control is performed to limit the object of the processing performed by the image management server 8 to the biometric information for which the consent information is "presumed consent" and "express consent", control other than this can also be performed. For example, instead of limiting the object of the processing according to the consent information (i.e., all biometric information will be the object of the processing), the processing can also be switched according to the consent information. For example, switching can be performed so that for biometric information having "disagreement", a masking process is performed, all other people appearing in the photograph are designated, and image classification is performed. By doing this, a complicated process of managing biometric information having "disagreement" can be performed automatically, and thus the convenience can be improved. Furthermore, a highly convenient process corresponding to the consent can be performed for a person who has given the consent, and the convenience can be improved while protecting privacy.

[0181] [Variation 2-3]

[0182] Although in the present embodiment, the biometric information and the feature acquired from the biometric information processing device for which the device authentication has been performed are processed by the external device (i.e., the image management server 8), the permission information for processing the biometric information can also be associated with the biometric information, and the processing can be performed by the image management server 8 according to the permission information. Specifically, the biometric information processing device 100 is also equipped with a permission information generation unit that functions as a functional configuration unit, and the permission information generation unit generates permission information for biometric information processing by the external device based on the consent information. In addition, the permission information that has been created is associated with the biometric information, and is output from the biometric information use unit to the external device. The image management server 8 processes the biometric information based on the permission information that has been acquired.

[0183] By associating the permission information with the biometric information in this way, the image management server 8 can judge whether it can process the biometric information and the feature without asking the biometric information processing device 100, and the degree of freedom of the processing timing is improved, and thus the convenience can be improved.

[0184] ​Further, the license information can also include information on services that the image management server 8 licenses to be executed. By doing so, in a case where a plurality of services have been prepared in the image management server 8, only the image management server 8 can grasp which processing is licensed to which biological information, and thus, the degree of freedom of processing timing is improved, and thus, convenience can be improved.

[0185] [Variant 2-4]

[0186] Further, although in the present embodiment, the unintended processing is not performed on the biological information by performing control so that the output is only performed to the authenticated external device, a technique such as encoding can also be introduced, and the use restriction can be made more accurate. Specifically, the biological information processing apparatus 100 is equipped with an encoding unit, and the biological information and the feature are output to the external device after having been encoded by the encoding unit. Further, the biological information processing apparatus 100 is equipped with a decoding key generation unit as a functional configuration unit, which generates a decoding key for decoding the encoded biological information and the feature. In addition, the decoding key generated by the decoding key generation unit is output from the biological information processing apparatus 100 to the external device, and when the external device is to use the biological information and the feature, they are decoded using the decoding key before they are used. By doing so, it is possible to prevent the data from being extracted from the biological information processing apparatus 100 and unintended use. In the previous embodiment, the imaging device 1 is a digital camera in which the biological information processing apparatus 100 is built-in, and there are many cases where the biological information is stored on a small portable medium such as a memory card or the like. In this case, even if the portable medium is extracted, if the person who extracted the medium cannot obtain the decoding key, they will not be able to use the biological information, and thus, unintended use of the biological information can be prevented.

[0187] [Variant 2-5]

[0188] Although in the present embodiment, the image classification has been given as an example of the service provided by the external device (i.e., the image management server 8), it can also be another service using other biological information and feature points.

[0189] For example, after the person has been specified from a plurality of images by using the registered feature, a process such as making it impossible to recognize the face of the person can also be performed based on the license information. As a method of making it impossible to recognize the face of the person, for example, methods such as masking the face, adding a mosaic, adding a blur filter, cropping the image so that the face is not shown, and the like are known.

[0190] As shown in Variations 1-3, in the case where a real value is used to realize the consent information, the processing parameter can also be controlled in accordance with the value of the consent information. For example, the strength of the blur filter can be adjusted, for example, in inverse proportion to the multiplication of the value of the consent information. By doing so, the amount of blur can be made smaller when the value of the consent information is high, and the amount of blur can be made larger when the degree of explicit consent is low, and privacy protection is automatically performed in accordance with the intention of the subject, and thus, control of the use of the biological information in accordance with the intention can be performed. As a substitute example of the blur filter, for example, a Gaussian filter can be used, in which the processing parameter thereof is the radius of the filter.

[0191] Further, even in the case where image classification is performed by a designated person, for example, the threshold value for judging that the person is the same person can be adjusted in inverse proportion to the multiplication of the value of the consent information. In this context, the value for judging that the person is the same person takes the distance of each pixel in the image, and becomes larger in different images. By doing so, the person whose value of the consent information is smaller will have a higher possibility of being misclassified as a different person, and the image of the person will have a lower possibility of being correctly classified. Therefore, the proportion expressed as the classification result decreases, and this is related to privacy protection for the person whose value of the consent information is smaller. Conversely, the higher the value of the consent information that the person has, the lower the possibility that the person will be misclassified as another person, and the higher the possibility that the photograph of the person will be correctly classified, and thus, even if the concern for privacy is high, the biological information can be used in the process that values the classification of the image of the person.

[0192] [Other Embodiments]

[0193] Although in the first and second embodiments, an example is explained in which a facial image is acquired to be used as the biological information, it is not particularly limited to an image including their face, as long as the biological information includes the information of the user. For example, the biological information can also be an image including information that can specify the user thereof, such as the iris, fingerprint, vein, and the like of an individual. Even in the case where such biological information is used, a feature can be extracted from the biological information in the same manner. Note that a camera device including an appropriate sensor corresponding to each type of biological information can also be used as the device that acquires the photographed image. For example, in the case where the subject of the photographed image of the user of the biological information is related to other biological information of the individual, such as the iris, fingerprint, and vein, and the like, and the consent is inferred from the state of the image, the other biological information can also be made usable.

[0194] [Variations of the Biological Information Processing Device]

[0195] Although it has been explained that the biological information processing apparatus 100 according to the present embodiment is incorporated into the image pickup apparatus 1 (digital camera), the structure of the biological information processing apparatus 100 is not particularly limited to this structure as long as it can perform the same processing. For example, the biological information processing apparatus 100 can also be a smartphone having a camera, and can also be incorporated into a network camera equipped with a pan / tilt / zoom function capable of adjusting an image pickup angle of view. In this case, as a method as a result of designating a person, a method of controlling pan / tilt / zoom to capture an image of a designated individual is assumed. In a case where the biological information processing apparatus 100 is used for this purpose, privacy invasion can be suppressed by controlling the use of biological information with respect to a person who has not given consent, and misuse of the biological information processing apparatus 100 can be prevented. In addition, in a case where the biological information processing apparatus 100 is incorporated into the network camera, a separate server that manages the network camera and records a captured video image can also be prepared, and the acquisition and registration processing of consent information can be performed by the server. In such a structure, when consent is being acquired from a user, a UI for acquiring consent information on the server, that is, a display of an effect for which biological information will be used and a display of a selection of whether the user consents to the use, is displayed on a display apparatus, and it can be made so that whether the user consents is acquired by the user inputting using an input device such as a mouse and a keyboard.

[0196] It has been explained that, in the biological information processing apparatus of the present embodiment, the CPU reads out and executes a program that has been installed on the nonvolatile memory. However, the present disclosure can also be implemented using a hardware circuit (for example, an ASIC) that implements functions such as those functions explained in the first embodiment. Figure 3

[0197] Other Embodiments

[0198] Embodiments of the present invention can also be implemented by a method in which software (a computer program product including a computer program / instruction) that performs the functions of the above-described embodiments is provided to a system or an apparatus via a network or various storage media, and a computer (central processing unit (CPU), micro processing unit (MPU)) of the system or the apparatus reads and executes the computer program / instruction.

[0199] While the present disclosure has been described with reference to embodiments, it is to be understood that the present disclosure is not limited to the disclosed embodiments. The scope of the following claims is to be construed in its broadest interpretation, taking into account all such modifications and equivalent structures and functions.

[0200] This application claims the benefit of Japanese Patent Application No. 2024-175520, filed October 7, 2024, which is hereby incorporated by reference herein in its entirety.​

Claims

1. A bioinformatics processing device configured to process human bioinformatics, the bioinformatics processing device comprising a processor and a non-transitory computer-readable storage medium storing instructions, the instructions causing the processor, when executed by the processor, to: Acquire the biological information; Using the biological information and the features extracted from the biological information; Obtain consent regarding the use of the biometric information of persons associated with the biometric information; The use of the biometric information may be restricted based on the value of the consent expressed. Consent status is presumed based on the acquisition status at the time the biometric information is acquired, the consent status indicating consent in relation to the use of the biometric information of the person associated with the biometric information; as well as Based on the presumed state of consent, restrictions related to the use of biometric information of persons associated with the biometric information and features extracted from the biometric information are relaxed.

2. The bioinformatics processing device according to claim 1, wherein, The bioinformatics processing device presumes the consent status of the person based on at least one or a combination of the following conditions: The locational relationship between the device with the bio-information processing equipment and the person has been incorporated when acquiring the bio-information. The state of the person at the time the biometric information is acquired; Biometric information and consent information of persons other than the persons mentioned above, in addition to the biometric information mentioned above; The hardware structure of the device that has been incorporated with the aforementioned bioinformatics processing equipment; Acquisition parameters when acquiring the biological information; The biometric information obtained is the person's past consent information; as well as The operating location of the device incorporating the bioinformatics processing equipment, and the installation method of the device incorporating the bioinformatics processing equipment.

3. The bioinformatics processing device according to claim 1, wherein, The biometric information includes images of human faces.

4. The bioinformatics processing device according to claim 1, wherein, The bio-information processing device is integrated into the camera device, and the images captured by the camera device become the input to the bio-information processing device.

5. The bioinformatics processing device according to claim 1, wherein, The bioinformatics processing device generates consent information that indicates the degree of explicit consent based on the value of consent and the presumed state of consent; The greater the degree of explicit consent, the higher the relaxation becomes.

6. The bioinformatics processing device according to claim 5, wherein, The bio-information processing device authorizes the display of a person's bio-information and features extracted from that bio-information based on the degree of explicit consent expressed in the consent information.

7. The bioinformatics processing device according to claim 5, wherein, The bioinformatics processing device authorizes the output of the person's bioinformatics and features extracted from the bioinformatics to external devices based on the degree of explicit consent indicated by the consent information.

8. The bioinformatics processing device according to claim 7, wherein, The bioinformatics processing device generates permission information for the external device to use the bioinformatics and features extracted from the bioinformatics; as well as The license information is output to the external device.

9. The bioinformatics processing device according to claim 8, wherein, The bioinformatics processing device outputs the permission information, the consent information, the bioinformatics, and the features extracted from the bioinformatics to an external device, wherein the external device has already undergone authentication processing.

10. The bioinformatics processing device according to claim 9, wherein, The bioinformatics processing device permits output to the external device that has already undergone the authentication process, provided that the explicit consent expressed in the consent information is greater than the presumed consent state.

11. The bioinformatics processing device according to claim 10, further comprising: An encoding unit is configured to encode the biological information and features extracted from the biological information; as well as A decoding key generation unit is configured to generate a decoding key used to decode encoded biological information and features extracted and encoded from the biological information, wherein... The decoding key, the encoded biological information, and the features extracted and encoded from the biological information are output to the external device.

12. An information processing system, comprising: Bioinformatics processing equipment, configured to process human bioinformatics; and external devices connected to the bioinformatics processing device and configured to receive and use bioinformatics generated by the processing of the bioinformatics, wherein the bioinformatics processing device: Acquire the biological information; as well as Obtain consent regarding the use of the biometric information of persons associated with the biometric information; as well as The external device presumes a consent state based on the acquisition state when acquiring the biometric information, and the consent state indicates consent related to the use of the biometric information of the person associated with the biometric information; as well as The external device, based on a presumed state of consent, relaxes restrictions related to the use of the biometric information of the person associated with the biometric information and features extracted from the biometric information.

13. A bioinformatics processing method using a bioinformatics processing device, the bioinformatics processing device being configured to process bioinformatics of a person; the bioinformatics processing method comprising: A bioinformatics acquisition step, wherein the bioinformatics processing device acquires human bioinformatics; A bioinformatics usage step, wherein the bioinformatics processing device uses the bioinformatics and features extracted from the bioinformatics; An explicit consent acquisition step, wherein the bioinformatics processing device acquires the person’s consent status in relation to the use of the bioinformatics information in the bioinformatics use step; A bioinformatics usage restriction step, wherein the bioinformatics processing device restricts the use of the bioinformatics during the bioinformatics usage step based on the value of the consent; as well as A consent state estimation step, wherein the bioinformatics processing device estimates a consent state based on the acquisition state at the time of acquiring the bioinformatics, the consent state representing consent in relation to the use of the bioinformatics of a person associated with the bioinformatics; The biometrics usage restriction step relaxes restrictions on the use of the biometrics of persons who have indicated consent under a presumed consent state and on features extracted from the biometrics.

14. A non-transitory computer-readable storage medium storing a program that causes a computer to perform the method according to claim 13.

15. A computer program product comprising a program that causes a computer to perform the method according to claim 13.

Images