Personal authentication system, personal authentication method, program, and system employing personal authentication system

Abstract

Provided are a personal authentication system, etc., capable of guaranteeing or improving authentication performance, and capable of improving convenience, for example. The personal authentication system is provided with a sensor 20 that is inserted into the external ear hole of the ear of a user and that acquires a plurality of pieces of image data relating to the inside of the external auditory canal. The personal authentication system is also provided with a personal authentication device 10 comprising: feature extracting units 105, 106 that extract features using the plurality of pieces of image data acquired by the sensor 20 as training data; a feature integrating unit 107 that generates a registration model capable of identifying the user from the obtained features; and an authenticating unit 109 that uses the registration model to authenticate the user on the basis of a plurality of pieces of image data acquired by the sensor 20.

Description

Personal authentication system, personal authentication method, program, and system using the personal authentication system

[0001] The present invention relates to a personal authentication system, a personal authentication method, a program, and a system using the personal authentication system, which perform personal authentication based on an image of the inside of the ear canal.

[0002] In recent years, biometric authentication technology has been used in a variety of fields. Biometric authentication is a technology that authenticates individuals based on biometric characteristics, such as physical and behavioral characteristics. The main physical characteristics used in biometric authentication include fingerprints, retinas, irises, palm shapes, and faces. The main behavioral characteristics used in biometric authentication include handwriting, voice, and gait. In situations where biometric authentication is used, users do not need to carry keys to open safes or ID cards to present when entering and exiting. Therefore, there is no risk of keys or ID cards being lost, stolen, or counterfeited. There is also no need to remember IDs (accounts) or passwords. Therefore, biometric authentication is known to be a highly convenient and secure technology, making it extremely useful.

[0003] Here, prior art related to biometric authentication that focuses particularly on the ear includes a personal authentication device described in Patent Document 1. The personal authentication device described in Patent Document 1 acquires multiple different pieces of biometric information from an ear canal, extracts features of the multiple different pieces of biometric information using machine learning, and performs personal authentication. This personal authentication device also includes a feature extraction unit that extracts features based on data acquired from two or more sensors, including a first sensor that acquires acoustic data from the user's ear canal, a second sensor that acquires image data from the user's ear canal, and a third sensor that acquires biochemical data from the user's ear canal, and a determination unit that builds a personal integration neural network based on the two or more features extracted by the feature extraction unit and determines the user based on the integrated data.

[0004] JP 2023-148128 A

[0005] The personal authentication device described in Patent Document 1 can easily combine multiple closely related authentication methods compared to conventional biometric authentication methods, which can significantly improve authentication performance. Furthermore, even if the usage environment or the individual's physical condition changes, different authentication methods can complement each other, which is expected to improve robustness.

[0006] However, in biometric authentication technology, there is a demand for guaranteeing stable authentication performance that is not affected by the surrounding environment, improving authentication performance, improving convenience for users, etc. Therefore, the present invention provides a personal authentication system, a personal authentication method, a program, and a system using the personal authentication system that can guarantee or improve authentication performance or improve convenience, etc.

[0007] The personal authentication system of the present invention is a personal authentication system that performs personal authentication based on a captured image of the inside of the ear canal, and includes: a first sensor that is inserted into the ear canal of the user's right ear and acquires image data inside the ear canal, and / or a second sensor that is inserted into the ear canal of the user's left ear and acquires image data inside the ear canal; a personal authentication device that can communicate with the first sensor and / or the second sensor, and includes a first feature extraction unit that extracts first features using image data acquired by the first sensor as learning data, and / or a second feature extraction unit that extracts second features using image data acquired by the second sensor as learning data; a feature integration unit that generates a registered model that can identify the user from integrated features obtained from the first features and / or the second features; and an authentication unit that uses the registered model to authenticate the user based on the image data acquired by the first sensor and / or the second sensor.

[0008] Here, it is preferable that the first sensor and / or the second sensor have a video recording function, and the authentication unit is configured to authenticate the user based on multiple image data in the video taken by the first sensor and the second sensor from the entrance of the ear canal to near the eardrum.

[0009] Furthermore, it is preferable that the first sensor and / or the second sensor have a light that illuminates the inside of the ear canal, and the personal authentication device has an image determination unit that determines whether the image data acquired by the first sensor and / or the second sensor is unclear, and that the image determination unit is configured to control the direction of illumination of the light of the first sensor and / or the second sensor if it determines that the image data is unclear.

[0010] Alternatively, it is preferable that the first sensor and / or the second sensor have a call function, the personal authentication device has an image determination unit that determines whether the image data acquired by the first sensor and / or the second sensor is unclear, and if the image determination unit determines that the image data is unclear, it notifies the user via the first sensor and / or the second sensor that the wearing position of the first sensor and / or the second sensor should be moved.

[0011] Alternatively, it is preferable that the personal authentication device has an image determination unit that determines whether the image data acquired by the first sensor and / or the second sensor is unclear, and an image processing unit that removes noise contained in the image data if the image determination unit determines that the image data is unclear.

[0012] Alternatively, it is preferable that the first sensor and / or the second sensor acquire image data from inside the ear canal at a predetermined timing and / or at a timing instructed by an external device, and the personal authentication device updates the registered model based on the integrated features obtained based on the image data.

[0013] Alternatively, the personal authentication device preferably has an image determination unit that determines whether or not earwax and / or ear hair is included in the image data acquired by the first sensor and / or the second sensor, and an image processing unit that removes the earwax and / or ear hair included in the image data when the image determination unit determines that the image data contains earwax and / or ear hair.

[0014] Alternatively, the personal authentication device preferably has an image determination unit that determines whether or not earwax and / or ear hair is contained in image data acquired by the first sensor and / or the second sensor, and the first feature extraction unit, the second feature extraction unit, and the feature integration unit are configured such that, when the image determination unit determines that earwax and / or ear hair is contained in the image data, they obtain integrated features based on the image data, and the authentication unit uses a registration model generated from the integrated features to authenticate the user based on the image data acquired by the first sensor and the second sensor that contains earwax and / or ear hair.

[0015] Alternatively, the personal authentication device preferably has an image determination unit that determines whether or not earwax and / or ear hair is included in image data acquired by the first sensor and / or the second sensor, and an image processing unit that processes the image data so that earwax and / or ear hair is included in the image data if the image determination unit determines that the image data does not include earwax and / or ear hair, and the first feature extraction unit, the second feature extraction unit, and the feature integration unit are configured to obtain integrated features based on the image data processed by the image processing unit and generate a registration model.

[0016] It is preferable that the first sensor and / or the second sensor have a main body, a camera, and a protruding base attached to the main body and connecting the main body and the camera, and that the attachment position of the base be changeable depending on the shape of the user's ear canal and / or the position of the eardrum.

[0017] Furthermore, the personal authentication system of the present invention is a personal authentication system that performs personal authentication based on a captured image of the inside of the ear canal, and includes: a first sensor that is inserted into the ear canal of the user's right ear and acquires image data inside the ear canal; a second sensor that is inserted into the ear canal of the user's left ear and acquires image data inside the ear canal; a personal authentication device that can communicate with the first sensor and the second sensor, and includes: a first feature extraction unit that extracts first features using image data acquired by the first sensor as learning data; a second feature extraction unit that extracts second features using image data acquired by the second sensor as learning data; a feature integration unit that generates a registered model that can identify the user from integrated features obtained by integrating the first and second features; and an authentication unit that authenticates the user based on the image data acquired by the first and second sensors using the registered model.

[0018] These personal authentication systems are preferably used in any one of a medical information system, a health checkup system, an admission management system, an automatic withdrawal system, and a delivery management system.

[0019] On the other hand, the personal authentication method of the present invention is a personal authentication method for authenticating a person based on an image captured inside the ear canal, and includes the steps of acquiring image data inside the ear canal with a first sensor inserted into the ear canal of the user's right ear and / or acquiring image data inside the ear canal with a second sensor inserted into the ear canal of the user's left ear; extracting, by the personal authentication device, a first feature amount using the image data acquired by the first sensor as learning data and / or extracting a second feature amount using the image data acquired by the second sensor as learning data; generating, by the personal authentication device, an enrollment model capable of identifying the user from integrated features obtained from the first feature amount and / or the second feature amount; and authenticating, by the personal authentication device, the user based on the image data acquired by the first sensor and / or the second sensor using the enrollment model.

[0020] The program of the present invention is also a program for causing a computer to operate as a personal authentication device capable of communicating with a first sensor that is inserted into the external auditory canal of a user's right ear and acquires image data within the external auditory canal and / or a second sensor that is inserted into the external auditory canal of the user's left ear and acquires image data within the external auditory canal, the personal authentication device having: a first feature extraction unit that extracts first features using image data acquired by the first sensor as learning data and / or a second feature extraction unit that extracts second features using image data acquired by the second sensor as learning data; a feature integration unit that generates a registered model capable of identifying the user from integrated features obtained from the first features and / or the second features; and an authentication unit that uses the registered model to authenticate the user based on the image data acquired by the first sensor and / or the second sensor.

[0021] The personal authentication system of the present invention can provide a personal authentication system that can guarantee or improve authentication performance, or improve convenience, etc. Furthermore, the personal authentication method or program of the present invention can achieve the same effects as the personal authentication system of the present invention.

[0022] FIG. 1 is a schematic overall configuration diagram of a personal authentication system according to an embodiment of the present invention. FIG. 2 is a schematic functional block diagram of a personal authentication device according to an embodiment of the present invention. FIG. 3 is a diagram showing a sensor according to an embodiment of the present invention, where (A) is a schematic configuration diagram and (B) is a schematic configuration diagram of an insertion part as seen from the direction of the arrow shown in (A). FIG. 4 is a diagram showing an image of using a sensor according to an embodiment of the present invention. FIG. 5 is a flow diagram during registration of a personal authentication method according to an embodiment of the present invention. FIG. 6 is a flow diagram during registration of a personal authentication method according to an embodiment of the present invention. FIG. 7 is a flow diagram during authentication of a personal authentication method according to an embodiment of the present invention.

[0023] The following describes in detail an embodiment of the present invention. However, the description of the constituent elements described below is an example (representative example) of an embodiment of the present invention, and the present invention is not limited to the following content unless the gist of the present invention is changed.

[0024] [Personal authentication system] First, a personal authentication system according to an embodiment of the present invention will be described with reference to Figures 1 to 4. The personal authentication system 1 according to the embodiment of the present invention includes a first sensor 20 (20R), a second sensor 20 (20L), and a personal authentication device 10 (see Figure 1).

[0025] [1] Sensor The sensor 20 is a sensor that acquires image data from inside the ear canal of a user (a person using the personal authentication system 1). In this embodiment, the sensor 20 is an earphone type sensor that is inserted into the external ear canal of the ear Y of the user U (see FIGS. 1, 3, and 4). For convenience in this description, the sensor 20 inserted into the external ear canal of the user's right ear will be referred to as the first sensor 20R, and the sensor 20 inserted into the external ear canal of the user's left ear will be referred to as the second sensor 20L.

[0026] The sensor 20 is, for example, an infrared sensor. The sensor 20 acquires image data from within the ear canal, including an image of the shape of the ear canal (an image showing the shape of the ear canal), an image of the vascular network within the ear canal, and an image showing the temperature distribution within the ear canal. Pulse rate estimation data specific to an individual acquired using analytical techniques such as an RGB camera or a thermal camera can also be used. The sensor 20 can also be a CCD camera or a CMOS sensor.

[0027] Here, the sensor 20 of this embodiment has a main body (control unit) 21, a communication unit 22, and an insertion unit 23 (see FIG. 3A). The insertion unit 23 is provided with a camera 24 and multiple lights 25 (see FIG. 3B). The sensor 20 also has a base 26 attached to the main body, and in this embodiment, the base 26 is a protrusion that connects the main body 21 and the camera 24. The attachment position of this base 26 can be changed depending on the shape of the inside of the ear canal and / or the position of the eardrum of the user U wearing the sensor 20.

[0028] The main body unit 21 performs various controls such as obtaining image data using the camera 24 and transmitting the captured image data using the communication unit 22. The communication unit 22 communicates with external devices such as the personal authentication device 10 and the terminal 30 via the network N.

[0029] The insertion unit 23 is a part that is inserted into the ear canal, and as described above, is provided with the camera 24 and the light 25. Furthermore, when the insertion unit 23 is inserted into the ear canal, it has a function (for example, LiDAR: Light Detection and Ranging) to acquire the distance from the camera 24 to the eardrum and a three-dimensional scan image (see FIG. 4B ).

[0030] The number and positions of the cameras 24 provided in the insertion section 23 can be arbitrarily modified, and the imaging direction of the cameras 24 can also be arbitrarily changed. For example, by changing the orientation of the cameras 24 in response to control instructions from the main body 21, the imaging location within the external ear canal can be arbitrarily changed. Of course, the cameras 24 also have functions such as zooming in and out. Similarly, the number and positions of the lights 25 provided in the insertion section 23 can also be arbitrarily modified, and the irradiation direction of the lights 25 can also be arbitrarily changed. In addition, the light intensity, light color (white, orange, red, purple), or light type (infrared / ultraviolet) of the lights 25 can be changed in response to control instructions from the main body 21.

[0031] In this case, it is preferable to illuminate the light 25 at a wavelength that allows the camera 24 to exhibit its imaging capabilities. Furthermore, as in this embodiment, the light 25 can be formed from multiple pieces, and the pieces can be arranged in a circular pattern. To optimally illuminate the interior of the ear canal with the camera 24, some pieces can be controlled to illuminate and some cannot. As an example, all pieces can be first illuminated. If the illumination is too bright and causes problems with imaging with the camera 24, one or more pieces can be turned off to adjust for better imaging. Conversely, one or more pieces can be turned on sequentially to adjust for better imaging. Furthermore, although the light 25 in this embodiment is composed of multiple pieces, it may also be configured as a continuous ring. Although not shown, the insertion section 23 can be fitted with flexible rubber or the like on the outside of the light 25 to improve fit to the ear, or the individual pieces of the light 25 can be configured to be embedded in a flexible material such as rubber.

[0032] The sensor 20 (camera 24) has a video capture function and captures video from the entrance of the ear canal to the vicinity of the eardrum. The sensor 20 is connected to the personal authentication device 10 and the terminal 30 via a network N (see FIG. 1 ) by wire or wirelessly, and image data (information) including the video captured by the sensor 20 is transmitted to the personal authentication device 10 and the terminal 30 via the network N. In addition, in this embodiment, video from the entrance of the ear canal to the vicinity of the eardrum is transmitted to the personal authentication device 10. However, still images captured by the sensor 20 may also be transmitted in addition to the video. In this case, it is preferable to transmit only the video, or still images in addition to the video, depending on the authentication specifications. Furthermore, as described below, the sensor 20 itself can extract features from the video or still images and transmit the features to the personal authentication device 10.

[0033] Additionally, the sensor 20 has the functions of a typical earphone, such as a speaker and a microphone, so that it can transmit voice from an external device to a user wearing the sensor 20, or transmit the user's voice to an external device. The sensor 20 can also be provided with a noise canceller function.

[0034] [2] Personal authentication device The personal authentication device 10 is a device that performs personal authentication based on image data of the inside of the ear canal acquired by the sensor 20, and has a receiving unit 101, a transmitting unit 102, an image determining unit 103, an image processing unit 104, a first feature extraction unit 105, a second feature extraction unit 106, a feature integration unit 107, a storage unit 108, an authentication unit 109, and a control unit 110 (see FIG. 2). In this description, the first feature extraction unit 105 and the second feature extraction unit 106 will each be simply referred to as the "feature extraction unit," and these will be collectively referred to as the "feature extraction unit."

[0035] The receiving unit 101 and the transmitting unit 102 communicate with external devices such as the sensor 20 and the terminal 30 via the network communication network N.

[0036] The image determination unit 103 makes various determinations as to whether the image data of the inside of the ear canal acquired by the sensor 20 is unclear. For example, the image determination unit 103 determines that the image data is unclear in the following cases: (1) The acquired image data is too dark / too bright (insufficient amount of light / excessive amount of light, etc.) (2) The location of the acquired image data is inappropriate (the insertion unit 23 is not inserted fully into the ear canal and only the entrance of the ear canal is captured, the insertion unit 23 is inserted at an angle, etc.) (3) The acquired image data contains noise such as blown-out highlights and crushed shadows (shadows) (4) The acquired image data contains earwax, ear hair, etc.

[0037] Furthermore, if the image determination unit 103 determines that the image data is unclear due to the factor (3) above, the image processing unit 104 removes noise contained in the image data. Note that the handling of image data determined to be unclear due to other factors will be described later.

[0038] The first feature extraction unit 105 extracts first features from image data acquired by the first sensor 20R as training data. Meanwhile, the second feature extraction unit 106 extracts second features from image data acquired by the second sensor 20L as training data. Furthermore, the feature integration unit 107 integrates the first features and the second features to obtain integrated features that can identify the user. The feature integration unit 107 generates a registered model from the integrated features. The registered model links the integrated features with the user's personal information (at least one of name, age (date of birth), height, weight, address, telephone number, company name, department name, job title, occupation, etc.), and the generated registered model is stored in the storage unit 108.

[0039] The storage unit 108 stores various information necessary for the personal authentication system 1, such as registered models, user information, image data inside the ear canal acquired by the sensor 20, first feature amounts, second feature amounts, integrated feature amounts, and setting files. The storage unit 108 may be configured as a part of the personal authentication device 10, such as an HDD or SSD, or may be an external storage (for example, a cloud-based database).

[0040] The authentication unit 109 uses (references) the registered model stored in the storage unit 108 to authenticate (identify) the user based on the image data acquired by the first sensor 20R and the second sensor 20L.

[0041] In short, the personal authentication device 10 according to this embodiment authenticates (identifies) a user using an enrollment model generated from a feature (integrated feature) obtained by integrating a feature (first feature) based on image data (shape feature) inside the ear canal of the right ear of the user U and a feature (second feature) based on image data (shape feature) inside the ear canal of the left ear of the user U. Of course, it is also possible to generate an enrollment model from only the first feature or only the second feature.

[0042] The control unit 110 controls the personal authentication device 10 itself and external devices such as the sensor 20 and the terminal 30 .

[0043] [3] Terminal The terminal 30 is a personal computer, smartphone, tablet, or the like owned by the user U (see FIG. 1 ). The terminal 30 communicates with external devices such as the personal authentication device 10 and the sensor 20 via the network communication network N.

[0044] The personal authentication system 1 of this embodiment has been described above, but the personal authentication device 10, sensor 20, and terminal 30 of the personal authentication system 1 can also communicate with each other without going through the network communication network N. For example, the sensor 20 and the terminal 30 can communicate with each other by short-range wireless communication (Bluetooth, etc.).

[0045] 5 to 8, a personal authentication method according to an embodiment of the present invention using the personal authentication system 1 of this embodiment will be described. The personal authentication method of this embodiment will be described separately for registration (registration phase) and authentication (authentication phase).

[0046] [1] Registration (Normal) A user of the personal authentication system 1 first registers features (registration model) based on his or her own left and right ears together with the personal information described above in the personal authentication device 10 (see FIG. 5).

[0047] For example, when user U1 (see Figure 1) wears his / her first sensor 20R on his / her right ear and his / her second sensor 20L on his / her left ear, the sensors 20 (20R, 20L) acquire the distance from the camera 24 to the eardrum (see Figure 4 (B)) (step S110).

[0048] Then, the sensor 20 (20R, 20L) emits light 25 and captures a video (video for registration) inside the ear canal with the camera 24 (step S120).

[0049] Thereafter, the sensor 20 (20R, 20L) transmits the captured video to the personal authentication device 10 via the communication unit 22 (step S130).

[0050] Meanwhile, the personal authentication device 10 receives the video using the receiving unit 101, and uses multiple image data in the received video as learning data to obtain integrated features that can identify user U1 using the first feature extraction unit 105, the second feature extraction unit 106, and the feature integration unit 107 (step S210).

[0051] Finally, the personal authentication device 10 generates an enrollment model from the integrated features by the feature integration unit 107 and stores it in the storage unit 108 (step S220).

[0052] With regard to this registered model, the personal authentication system 1 allows the sensor 20 to reacquire image data from inside the ear canal at a predetermined timing, when an instruction is received from an external device (such as the personal authentication device 10 or the terminal 30), or when the sensors 20 (20R, 20L) are attached to both ears, and the personal authentication device 10 can regenerate (update) the registered model from the reacquired image data. The predetermined timing is a timing managed by the main body 21 of the sensor 20 or the memory unit 108 of the personal authentication device 10, such as once a day (when attached for the first time that day), once a week (at the beginning of the week), or once a month (at the beginning of the month).

[0053] [2] Authentication (Normal) After completing registration, user U1 performs authentication using the personal authentication system 1 when necessary (see FIG. 7). In this explanation, a necessary case refers to when user U1 wants to log in to his / her own terminal 301 (personal computer). Note that only user U1 can log in to terminal 301 (no login authority).

[0054] In this case, user U1 wears his / her own first sensor 20R on his / her right ear and his / her own second sensor 20L on his / her left ear. Then, as in the registration phase, the sensors 20 (20R, 20L) acquire the distance from the camera 24 to the eardrum, illuminate the light 25, and capture a video of the inside of the ear canal with the camera 24, and transmit the captured video (authentication video) to the personal authentication device 10 (steps S310 to S330).

[0055] At this time, the sensor 20 also transmits information about the terminal 30 and an authentication request for user U1 to the personal authentication device 10. For example, the sensor 20 acquires information about the terminal 30 by short-range wireless communication, and transmits an authentication request for user U1 to log in to the terminal 30 and the captured video (authentication video) to the personal authentication device 10. The authentication request for user U1 to log in to the terminal 30 may be transmitted from the terminal 30 to the personal authentication device 10.

[0056] Meanwhile, the personal authentication device 10 receives the video using the receiving unit 101, and based on the multiple image data in the received video, the first feature extraction unit 105, the second feature extraction unit 106, and the feature integration unit 107 obtain integrated features that can identify user U1 (that can determine whether or not the user is U1) (step S410).

[0057] Then, the personal authentication device 10 compares the integrated feature obtained in step S310 with the registered model stored in the storage unit 108 by the authentication unit 109, and authenticates the user (step S420).

[0058] Finally, the personal authentication device 10 notifies the result (authentication result) via the transmission unit 102 (step S430). For example, if the recognition result is OK, the personal authentication device 10 can transmit to the terminal 301 an instruction to log in with the account of user U1. Conversely, if the recognition result is NG, the personal authentication device 10 can transmit to the terminal 301 an instruction to display an error message. Alternatively, the personal authentication device 10 can notify via the sensor 20 of a beep or a message indicating an error.

[0059] Therefore, when user U2 wears the sensor 20 owned by user U1 in an attempt to log in to terminal 301 where only user U1 has login authority, the recognition result will be NG, and an error message will be displayed on terminal 301, or a beep or message indicating an error will be notified via sensor 20. Of course, if user U1 attempts to log in to terminal 302 where only user U2 has login authority, authentication will be NG.

[0060] Here, the integrated feature obtained by the feature integration unit 107 is calculated as feature difference data between the first feature and the second feature. Alternatively, the integrated feature may be calculated by any one of addition, subtraction, multiplication, and division, or by a combination of these operations.

[0061] The shape of the inside of the ear canal differs for each user. Furthermore, even for the same user, the shape of the inside of the ear canal of the right ear and the shape of the inside of the ear canal of the left ear are different. Therefore, even if there are multiple users with extremely similar ear canal shapes in one ear, it is extremely unlikely that there are users with extremely similar ear canal shapes in both ears, so users can be authenticated (identified) with high accuracy.

[0062] Furthermore, the sensor 20 captures video of the inside of the ear canal, and the personal authentication device 10 authenticates the user based on multiple image data in the video (generating first features, second features, integrated features, and a registration model). The multiple image data included in the video each have different shooting locations, angles, magnifications, etc., and therefore authentication accuracy is higher than when authenticating a user based on a single still image.

[0063] In this description, the personal authentication system 1 performs authentication when user U1 logs in to terminal 301, but it can also perform authentication when a user accesses an application or database installed on the terminal. A personal authentication system employing an earphone-type sensor 20, as in the present invention, can perform authentication without the user being aware of it, reducing the authentication effort and enabling constant personal authentication. Therefore, login permission and access permission can be granted easily and immediately, and a significant improvement in security level can be achieved.

[0064] As a supplementary note, various methods are conceivable for the processing of the authentication unit 109, including, for example, a method using a statistical distance measure such as a discriminant function, Euclidean distance, or Mahalanobis distance, and a method using a machine learning model such as a decision tree, SVM, or neural network. Furthermore, when a discriminant function or statistical distance measure is used for the registration model, statistical quantities such as a discrimination threshold and mean / variance correspond to features that can identify a user. On the other hand, when a machine learning model such as a decision tree, SVM (Support Vector Machine), or neural network is used for the registration model, thresholds, SVM parameters, neural network model parameters, and the like correspond to features that can identify a user.

[0065] [3] Registration (quasi-normal) The above-mentioned registration and authentication (see Figures 5 and 7) are normal cases in which no irregularities occur. Below, we will explain registration and authentication in the case where an irregularity occurs (see Figures 6 and 8).

[0066] First, user U1 registers features (registration model) based on his / her left and right ears in the personal authentication device 10. As shown in FIG. 6, when user U1 wears his / her own sensor 20 in both ears, the sensor 20 captures a video of the inside of the ear canal using the camera 24 and transmits the captured video (registration video) to the personal authentication device 10 (step S130).

[0067] Meanwhile, the personal authentication device 10 receives the video via the receiving unit 101, and the image determination unit 103 performs various determinations on the image data selected from the video (image data used to extract the first feature and the second feature) as to whether it is unclear or not (step S201).

[0068] If the result of this determination is satisfactory (clear), registration is carried out according to the flow shown in FIG.

[0069] However, if the acquired image data is determined to be too dark / too bright, the personal authentication device 10 controls the irradiation direction of the light 25 of the sensor 20 (step S202). Specifically, the control unit 110 of the personal authentication device 10 transmits an instruction to the sensor 20 to specify (change) the irradiation direction of the light 25, and the main body 21 of the sensor 20 controls the irradiation direction of the light 25 in accordance with the instruction. In this case, the personal authentication device 10 can control the amount of light and color of light of the light 25 of the sensor 20, and can also precisely control which of the multiple lights 25 to change the irradiation direction, amount of light, or color of light.

[0070] This makes it possible to stably acquire image data that is neither too dark nor too bright and clearly shows the shape of the inside of the ear canal, and to generate integrated features and a registered model using the image data as training data. This ensures the authentication performance of authentication using the registered model.

[0071] Alternatively, if it is determined that the location where the acquired image data was taken is inappropriate, the personal authentication device 10 notifies the user U1 via the sensor 20 that the mounting position of the sensor 20 should be changed (step S203).

[0072] As described above, the sensor 20 has the functions of a typical earphone, such as a speaker and a microphone, and therefore the image determination unit 103 can send voice instruction data (e.g., "It is not worn correctly" or "It is tilted") previously stored in the storage unit 108 to the user U1 to urge the user U1 to wear the sensor 20 correctly. Alternatively, a third party can view and check the video (registration video) sent from the sensor 20 in real time on the personal authentication device 10 side, and convey such instructions to the user U1.

[0073] This allows stable acquisition of image data that clearly shows the shape inside the ear canal, and the image data can be used as training data to generate integrated features and a registered model, thereby ensuring the authentication performance of the authentication.

[0074] Alternatively, if it is determined that the acquired image data contains noise such as blown-out highlights or crushed shadows (shadows), the personal authentication device 10 removes the noise using the image processing unit 104 (step S204). A possible method for the image processing unit 104 to remove the noise is to generate noise-free image data from noisy image data using an AI model that has been trained on image data containing noise such as blown-out highlights or crushed shadows (shadows).

[0075] This allows image data that clearly shows the shape of the inside of the ear canal, without any noise such as blown-out highlights or crushed shadows (shadows), to be used as training data to generate integrated features and a registered model, thereby ensuring the authentication performance of the authentication.

[0076] Furthermore, if it is determined that the acquired image data contains earwax, ear hair, etc., the personal authentication device 10 obtains integrated features using the image data containing earwax, ear hair, etc. as training data using the first feature extraction unit 105, the second feature extraction unit 106, and the feature integration unit 107 (step S205).

[0077] The presence of earwax, ear hair, etc. in the ear canal is a possibility for all users. Therefore, when the image determination unit 103 determines that earwax, ear hair, etc. are present in the acquired image data, the personal authentication device 10 generates two registered models for user U1: one generated from an image in which earwax, ear hair, etc. are present in the ear canal, and the other generated from an image in which earwax, ear hair, etc. are not present in the ear canal. Each registered model is associated with user U1 and stored in the storage unit 108.

[0078] [4] Authentication (quasi-normal) Next, we will explain authentication when an irregularity occurs (see Figure 8). For authentication, user U1 wears his / her own sensor 20 in both ears, and the sensor 20 captures a video of the inside of the ear canal using the camera 24, and transmits the captured video (authentication video) to the personal authentication device 10 (step S330).

[0079] Meanwhile, the personal authentication device 10 receives the video via the receiving unit 101, and the image determination unit 103 performs various determinations on the image data selected from the video (image data used to extract the first feature and the second feature) as to whether it is unclear or not (step S401).

[0080] If the judgment result is acceptable (clear), authentication is performed according to the flow shown in Figure 6. However, if the acquired image data is judged to be too dark / too bright, the personal authentication device 10 controls the irradiation direction of the light 25 of the sensor 20, as in step S202 (step S402). If the acquired image data is judged to be captured at an inappropriate location, the personal authentication device 10 notifies the user U1 via the sensor 20 that the sensor 20 should be moved (step S403), as in step S203. If the acquired image data is judged to contain noise such as blown-out highlights or crushed shadows (shadows), the personal authentication device 10 removes the noise using the image processing unit 104, as in step S204 (step S404).

[0081] This allows authentication to be performed based on image data that clearly shows the shape of the inside of the ear canal, preventing misrecognition due to unclear image data and ensuring stable authentication performance.

[0082] Furthermore, if it is determined that the acquired image data contains earwax, ear hair, etc., the personal authentication device 10 uses the image data containing earwax, ear hair, etc. as training data to obtain integrated features using the first feature extraction unit 105, the second feature extraction unit 106, and the feature integration unit 107 (step S405), as in step S205. In this case, the authentication unit 109 performs authentication using the registered model generated in step S205 (registered model generated from an image in which earwax, ear hair, etc. are present in the ear canal) (step S406).

[0083] This ensures stable authentication performance regardless of whether earwax, ear hair, etc. are present in the ear canal of user U1. If the image determination unit 103 determines that the acquired image data contains earwax, ear hair, etc., the image processing unit 104 can remove the earwax, ear hair, etc., just as it does when it is determined that the acquired image data contains noise such as blown-out highlights or crushed shadows (shadows). This process is similar to the registration phase, and if the personal authentication device 10 determines that the acquired image data contains earwax, ear hair, etc., it can also generate a registration model from the image data from which the earwax, ear hair, etc. have been removed.

[0084] [Variant Example] The personal authentication system 1 according to the embodiment of the present invention described above is an example (representative example) of an embodiment of the present invention, and although in this description both the first sensor 20R and the second sensor 20L are used, it is also possible to use only one of them, and furthermore, as long as the gist of the present invention is not changed, the design can be changed as desired as shown below.

[0085] For example, in the present embodiment, the image determination unit 103, which makes various determinations as to whether image data of the inside of the ear canal acquired by the sensor 20 is unclear, is included in the personal authentication device 10, but the sensor 20 may also include the image determination unit 103. In this case, if the sensor 20 determines that the image data acquired by the image determination unit 103 is too dark / too bright, for example, the sensor 20 itself (by the main body unit 21) can control (adjust) the irradiation direction, amount of light, color of light, etc. of the light 25 before transmitting the video to the personal authentication device 10.

[0086] Similarly, the sensor 20 may have a first feature extraction unit 105, a second feature extraction unit 106, and a feature integration unit 107. In this case, the sensor 20 transmits features (integrated features) to the personal authentication device 10 instead of video (image data).

[0087] In this embodiment, the personal authentication device 10 has the authentication unit 109, which authenticates (specifies) a user based on image data acquired by the sensors 20 (20R, 20L) using a registered model, but the terminal 30 may have the authentication unit 109. In this case, the terminal 30 receives a video of the inside of the ear canal and an authentication request from the sensor 20, and can perform authentication by referring to the registered model stored in the memory unit 108 of the personal authentication device 10 or in an external storage.

[0088] Furthermore, in the present embodiment, an example has been described in which it is authenticated whether the person wearing the sensor 20 is user U1, but the personal authentication system etc. according to the present invention can also authenticate (identify) which user (e.g., whether it is user U1 or user U2 (see FIG. 1 )) has worn the sensor 20. In this case, the personal authentication device 10 obtains an integrated feature based on image data of the inside of the ear canal acquired by the sensor 20, and performs pattern matching between the integrated feature and a plurality of registered models stored in the storage unit 108 to identify the matching registered model (user).

[0089] In addition, after generating integrated features and a registration model from the received video, the personal authentication device 10 may delete the video from the storage unit 108 to improve security.

[0090] [Application Example] An example of an application of the present invention is the construction industry. In recent years, the number of workers in the Japanese construction industry has decreased, leading to a demand for improvements to the working environment at construction sites. For example, the use of professional headsets that enable clear voice calls (with noise-canceling functionality) even in noisy environments has become widespread. Therefore, the present invention can be applied to systems that employ such professional headsets as the sensor 20 and can improve the performance of user (speaker) identification. Furthermore, speaker identification can be used in systems that verify license possession at work sites or identify individuals to prevent information leaks.

[0091] Additionally, the present invention can be applied to medical information systems, health checkup systems, and entrance management systems. For example, by performing personal authentication based on the shape of the inside of the ear canal, which is assumed to be unique, it is possible to realize an entrance management system that manages entry to highly secure rooms and facilities that require high authentication accuracy. In other words, it is possible to realize an entrance management system that does not cause problems such as allowing unauthorized persons to enter due to authentication errors. Furthermore, since no new equipment for entry management is required, it is easy to introduce and does not cause problems associated with equipment installation.

[0092] Alternatively, in a medical information system where permissions for viewing documents and using equipment are determined, real-time authentication and authorization can be performed when granting access rights within the system to various personnel such as doctors, nurses, and pharmacists, thereby reducing the effort and time required to grant access rights.

[0093] Furthermore, the present invention can realize a health checkup system that performs personal authentication based on image data of the inside of the ear canal and can simultaneously determine abnormalities (injuries, growths) in the ear canal and changes in physical condition from the data. Furthermore, by using the sensor 20 that has a room temperature measurement function, a humidity measurement function, and a body temperature measurement function of the user U, the present invention can also be configured to recommend how to use the device in a usage environment, rest times, etc., based on environmental data such as the room temperature and humidity of the location where the user U is located, and vital data such as changes in the user U's body temperature and usage time (time the sensor 20 is worn).

[0094] In this embodiment, authentication of humans has been described, but the present invention can also be applied to living things that have ear canals, such as livestock and pets, and it is possible to determine whether or not a livestock or the like is owned by the user.

[0095] Furthermore, with regard to the sensor 20, a hole communicating with the outside may be provided in the insertion section 23 so that external sounds can be picked up even while the sensor 20 is being worn, or a microphone for picking up external sounds may be attached to the sensor 20 itself.

[0096] Since the sensor 20 is an important device for personal authentication, the insertion portion 23 is provided with a pressure sensor and sound-emitting member (not shown) that detects a sudden change in pressure (the insertion portion 23 is removed from the ear, causing a sudden change in the output of the pressure sensor), and when the sensor 20 is removed from the ear, the sound-emitting member makes a sound, thereby notifying the user that the sensor 20 has fallen off the ear. Although this may increase the size of the device, a GPS chip or the like may be incorporated into the sensor 20 so that if the sensor 20 is dropped, it can be immediately found based on location information from a terminal or the like owned by the user.

[0097] Furthermore, if the sensor 20 breaks down or is lost, it may become impossible to log in to a smartphone or PC based on authentication. In such a case, it is possible to log in by entering a password separately provided on the smartphone, or to install a contactless card reader component on the smartphone so that login can be performed using a card key, etc. Furthermore, a voice authentication device or fingerprint authentication device can be provided on the smartphone so that the smartphone can be activated by voice or fingerprint, etc. In this case, security can be further enhanced by combining voice authentication or fingerprint authentication with the ear canal authentication of the present invention.

[0098] Furthermore, since it is possible to measure the number of times the sensor 20 is worn, it is possible to prompt the user to clean the insertion part 23 according to the number of times it is worn, or to record the user's working hours (start time, end time), etc., thereby providing various conveniences. Furthermore, since the sensor 20 can acquire video of the inside of the ear canal, it is possible to monitor the progress of the user's otitis media and the amount of earwax accumulation, etc., thereby further improving convenience.

[0099] In addition, the sensor 20 is provided with charging contacts and a coil for contactless charging (not shown), a measuring means for measuring the degree of contamination on the surface of the camera 24 when the sensor 20 is set in a charger for charging, and the degree of contamination on the camera 24 can be measured by, for example, bringing the measuring means into contact with the surface of the camera and measuring the electrical capacitance or electrical resistance of the sensor 20. The degree of contamination according to the measurement status can then be displayed on a display or the like provided on the charger to inform the user.

[0100] The present invention provides a personal authentication system that can guarantee or improve authentication performance, or increase convenience, and is industrially useful because its biometric authentication technology can be utilized in a variety of fields, such as entrance management systems, health checkup systems, and medical information systems used in medical institutions.

[0101] REFERENCE SIGNS LIST 1 Personal authentication system 10 Personal authentication device 101 Receiving unit 102 Transmitting unit 103 Image determination unit 104 Image processing unit 105 First feature extraction unit 106 Second feature extraction unit 107 Feature integration unit 108 Storage unit 109 Authentication unit 110 Control unit 20 Sensor 20R First sensor 20L Second sensor 21 Main unit 22 Communication unit 23 Insertion unit 24 Camera 25 Light 26 Base 30, 301, 302 Terminal U, U1, U2 User Y User's ear N Network communication network

Claims

1. A personal authentication system that performs personal authentication based on captured images of the inside of the ear canal, comprising: a sensor that is inserted into the external auditory canal of a user's ear and acquires multiple pieces of image data from inside the ear canal; a personal authentication device that can communicate with the sensor and has: a feature extraction unit that extracts features from the multiple pieces of image data acquired by the sensor as learning data; a feature integration unit that generates a registered model that can identify the user from the features obtained by the feature extraction unit; and an authentication unit that uses the registered model to authenticate the user based on the multiple pieces of image data acquired by the sensor.

2. The personal authentication system according to claim 1, wherein a plurality of image data are acquired by varying the position of the sensor inserted into the external ear canal of the user's ear.

3. The personal authentication system according to claim 1, wherein the sensor acquires a plurality of image data by varying at least one of the photographing location, photographing angle, and photographing magnification within the ear canal.

4. A personal authentication system as described in claim 1, wherein the sensor has a video recording function, and the authentication unit authenticates the user based on multiple image data in the video captured by the sensor from the entrance of the ear canal to near the eardrum.

5. A personal authentication system as described in claim 1, wherein the sensor has a light that illuminates the inside of the ear canal, the personal authentication device has an image determination unit that determines whether the image data acquired by the sensor is unclear, and the image determination unit controls the direction of irradiation of the light of the sensor when it determines that the image data is unclear.

6. A personal authentication system as described in claim 1, wherein the sensor has a communication function, the personal authentication device has an image determination unit that determines whether image data acquired by the sensor is unclear, and if the image determination unit determines that the image data is unclear, it notifies the user via the sensor that they should move the position of the sensor.

7. The personal authentication system according to claim 1, wherein the personal authentication device comprises an image determination unit that determines whether the image data acquired by the sensor is unclear, and an image processing unit that removes noise contained in the image data when the image determination unit determines that the image data is unclear.

8. The personal authentication system of claim 1, wherein the sensor acquires image data from inside the ear canal at a predetermined timing and / or at a timing instructed by an external device, and the personal authentication device updates the registered model based on the integrated features obtained based on the image data.

9. The personal authentication system of claim 1, wherein the personal authentication device comprises an image determination unit that determines whether or not earwax and / or ear hair is contained in the image data acquired by the sensor, and an image processing unit that removes the earwax and / or ear hair contained in the image data if the image determination unit determines that the image data contains earwax and / or ear hair.

10. The personal authentication system of claim 1, wherein the personal authentication device has an image determination unit that determines whether or not earwax and / or ear hair is contained in the image data acquired by the sensor, and the feature extraction unit and the feature integration unit obtain the integrated feature based on the image data when the image determination unit determines that earwax and / or ear hair is contained in the image data, and the authentication unit uses a registration model generated from the integrated feature to authenticate the user based on the image data acquired by the sensor that contains earwax and / or ear hair.

11. The personal authentication device comprises an image determination unit that determines whether or not earwax and / or ear hair is contained in the image data acquired by the sensor, and an image processing unit that processes the image data so that earwax and / or ear hair is contained in the image data if the image determination unit determines that the image data does not contain earwax and / or ear hair, and the feature extraction unit and the feature integration unit obtain the integrated features based on the image data processed by the image processing unit and generate the registration model. The personal authentication system described in claim 8.

12. The personal authentication system described in claim 1, wherein the sensor has a main body, a camera, and a protruding base attached to the main body and connecting the main body and the camera, and the attachment position of the base can be changed depending on the shape of the user's ear canal and / or the position of the eardrum.

13. A personal authentication system according to any one of claims 1 to 12, comprising: the sensor comprising: a first sensor inserted into the external auditory canal of the user's right ear and acquiring multiple image data from within the external auditory canal; and a second sensor inserted into the external auditory canal of the user's left ear and acquiring multiple image data from within the external auditory canal; the feature extraction unit comprising: a first feature extraction unit that extracts first features using the multiple image data acquired by the first sensor as training data; and a second feature extraction unit that extracts second features using the multiple image data acquired by the second sensor as training data; a feature integration unit that generates a registered model capable of identifying the user from integrated features obtained by integrating the first features and the second features; and an authentication unit that uses the registered model to authenticate the user based on the multiple image data acquired by the first sensor and the second sensor.

14. Any one of a medical information system, a health checkup system, an admission management system, an automatic withdrawal system, and a delivery management system, which uses the personal authentication system according to any one of claims 1 to 12.

15. A personal authentication method for authenticating a person based on an image captured inside the ear canal, comprising: a step of acquiring multiple image data from inside the ear canal using a sensor inserted into the ear canal of the user's ear; a step of extracting features from the multiple image data acquired by the sensor as learning data using a personal authentication device; a step of generating a registration model capable of identifying the user from the features acquired by the personal authentication device; and a step of authenticating the user using the registration model based on the multiple image data acquired by the sensor using the personal authentication device.

16. A program for operating a computer as a personal authentication device that is capable of communicating with a sensor that is inserted into the external auditory canal of a user's ear and acquires multiple image data from within the ear canal, and that has: a feature extraction unit that extracts features from the multiple image data acquired by the sensor as learning data; a feature integration unit that generates a registered model that can identify the user from the features obtained by the feature extraction unit; and an authentication unit that uses the registered model to authenticate the user based on the multiple image data acquired by the sensor.

Images