Personal authentication device, personal authentication method, and personal authentication program, and system having a personal authentication device

Abstract

To provide a personal authentication device, etc. capable of achieving improvement of authentication performance without being much influenced by an ambient environment.SOLUTION: A personal authentication device 10 (10b) for executing personal authentication by acquiring a plurality of different pieces of biological information in the external auditory meatus, and extracting a feature amount of the plurality of different pieces of biological information using machine learning includes: a feature amount extraction part 11 for extracting a feature amount on the basis of data acquired from two or more sensors of a first sensor 21 for acquiring acoustic data on the external auditory meatus of a user, a second sensor 22 for acquiring image data on the external auditory meatus of the user, and a third sensor 23 for acquiring biochemical data on the external auditory meatus of the user; and a determination part 13 for constructing an individuality integrated neural network on the basis of two or more feature amounts extracted by the feature amount extraction part 11, and determining the user on the basis of the integrated data.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a personal authentication device, a personal authentication method, and a personal authentication program that acquire a plurality of different biometric information in the external auditory canal, extract feature amounts of the plurality of different biometric information using a neural network, construct a personalized integrated neural network, and perform personal authentication based on integrated data created using the personalized integrated neural network, and a system having the personal authentication device.

Background Art

[0002] In recent years, biometric authentication technology has been used in various fields. Biometric authentication is a technology for authenticating an individual based on biometric features such as a person's physical features and behavioral features. Main physical features used in biometric authentication include fingerprints, retinas, irises, palm prints, faces, etc. Also, main behavioral features used in biometric authentication include handwriting, voice, walking style, etc.

[0003] For example, fingerprint authentication is used to unlock smartphones. Also, face authentication is used when entering an airport or a theme park, and retina authentication is used when entering a highly secure room. There is also a safe that has both face authentication and iris authentication for sale.

[0004] In this way, in situations where biometric authentication is used, users do not need to carry around keys to open safes or ID cards to present when entering or leaving. Therefore, there is no risk of losing, stealing, or forging keys or ID cards. Also, there is no need to remember IDs (accounts) or passwords. Thus, biometric authentication is known as a very useful technology with high convenience and security.

[0005] On the other hand, as prior art related to biometric authentication, there are those described in Patent Documents 1 to 3. Patent Document 1 describes a biometric authentication system, and in relation to biometric authentication technology, it describes a technology for registering templates in a system that performs biometric authentication using multiple types of templates. Patent Document 1 also lists fingerprints, veins, irises, and faces as examples of multiple types of templates.

[0006] Furthermore, Patent Document 2 describes a personal identification device and a personal identification method. The personal identification device in question is designed to be difficult to forge, to avoid psychological resistance from the user, to have minimal physical constraints during use, and to be less affected by the user's psychological or health state. It utilizes biometric information such as the shape and anatomical features of the ear organs (ear), including information on the shape of the eardrum, the vascular pattern of the eardrum, and the vascular pattern of the external auditory canal.

[0007] Furthermore, Patent Document 3 describes a personal authentication device, a personal authentication method, and a personal authentication program aimed at reducing the psychological and / or physical burden on the user being authenticated. The personal authentication device comprises: an acoustic signal transmission means for transmitting a first acoustic signal to a part of the user's head; an acoustic signal observation means for observing a second acoustic signal, which is the acoustic signal after the first acoustic signal has propagated through a part of the head; an acoustic characteristic calculation means for calculating acoustic characteristics from the first and second acoustic signals; and a user identification means for identifying the user based on the acoustic characteristics or user-related features extracted from the acoustic characteristics. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Patent No. 5218991 [Patent Document 2] Patent No. 3775063 [Patent Document 3] Patent No. 6855381 [Overview of the project] [Problems that the invention aims to solve]

[0009] However, as mentioned above, biometric authentication can be used in various fields, so there is a need for biometric authentication technology that is less susceptible to the influence of the surrounding environment and can improve authentication performance.

[0010] Therefore, the technology described in Patent Document 2 is based solely on image data, such as the shape of the eardrum, the blood vessel pattern of the eardrum, or the blood vessel pattern of the external auditory canal, and thus has limitations in authentication performance. Similarly, the technology described in Patent Document 3 is based solely on acoustic signals, and therefore also has limitations in authentication performance.

[0011] On the other hand, the technology described in Patent Document 1 uses multiple types of biometric information (for example, the first biometric information is a fingerprint, and the second biometric information is the vein pattern of the fingerprint), but since multiple types of biometric information can be broadly classified as the same concept of information, namely image data, it can be said that there are limitations to its authentication performance.

[0012] Therefore, the present invention provides a personal authentication device, a personal authentication method, and a personal authentication program, as well as a system having a personal authentication device, which are less susceptible to the influence of the surrounding environment and can improve authentication performance. [Means for solving the problem]

[0013] The present invention provides a personal authentication device that acquires multiple different biometric information from the external auditory canal, extracts feature quantities of the multiple different biometric information using machine learning, and performs personal authentication. The device comprises a feature extraction unit that extracts feature quantities based on data acquired from two or more sensors, including a first sensor that acquires acoustic data from the user's external auditory canal, a second sensor that acquires image data from the user's external auditory canal, and a third sensor that acquires biochemical data from the user's external auditory canal; and a determination unit that constructs an integrated neural network for personalization based on the two or more feature quantities extracted by the feature extraction unit and determines the user based on the integrated data.

[0014] Furthermore, the first sensor is preferably a sensor selected from the group consisting of a bone conduction microphone, a vibration sensor, and an acceleration sensor, and acquires one or more of the following as acoustic data: vibration sound in the external auditory canal associated with the user's voice, vibration sound associated with blood flow, and sound due to otoacoustic radiation, and the feature extraction unit extracts time-frequency features based on the vibration sound in the external auditory canal associated with the user's voice and / or vibration sound associated with blood flow and / or sound due to otoacoustic radiation.

[0015] Furthermore, the second sensor is preferably an infrared sensor and / or an RGB camera, which acquires images showing the shape and temperature distribution within the user's external auditory canal, as well as an image of the vascular network within the external auditory canal, as image data. The feature extraction unit estimates the heart rate using the image showing the temperature distribution within the external auditory canal and the image of the vascular network within the external auditory canal, and extracts features based on the heart rate, the image showing the shape and temperature distribution within the external auditory canal, and the image of the vascular network within the external auditory canal.

[0016] Furthermore, the second sensor preferably incorporates a pinhole lens and / or a fisheye lens to enable focusing over a wide range.

[0017] Furthermore, the second sensor is preferably a sensor equipped with lenses having multiple focal lengths, and the feature extraction unit extracts features based on data obtained from one or more focal lengths of the lenses.

[0018] Furthermore, the third sensor is preferably an evanescent wave analysis sensor, a gas chromatography analysis sensor, or a semiconductor odor sensor, and it is preferable that the biochemical data obtains one or more of the following: secretions in the user's external auditory canal, odors in the external auditory canal, and volatile organic compounds emitted from the human body, and that the feature extraction unit extracts features based on the secretions and / or odors.

[0019] Further, the feature quantity extraction unit generates two or more learned models by performing machine learning using data acquired from two or more of the first sensor, the second sensor, and the third sensor as learning data, and uses the learned models to extract feature quantities based on the data acquired from two or more sensors. It is preferable that this is the case.

[0020] Further, it is preferable that the determination unit determines the user by comparing the feature quantity for each registered user with the feature quantity extracted by the feature quantity extraction unit.

[0021] Further, the feature quantity extraction unit extracts two or more of acoustic feature quantities, image feature quantities, and biochemical feature quantities based on the data acquired from the first sensor, the second sensor, and the third sensor, and the determination unit constructs a personal integrated neural network using these feature quantities, and uses the personal integrated neural network to create integrated data in which two or more of the determination accuracies based on the acoustic feature quantity, the determination accuracy based on the image feature quantity, and the determination accuracy based on the biochemical feature quantity are integrated, and determines the user based on the integrated data. It is preferable that this is the case.

[0022] Further, the present invention is a personal authentication system having a microphone and / or earphone including these first sensor, second sensor, and third sensor, and these personal authentication devices. Note that this personal authentication system is preferably used for an entrance management system or a health examination system.

[0023] In addition, the personal authentication method of the present invention is a method of acquiring a plurality of different biometric information in the external auditory canal, extracting feature amounts of the plurality of different biometric information using machine learning, and performing personal authentication. Based on data obtained from two or more of the following sensors: a first sensor that acquires acoustic data in the user's external auditory canal, a second sensor that acquires image data in the user's external auditory canal, and a third sensor that acquires biochemical data in the user's external auditory canal, a step of extracting feature amounts by a feature amount extraction unit; and a step of constructing an individual integrated neural network by a determination unit based on two or more extracted feature amounts and determining the user based on integrated data.

[0024] In addition, the personal authentication program of the present invention causes a computer to function as an apparatus that acquires a plurality of different biometric information in the external auditory canal, extracts feature amounts of the plurality of different biometric information using machine learning, and performs personal authentication. The apparatus has a feature amount extraction unit that extracts feature amounts based on data obtained from two or more of the following sensors: a first sensor that acquires acoustic data in the user's external auditory canal, a second sensor that acquires image data in the user's external auditory canal, and a third sensor that acquires biochemical data in the user's external auditory canal; and a determination unit that constructs an individual integrated neural network based on two or more feature amounts extracted by the feature amount extraction unit and determines the user based on integrated data.

Advantages of the Invention

[0025] The present invention provides a personal authentication device that acquires multiple different biometric information from the external auditory canal, extracts feature quantities from these multiple different biometric information using machine learning, and performs personal authentication. The device comprises a feature extraction unit that extracts feature quantities based on data acquired from two or more sensors, including a first sensor that acquires acoustic data from the user's external auditory canal, a second sensor that acquires image data from the user's external auditory canal, and a third sensor that acquires biochemical data from the user's external auditory canal; and a determination unit that constructs an integrated neural network based on the two or more feature quantities extracted by the feature extraction unit and determines the user based on the integrated data. Compared to conventional biometric authentication methods, this configuration makes it easy to combine multiple adjacent authentication methods, thereby significantly improving authentication performance. In addition, even if the usage environment or the individual's physical condition changes, different authentication methods can complement each other, so improved robustness can be expected.

[0026] Furthermore, the ear canal is less susceptible to the influence of the surrounding environment, making it a location where stable authentication performance can be expected. This allows for the construction of a highly robust authentication principle that cannot be achieved with conventional biometric authentication methods. The ear canal is a closed space isolated from the outside world, and the fact that it has little noise that would interfere with authentication, such as unwanted sounds, light, and smells, as well as a stable environment with appropriate humidity, is a major advantage.

[0027] Furthermore, the inside of the external auditory canal is less susceptible to the influence of the surrounding environment, making it a location where stable authentication performance can be expected. This allows for the construction of a highly robust authentication principle that cannot be achieved with conventional biometric authentication methods.

[0028] Furthermore, the personal authentication method of the present invention can achieve the same effects and advantages as the personal authentication device of the present invention.

[0029] Furthermore, the personal authentication program of the present invention allows a computer to operate as a device equivalent to the personal authentication device of the present invention. [Brief explanation of the drawing]

[0030] [Figure 1]This is a schematic functional block diagram of a personal authentication device according to Embodiment 1 of the present invention. [Figure 2] This is a flowchart of the personal authentication method according to Embodiment 1 of the present invention. [Figure 3] This is a schematic functional block diagram of a personal authentication device according to Embodiment 2 of the present invention. [Figure 4] This is a flowchart illustrating the learning process for the personal authentication method according to Embodiment 2 of the present invention. [Figure 5] This is a flowchart of the authentication process for the personal authentication method according to Embodiment 2 of the present invention. [Figure 6] This figure shows an example of earphones equipped with a sensor. [Figure 7] This is a diagram to explain the mechanism of the rPPG method. [Figure 8] This is a diagram illustrating the different types of odor sensors. [Modes for carrying out the invention]

[0031] The embodiments of the present invention will be described in detail below, but the description of the constituent elements described below is just one example (representative example) of the embodiments of the present invention, and the present invention is not limited to the following unless its gist is changed.

[0032] [Personal Authentication System] Figures 1 and 3 are schematic functional block diagrams of a personal authentication device according to an embodiment of the present invention. As shown in Figures 1 and 3, the personal authentication system 1 includes a sensor device 20 and a personal authentication device 10.

[0033] [Sensor device] The sensor device 20 is capable of acquiring multiple different biological information from the external auditory canal. As shown in Figure 3, the sensor device 20 includes a first sensor 21, a second sensor 22, and a third sensor 23.

[0034] The first sensor 21 is a sensor that acquires acoustic data from within the ear canal of the user (the person using the personal authentication system 1). The first sensor is selected from a group consisting of, for example, a bone conduction microphone, a vibration sensor, and an acceleration sensor, and acquires vibrations within the ear canal associated with the user's speech as acoustic data.

[0035] In particular, the way sound is transmitted (propagated) within the external auditory canal differs from person to person. Therefore, in order to investigate how sound is propagated within the external auditory canal, it is desirable to acquire acoustic data using multiple methods, such as bone conduction microphones and vibration sensors, in addition to a regular microphone. For example, as shown in Figure 1, it is desirable to acquire multiple acoustic data of the sound and vibrations emitted by the user using two sensors (first sensor A211, first sensor B212).

[0036] The second sensor 22 is a sensor that acquires image data from inside the user's external auditory canal. The second sensor 22 is, for example, an infrared sensor. The second sensor 22 acquires image data from inside the external auditory canal, and this image data includes images of the shape inside the external auditory canal (images that show the shape inside the external auditory canal), images of the vascular network inside the external auditory canal, and images showing the temperature distribution inside the external auditory canal. In addition, individual pulse estimation data acquired using analysis techniques such as RGB cameras and thermal cameras can also be used.

[0037] The third sensor 23 is a sensor that acquires biochemical data from within the user's external auditory canal. The third sensor 23 also acquires biochemical data of secretions and / or odors within the user's external auditory canal using evanescent wave analysis and / or gas chromatography analysis or a semiconductor odor sensor. Furthermore, data obtained by analyzing odor components uniquely emitted by individuals, using odor sensors that measure the quality and intensity of odors, can also be used.

[0038] The data (information) acquired by the first sensor 21, the second sensor 22, and the third sensor 23 is transmitted to the personal authentication device 10.

[0039] On the other hand, Figure 6 shows an example of an earphone equipped with sensors. The sensor device 20 can be an earphone-type device worn on the user's ear. The earphone is equipped with each sensor (first sensor 21, second sensor 22, third sensor 23), but one sensor device may be equipped with all sensors. As shown in Figure 6, one sensor device (sensor device 20) may be equipped with the first sensor 21, and the other sensors (second sensor 22, third sensor 23) may be equipped with other sensor devices. This configuration has the advantage that a single device such as an earphone can be equipped with sensors capable of acquiring multiple data so that multiple authentication methods can be performed.

[0040] Furthermore, compared to conventional biometric authentication methods, it is easy to combine multiple adjacent authentication methods (multimodal authentication), which significantly improves authentication performance. In addition, even if the usage environment or the individual's physical condition changes, different authentication methods can complement each other, so improved robustness can be expected.

[0041] Furthermore, the inside of the external auditory canal is less susceptible to the influence of the surrounding environment, making it a location where stable authentication performance can be expected. This allows for the construction of a highly robust authentication principle that cannot be achieved with conventional biometric authentication methods.

[0042] [Second sensor] Below, we will explain the technology using the RGB camera and thermal camera mentioned as examples of the second type of sensor.

[0043] [RGB Camera] The pulse rate estimation method using ambient light, specifically the rPPG (Remote Photoplethysmography) method, which utilizes an RGB camera, was proposed in 2008. Figure 7 illustrates the mechanism of the rPPG method. This method uses an RGB camera to estimate the pulse signal from a distance from the subject. Specifically, as shown in Figure 7, a portion of the forehead or cheek is manually selected as the Region of Interest (ROI), and the spatial average value of pixels within the ROI is extracted for each frame of the video for the red, green, and blue channels of the RGB camera to be used as the pulse signal.

[0044] The human face is a suitable region for setting an ROI because it has a high concentration of capillaries. Furthermore, it has been shown that the signal in the green channel contains more pulse components than the signals in the red and blue channels. This is based on the fact that the light absorption spectrum of hemoglobin, the main constituent pigment in blood, has a peak around 520-580 nm, which is the passband of the green filter in an RGB camera.

[0045] In rPPG, BSS-based methods using principal component analysis and independent component analysis are effective techniques. BSS is a technique for separating individual signals from an observed signal that is a mixture of multiple unknown signals, and in the field of biosignal analysis, it is used to remove noise from electrocardiograms and electroencephalograms.

[0046] In the example shown in Figure 7, the entire face was used as the ROI, and the spatial average of the red, green, and blue channels was taken for each frame to extract three signals. Subsequently, the signals were separated into three independent signals using independent component analysis, and the second component of the separated signals was used as the pulse wave signal. Using all three channels of signals and independent component analysis yields a value closer to the reference pulse rate measured by rPPG than using only the green channel, which is considered to have the strongest pulse wave signal intensity.

[0047] Furthermore, one challenge when extracting pulse wave signals using rPPG is the influence of lighting. The intensity and color of the lighting itself, as well as their changes over time, and specular reflection caused by lighting hitting the skin, are known to lower the SNR (Single-to-Noise Ratio) of the pulse wave signal.

[0048] One method for removing the color of the light from the skin is the separation of skin pigment components. It is known that, assuming that the spatial structures of melanin and hemoglobin, the main skin pigments, are independent of each other in the structure of human skin, independent component analysis can yield a color plane composed of melanin and hemoglobin pigments from the RGB color space that is independent of the light.

[0049] Furthermore, it has been demonstrated that hemoglobin pigments obtained by pigment component separation methods can be used for pulse rate estimation. Reflected light from the skin consists of a diffuse reflection component that changes with pulse rate and a specular reflection component that indicates the color of the illumination and does not indicate the pulse rate. Therefore, by estimating the ratio of these two orthogonal components, the specular reflection component can be removed. It has also been proven that optical skin reflection models are useful for improving the signal-to-noise ratio (SNR) of pulse wave signals.

[0050] Furthermore, methods have been employed to select one ROI, such as the cheek, forehead, or the entire face, and extract the signal. It has been shown that by selecting multiple ROIs and utilizing multiple signals, a pulse wave signal with a high SNR can be obtained. This is more versatile than methods that select ROIs to a specific part, provided that ROIs are weighted using an SNR index and the assumption is made that ROIs with high SNRs are distributed as clusters rather than sparsely.

[0051] [Thermal camera] The following describes a method for estimating pulse rate using a thermal camera. Pulse rate estimation methods using thermal cameras are less common than methods that measure respiratory rate using RGB cameras. A distinctive feature of pulse rate estimation using thermal cameras is a technique that identifies the location of blood vessels based on their brightness and utilizes the temperature changes of those vessels. The blood vessels identified are often those in the neck or face.

[0052] In addition to detecting blood vessels, a method for estimating pulse rate using a thermal camera also involves setting a region of interest (ROI) on the face, similar to the method using an RGB camera. This method reveals the temporal changes in facial brightness values ​​in thermal images. This method is called EVM (Eulerian Video Magnitification), a technique that uses spatiotemporal processing to amplify subtle color changes and imperceptible movements within the image. A wide-bandpass filter and low amplification are applied in the first pass, and a narrow-bandpass filter and high amplification are applied in the second pass, focusing on the size of the ROI. It has been shown that the signal extracted from the ROI using this method has a periodicity similar to that of a pulse wave obtained from an electrocardiogram.

[0053] [Third sensor] Next, I will explain the technology using the odor sensor mentioned as an example of a third type of sensor.

[0054] [Odor Sensor] Various odor sensors have been developed using different principles. Below, we will describe the method using quartz crystal oscillators, which has been researched and developed for some time, and sensors that combine recent MEMS technology and machine learning. Furthermore, odor sensors (odor measuring sensors) come in various types, as shown in Figure 8. The present invention can utilize a variety of odor sensors, including the semiconductor odor sensor shown in Figure 8.

[0055] Quartz crystal odor sensors are devices that integrate a nozzle or pump for drawing in odors, a sensor that measures the odor and converts it into an electrical signal, and a SIM card for sending and receiving data. The sensor is made of a quartz crystal, which is covered with a sensitive membrane. When an odor-causing substance is adsorbed onto the sensitive membrane, the vibration frequency of the quartz crystal changes according to its weight. Since different types of sensitive membranes are used for each quartz crystal, the pattern of change in vibration frequency for a given odor will also differ. The sum of these change patterns in the sensor is used to identify the characteristics of that odor.

[0056] Another example is the Membrane-type Surface Stress Sensor (MSS). The operating principle of an MSS is as follows: first, strain is generated by the force produced when gas molecules adsorb onto a sensitive film coated in the center of the MSS. Then, a detection unit embedded in the MSS electrically detects that strain has occurred.

[0057] The odor sensor works as follows: First, it draws in outside air. The device is equipped with an activated carbon filter, and by drawing in outside air through this filter, the sensor is cleaned. After that, it draws in the target odor and allows it to circulate within the sensor for a certain period of time. Once the sensor has finished reacting, the internal air is exhausted, and outside air is drawn in again for cleaning.

[0058] The odor data obtained in this way is stored in a memory area such as a database. Then, a trained model is created by using this data as training data. Finally, using this trained model, the analysis result of which odor the currently measured odor is similar to is analyzed and displayed as an "80% agreement rate".

[0059] On the other hand, there is a method that uses a small CMOS odor sensor (module sensor) to measure how odor molecules attach and detach, and recognize all odors as patterns. This sensor is composed of an array of sensor elements with different adsorption characteristics and is capable of acquiring information about "odors" that only humans could perceive. Unlike conventional gas sensors, it is capable of recognizing any odor as a pattern, much like the nose of a living organism. Furthermore, because it can, in principle, display any odor as a pattern, it can be widely applied, from distinguishing between different brands of alcohol to detecting diseases by measuring bad breath and body odor.

[0060] Thus, a crucial challenge is how to integrate features that have time-series information, such as audio, with features that do not have dynamic information (i.e., features that do not have time-series information), such as still images. Therefore, the personal authentication device of the present invention constructs a personalization integrated neural network (described later) by integrating periodic data similar to pulse waves obtained from image data in the external auditory canal with time-series data such as audio and odor.

[0061] [certification] In the authentication process of this invention, user authentication is performed using feature integration data and a registration model that includes the feature integration data stored in memory. Information about the user to be authenticated is pre-stored in the personal authentication device as a registration model. Then, the feature integration data based on data acquired by sensors is compared with the registration model, and personal authentication is performed by determining the similarity and identifying those that exceed a predetermined threshold. Furthermore, the authentication results can be notified or displayed. The authentication results can also be used as signals or messages to initiate other operations.

[0062] [Embodiment 1] Figure 1 is a functional block diagram of Embodiment 1 of the present invention. The personal authentication device 10 (10a) is a device that authenticates a user by sensing the voice and vibrations emitted by the user with the first sensor A211 and the first sensor B212.

[0063] The personal authentication device 10a has multiple sensors. In the example shown in Figure 1, the personal authentication device 10a has two sensors: a first sensor A211 and a first sensor B212.

[0064] The first sensor A211 and the first sensor B212 can use various microphones to capture human speech, such as condenser microphones, bone conduction microphones, and throat microphones. It is also possible to use sensors that capture vibrations and movements generated by a person's body, such as vibration sensors and acceleration sensors.

[0065] Although this embodiment 1 shows an example using two sensors, the first sensor A211 and the first sensor B212, the number of sensors may be even greater. For example, the number of sensors may be three, four, or more. Furthermore, by using different types of sensors or placing them in different locations, it is possible to acquire audio with different time and frequency features even for the same utterance from the same user (speaker).

[0066] Furthermore, by acquiring multiple bone conduction sounds that are less prone to noise, and calculating inter-feature difference data using the acquired multiple bone conduction sounds, it is possible to reduce the error in inter-feature difference data that occurs for each acquired data due to noise, thereby achieving stable authentication performance. For this reason, it is preferable that the first sensor A211 and the first sensor B212 are capable of acquiring bone conduction sounds.

[0067] The first feature extraction unit A1111 extracts time-frequency features from sound and vibration data acquired by the first sensor A211. Meanwhile, the first feature extraction unit B1112 extracts time-frequency features from sound and vibration data acquired by the first sensor B212.

[0068] Here, the time-frequency features can be various features that can represent the time-frequency characteristics of the signal acquired from the sensor, such as power spectra, logarithmic power spectra, or Mel-log power spectra calculated by frequency analysis of sound or vibration, prediction coefficients obtained from parametric methods such as linear predictive analysis, or cepstrum coefficients obtained from homomorphic analysis.

[0069] The feature difference data calculation unit 131 is an example of a feature integration data creation unit (integrated data creation unit). In other words, feature integration data can be created by calculating the difference data between features. The feature difference data calculation unit 131 calculates the difference between the time-frequency features extracted by the first feature extraction unit A1111 and the time-frequency features extracted by the first feature extraction unit B1112 as feature difference data. Furthermore, this section functions as an integrated data creation unit 131, which can create integrated data by combining the features extracted by the first feature extraction unit A1111 and the features extracted by the second feature extraction unit B1112.

[0070] Methods for calculating the difference between time and frequency features include, for example, using power spectra as features, calculating by division, or using logarithmic power spectra as features, calculating by subtraction. In other words, the feature difference data calculation unit 131 can be configured to calculate the difference or ratio between time and frequency features extracted from data acquired by multiple sensors as feature difference data.

[0071] As feature integration data, other configurations may be used, such as those calculated using addition, subtraction, multiplication, or division, or those that combine these operations. In particular, by using the difference or ratio of features based on data acquired by multiple sensors as inter-feature difference data, it becomes easier to extract the biological transmission characteristics included in the time and frequency features, since the sound transmitted through the body by multiple sensors is acquired.

[0072] Furthermore, the feature difference data calculation unit 131 may include spectral analysis means that perform spectral analysis on data acquired according to the mounting positions of multiple sensors at predetermined unit time intervals. The difference or ratio of spectral analysis results for each mounting position of the multiple sensors obtained by performing spectral analysis by the spectral analysis means can then be used. This unit time can be, for example, 5 to 50 msec or 10 msec to 25 msec.

[0073] The authentication difference data storage unit 121 is part of the storage unit 12 (see Figure 3). In other words, the authentication difference data storage unit 121 is part of the storage area and stores the feature difference data created by the feature difference data calculation unit 131. At this time, it is also possible to calculate the average of the feature difference data over multiple time frames and store it, or to calculate and store statistical data such as mean and variance. Furthermore, this part can also be an integrated data storage unit 121 that stores the integrated data created by the integrated data creation unit 131.

[0074] The authentication unit 132 performs authentication to identify which user (individual) the feature difference data stored in the authentication difference data storage unit 121 belongs to. Various methods can be considered for the processing of the authentication unit 132, such as using a discriminant function or statistical distance measures such as Euclidean distance or Mahalanobis distance, or using machine learning models such as decision trees, SVMs, or neural networks.

[0075] The registration model unit 122 is part of the memory unit 12 (see Figure 3). In other words, the registration model unit 122 is part of the memory area and stores the registration model for each user (individual) used when authenticating with the authentication unit 132. The registration models stored in the registration model unit 122 correspond to the authentication method performed by the authentication unit 132 and are created in advance through learning using feature difference data of the user (individual) to be authenticated.

[0076] This registration model can utilize models that have been pre-registered on the same device or on devices with a common configuration. Furthermore, the registration model is created by acquiring data by having the user to be authenticated speak typical speech samples, and then training based on this data.

[0077] When using discriminant functions or statistical distance measures in the registration model, statistical quantities such as the discrimination threshold and mean / variance correspond to features that can identify a user. On the other hand, when using machine learning models such as decision trees, SVMs (Support Vector Machines), or neural networks in the registration model, thresholds, SVM parameters, and neural network model parameters correspond to features that can identify a user.

[0078] The authentication results can be notified to or displayed externally from the authentication unit 132. Furthermore, the authentication results can be used or recorded by any means necessary. For example, authentication results that identify a user can be notified externally via sound, images, or other means. Furthermore, the notified authentication results can also be used as signals or messages that trigger the activation or deactivation of the target device or system.

[0079] Figure 2 is a flowchart of the personal authentication method according to this embodiment 1. The flow shown in Figure 2 will be explained below with reference to Figure 1.

[0080] First, data is acquired using multiple sensors (first sensor A211, first sensor B212) (step S11). In this embodiment 1, the multiple sensors acquire data such as the sound a person makes and vibrations and movements generated from a person's body.

[0081] Next, features are extracted based on the acquired data (step S21). In this embodiment 1, for example, time-frequency features are extracted from the acquired sound and vibration data. At this time, the first feature extraction unit A1111 extracts time and frequency features based on the data acquired by the first sensor A211, and the first feature extraction unit B1112 extracts time and frequency features based on the data acquired by the first sensor B212.

[0082] Then, the feature difference data calculation unit 131 creates integrated feature data (calculates feature difference data) (step S31). The feature difference data created by the feature difference data calculation unit 131 is stored in the authentication difference data storage unit 121.

[0083] Finally, the authentication unit 132 authenticates the user (individual) using this feature difference data and the registration model pre-stored in the registration model unit 122 (step S41). The authentication unit 132 also notifies an external party of the authentication result (step S51).

[0084] [Embodiment 2]

[0085] Figure 3 is a functional block diagram of Embodiment 2 of the present invention. The personal authentication device 10 (10b) is a device that authenticates a user based on data acquired by three sensors, namely a first sensor 21, a second sensor 22, and a third sensor 23.

[0086] The personal authentication device 10b includes a feature extraction unit 11 that extracts feature quantities based on data acquired from two or more sensors among the first sensor 21, the second sensor 22, and the third sensor 23; a storage unit 12; and a determination unit 13 that determines the user based on the feature quantities extracted by the feature extraction unit 11.

[0087] The data acquired by each sensor is transmitted to the personal authentication device 10b, and upon receiving the data, the personal authentication device 10b extracts features using the feature extraction unit 11. The feature extraction unit 11 can also be configured to include a first feature extraction unit 111 that extracts features based on data (acoustic data) acquired by the first sensor 21, a second feature extraction unit 112 that extracts features based on data (image data) acquired by the second sensor 22, and a third feature extraction unit 113 that extracts features based on data (biochemical data) acquired by the third sensor 23.

[0088] For example, the first feature extraction unit 111 extracts acoustic features based on acoustic data. Specifically, when the first sensor 21 acquires vibrations in the ear canal associated with the user's speech as acoustic data, the first feature extraction unit 111 extracts features (time-frequency features) based on those vibrations.

[0089] Furthermore, the second feature extraction unit 112 extracts image features based on the image data. Specifically, when the second sensor 22 acquires image data such as an image of the shape of the user's external auditory canal, an image of the vascular network within the external auditory canal, and an image showing the temperature distribution within the external auditory canal, the second feature extraction unit 112 estimates the heart rate using the image of the vascular network and the image showing the temperature distribution, and extracts features based on the estimated heart rate and the shape of the external auditory canal.

[0090] The third feature extraction unit 113 extracts biochemical features based on biochemical data. Specifically, if the third sensor 23 acquires biochemical data such as the user's secretions or odor, it extracts features based on those secretions or odors.

[0091] Here, the first feature extraction unit 111 performs machine learning on a neural network using acoustic data as training data, generates a trained model, and can use this trained model to extract acoustic features based on a user's acoustic data. Similarly, the second feature extraction unit 112 and the third feature extraction unit 113 can generate trained models and use these trained models to extract image features and biochemical features. Examples of neural networks include CNNs (Convolutional Neural Networks) and autoencoders. Furthermore, the first feature extraction unit 111 can extract features using methods other than neural networks, such as SVMs and decision trees.

[0092] The features extracted by the first feature extraction unit 111, the second feature extraction unit 112, the third feature extraction unit 113, etc., and the generated trained models are stored (registered) in the memory unit 12. The feature extraction unit 11 can communicate with the memory unit 12. The integrated data storage unit 12 can also store the individualized integrated neural network (described later) constructed by the feature extraction unit 11, or store integrated data created using the individualized integrated neural network. The storage unit 12 may be configured as part of the personal authentication device 10b, such as an HDD or SSD, or it may be external storage (for example, a cloud-based database).

[0093] The determination unit 13 determines the user based on the features extracted by the feature extraction unit 11. For example, it can determine the user by comparing the features for each user registered in the memory unit 12 with the features extracted by the feature extraction unit 11. In other words, since the feature pattern for each user (hereinafter referred to as "feature pattern") is registered in the memory unit 12 during training (described later), the determination unit 13 can compare the feature pattern with the feature pattern extracted by the feature extraction unit 11 during authentication (described later) and determine that the person with the closest feature pattern is the user (the person whose data was acquired by each sensor).

[0094] Since the determination unit 13 can communicate with the storage unit 12, it can refer to the feature quantities (feature patterns) registered in the storage unit 12 during authentication.

[0095] Furthermore, the determination unit 13 can determine the user by integrating multiple features (acoustic features, image features, and biochemical features). For example, the determination unit 13 can determine the user by integrating two or more determination probabilities (likelihoods) from among those based on acoustic features, image features, and biochemical features. This means that even if a single feature (for example, an acoustic feature) is insufficient to clearly determine whether a user is person A or person B due to similar feature patterns, adding other features (for example, an image feature) allows for a comprehensive and clear determination of whether the user is person A or person B.

[0096] [Personal Authentication Method] Figure 4 is a flowchart of the learning phase of the personal authentication method according to an embodiment of the present invention. Figure 5 is a flowchart of the authentication phase of the personal authentication method according to an embodiment of the present invention. Hereinafter, the personal authentication method according to an embodiment of the present invention using the personal authentication system 1 will be described with reference to the flowcharts in Figures 4 and 5 and Figure 3.

[0097] [During learning] The learning phase is the stage before the personal authentication system 1 is used (the preparation phase). During learning, the sensor device 20 acquires various types of data (biological information) through each sensor. Specifically, the first sensor 21 acquires biological information (acoustic data) (step S101). The second sensor 22 acquires biological information (image data) of a different type from the acoustic data (step S102). The third sensor 23 acquires biological information (biochemical data) of a different type from the acoustic data and image data (step S103). The various data (biometric information) acquired by each sensor are transmitted to the personal authentication device 10b.

[0098] Next, the personal authentication device 10b extracts features using the feature extraction unit 11 based on the various data (biometric information) it receives. Specifically, the first feature extraction unit 111 extracts acoustic features based on acoustic data (step S201). The second feature extraction unit 112 extracts image features based on image data (step S202). The third feature extraction unit 113 extracts biochemical features based on biochemical data (step S203).

[0099] The features extracted by the feature extraction unit 11, as well as the trained models created during feature extraction, are stored in the memory unit 12 (step S301). Alternatively, an individualized integrated neural network (described later) is constructed, and this individualized integrated neural network is stored in the memory unit 12. This type of training is performed for multiple users, and each user's feature pattern is registered in the memory unit 12.

[0100] [During authentication] The authentication phase refers to the stage (operational phase) when the personal authentication system 1 is actually used. During authentication, the sensor device 20 acquires various data from each sensor, similar to the learning phase (steps S101, 102, 103), and the acquired data is transmitted to the personal authentication device 10b.

[0101] Next, the personal authentication device 10b extracts features using the feature extraction unit 11 based on the various data received, just as during training (steps S201, 202, 203).

[0102] Then, the personal authentication device 10b determines the user based on the extracted feature quantities by the determination unit 13, either by referring to (comparing) the feature patterns for each user registered in the memory unit 12 during learning, or by using an integrated neural network for personalization (step S401). In particular, the personal authentication device 10b converts feature quantities that have time-series information, such as voice and smell, and feature quantities that do not have dynamic information (i.e., do not have time-series information), such as still images, into data related to pulse waves (periodic data), and integrates these to construct an integrated neural network for personal characteristics. Then, using this integrated neural network for personal characteristics, it creates integrated data that shows the personal characteristics of each user, and identifies the user based on this integrated data.

[0103] Finally, the determination unit 13 outputs (notifies) the determination result (authentication result) (step S501).

[0104] Thus, the personal authentication device and personal authentication method of the present invention can improve authentication performance because the user is determined by features extracted based on multiple different biometric information. For example, if there are two users whose sound transmission patterns within the ear canal (acoustic data) are extremely similar, their acoustic features may also be extremely similar, making determination based solely on this difficult. However, since the shape of the ear canal (image data) is unlikely to be extremely similar, the correct user can ultimately be clearly determined by image features.

[0105] Furthermore, since the various biological information acquired by the sensor device 20 is from within the external auditory canal, the personal authentication device of the present invention is less susceptible to the influence of the surrounding environment and can achieve stable judgment accuracy (authentication accuracy). For example, because the external auditory canal is a closed space, secretions and odors tend to accumulate, making it easy to acquire biochemical data, and it is thought that there are individual differences (different characteristic patterns can be obtained for each user).

[0106] The processing of the storage unit 12 and the determination unit 13 described in this second embodiment can be the same as the processing described in this first embodiment. For example, the storage unit 12 can be configured as an authentication difference data storage unit 121 and a registration model unit 122, and the determination unit 13 can be configured as a feature difference data calculation unit and an authentication unit.

[0107] As a result, the differences between the features extracted by each feature extraction unit (first feature extraction unit 111, second feature extraction unit 112, third feature extraction unit 113) are calculated as inter-feature difference data. Then, using this inter-feature difference data and the registration model pre-stored in the storage unit 12 (registration model unit 122), the user (individual) can be authenticated.

[0108] [Examples of application] One example of the application of this invention is the construction industry. In recent years, the Japanese construction industry has seen a decline in the number of workers, leading to a demand for improvements in the working environment at construction sites. For example, the use of professional headsets equipped with bone conduction microphones that enable clear voice communication even in noisy environments is expanding. Therefore, the present invention can be applied to devices and methods that can improve the performance of user (speaker) identification using voice sensors such as bone conduction microphones. Furthermore, it can be applied to methods for verifying license ownership at work sites and for identifying individuals to prevent information leakage, using speaker identification.

[0109] Furthermore, the present invention can be applied to access control systems and health checkup systems. For example, by performing personal authentication based on multiple different types of data such as acoustic data, image data, and biochemical data, it is possible to realize an access control system that manages access to highly secure rooms and facilities where authentication accuracy is required. In other words, it is possible to realize an access control system that eliminates problems such as allowing unauthorized persons to enter due to authentication errors.

[0110] Furthermore, it is possible to realize a health checkup system that performs personal authentication using image data and biochemical data, while simultaneously determining abnormalities (injuries, growths) in the external auditory canal and changes in physical condition from this data. [Industrial applicability]

[0111] This invention relates to a personal authentication device that is less susceptible to the influence of the surrounding environment and can improve authentication performance. Because its biometric authentication technology can be utilized in various fields such as building access control systems and health checkup systems, it is industrially useful. [Explanation of symbols]

[0112] 1. Personal Authentication System 10, 10a, 10b Personal Authentication Device 11 Feature Extraction Unit 111 First Feature Extraction Unit 1111 First feature extraction unit A 1112 First feature extraction unit B 112 Second Feature Extraction Unit 113 Third Feature Extraction Unit 12. Memory Unit / Integrated Data Storage Unit 121 Authentication Difference Data Storage Unit / Integrated Data Storage Unit 122 Registered Model Section 13 Judgment section 131 Feature Difference Data Calculation Unit / Integrated Data Creation Unit 132 Authentication Department 20 Sensor device 21 First Sensor 211 First Sensor A 212 Second Sensor B 22 Second sensor 23. The third sensor

Claims

1. This device acquires multiple different biometric data from the external auditory canal, extracts the characteristic features of these multiple different biometric data using machine learning, and performs personal authentication. A first sensor that acquires acoustic data from inside the user's ear canal. A second sensor for acquiring image data of the user's external auditory canal, A third sensor for acquiring biochemical data from the external auditory canal of the user, Among them, a feature extraction unit extracts features based on data acquired from two or more sensors, A determination unit constructs an individualized neural network based on two or more features extracted by the feature extraction unit, and uses the individualized neural network to extract features based on integrated data obtained by integrating data relating to the user acquired from two or more sensors to determine the user. It has, The second sensor is an infrared sensor and / or an RGB camera, and the image data acquired includes an image showing the shape and temperature distribution within the user's external auditory canal, as well as an image of the vascular network within the external auditory canal. The feature extraction unit estimates the heart rate using an image showing the temperature distribution within the external auditory canal and an image of the vascular network within the external auditory canal, and extracts feature quantities based on the heart rate, the shape of the external auditory canal, the image showing the temperature distribution within the external auditory canal, and the image of the vascular network within the external auditory canal. Personal authentication device.

2. The third sensor is an evanescent wave analysis sensor, a gas chromatography analysis sensor, or a semiconductor odor sensor, and the biochemical data obtained is one or more of the following: secretions in the external auditory canal of the user, odors in the external auditory canal, and volatile organic compounds discharged from the human body. The feature extraction unit extracts feature quantities based on the secreted substance and / or the odor and / or the volatile organic compound. The personal authentication device according to claim 1.

3. The first sensor is a sensor selected from the group consisting of a bone conduction microphone, a vibration sensor, and an acceleration sensor, and acquires one or more of the following as acoustic data: vibration sound in the external auditory canal associated with the user's voice, vibration sound associated with blood flow, and sound due to otoacoustic radiation. The feature extraction unit extracts time-frequency features based on vibrational sounds within the ear canal associated with the user's speech and / or vibrational sounds associated with blood flow and / or sounds from otoacoustic emissions. The personal authentication device according to claim 1 or 2.

4. The second sensor is equipped with a pinhole lens and / or a fisheye lens to enable focusing over a wide range. A personal authentication device according to any one of claims 1 to 3.

5. The second sensor is a sensor equipped with lenses having multiple focal lengths, The feature extraction unit extracts features based on data obtained from one or more focal lengths of the lens. A personal authentication device according to any one of claims 1 to 4.

6. The feature extraction unit generates two or more trained models by performing machine learning using data acquired from two or more sensors, including the first sensor, the second sensor, and the third sensor, as training data, and then uses these trained models to extract features based on the data acquired from the two or more sensors. A personal authentication device according to any one of claims 1 to 5.

7. The determination unit determines the user by comparing the feature quantities for each registered user with the feature quantities extracted by the feature quantity extraction unit. A personal authentication device according to any one of claims 1 to 6.

8. The feature extraction unit extracts two or more features from acoustic features, image features, and biochemical features based on the data acquired from the first sensor, the second sensor, and the third sensor. The determination unit constructs an individual-integrated neural network using these features, and uses this individual-integrated neural network to create integrated data in which two or more determination probabilities from among the determination probabilities based on acoustic features, the determination probabilities based on image features, and the determination probabilities based on biochemical features are integrated, and the user is determined based on this integrated data. A personal authentication device according to any one of claims 1 to 7.

9. A microphone and / or earphone comprising a first sensor, a second sensor, and a third sensor according to any one of claims 1 to 8, A personal authentication device according to any one of claims 1 to 8, A personal authentication system that has the following features.

10. An access control system or health check system using the personal authentication system described in claim 9.

11. This method involves acquiring multiple different biometric data from the external auditory canal, extracting the characteristic features of these multiple biometric data using machine learning, and performing personal authentication. A first sensor that acquires acoustic data from inside the user's ear canal. A second sensor for acquiring image data of the user's external auditory canal, A third sensor for acquiring biochemical data from the external auditory canal of the user, Of these, the process includes extracting features using a feature extraction unit based on data acquired from two or more sensors, The process involves constructing an individualized neural network by a determination unit based on two or more extracted feature quantities, and using the individualized neural network to extract feature quantities based on integrated data obtained from two or more sensors to determine the user, and It has, The second sensor is an infrared sensor and / or an RGB camera, and the image data acquired includes an image showing the shape and temperature distribution within the user's external auditory canal, as well as an image of the vascular network within the external auditory canal. The feature extraction unit estimates the heart rate using an image showing the temperature distribution within the external auditory canal and an image of the vascular network within the external auditory canal, and extracts feature quantities based on the heart rate, the shape of the external auditory canal, the image showing the temperature distribution within the external auditory canal, and the image of the vascular network within the external auditory canal. Personal authentication method.

12. Computers, This device acquires multiple different biometric data from the external auditory canal, extracts the characteristic features of these multiple different biometric data using machine learning, and performs personal authentication. A first sensor that acquires acoustic data from inside the user's ear canal. A second sensor for acquiring image data of the user's external auditory canal, A third sensor for acquiring biochemical data from the external auditory canal of the user, Among them, a feature extraction unit extracts features based on data acquired from two or more sensors, A determination unit constructs an individualized neural network based on two or more features extracted by the feature extraction unit, and uses the individualized neural network to extract features based on integrated data obtained by integrating data relating to the user acquired from two or more sensors to determine the user. It has, The second sensor is an infrared sensor and / or an RGB camera, and the image data acquired includes an image showing the shape and temperature distribution within the user's external auditory canal, as well as an image of the vascular network within the external auditory canal. The feature extraction unit estimates the heart rate using an image showing the temperature distribution within the external auditory canal and an image of the vascular network within the external auditory canal, and extracts feature quantities based on the heart rate, the shape of the external auditory canal, the image showing the temperature distribution within the external auditory canal, and the image of the vascular network within the external auditory canal. A personal authentication program that functions as a device.

Images