Authentication device, biological information acquisition device, and electronic apparatus

Abstract

Provided is a small-scale and highly accurate biological information acquisition device. An acquired photoplethysmogram is converted into a plurality of frequency components by fast Fourier transform, and a portion of the resultant data is quantized and input to a neural network (NN) to perform personal identification or emotion recognition. In this case, quantization to a low number of bits makes it possible to use a small-scale NN and to perform high-speed determination. Furthermore, performing quantization reflecting flag representation makes it possible to enhance accuracy. Since the NN is small-scale, biological information acquisition processing can be performed at high speed and safely without using a highly functional cloud-based workstation or the like.

Description

Authentication device, biometric information acquisition device, and electronic device

[0001] One aspect of the present invention relates to a biometric information acquisition device and an electronic device including the biometric information acquisition device.

[0002] Note that one embodiment of the present invention is not limited to the above technical field. Examples of the technical field of one embodiment of the present invention disclosed in this specification and the like include semiconductor devices, display devices, light-emitting devices, power storage devices, memory devices, electronic devices, lighting devices, input devices, input / output devices, driving methods thereof, and manufacturing methods thereof. A semiconductor device refers to any device that can function by utilizing semiconductor characteristics.

[0003] In the information society, biometric authentication, which is a stronger authentication method than password authentication, is being introduced to prevent unauthorized access. Biometric authentication is an authentication method that uses biometric information unique to an individual, and methods that mainly use image data, such as fingerprint authentication, palm print authentication, iris authentication, and face authentication, are widely used.

[0004] In biometric authentication using image data, image data acquired during authentication is analyzed and an individual is identified based on whether the image data matches pre-registered image data. For example, Patent Literature 1 discloses an example of personal authentication using an imaging device incorporating a light source.

[0005] Whether or not the image data matches the above can be inferred by artificial intelligence. For example, Non-Patent Document 1 reports an example of an AI accelerator that has multiple neural networks and is capable of performing different inferences in parallel.

[0006] There are also attempts to recognize human emotions. If we could read human emotions, it would be possible to provide services according to emotions, and accumulate emotional data for use in healthcare. Furthermore, by providing external stimuli to alleviate negative emotions, it would be possible to prevent crimes and accidents.

[0007] Human emotions are easily revealed in facial expressions, but because people can hide changes in their appearance, it is difficult to read their true emotions from their appearance alone. Pulse waves are one of the bodily changes that are difficult for people to intentionally manipulate. Non-Patent Documents 2 and 3 report on the recognition of emotions using pulse waves.

[0008] WO2020 / 049398

[0009] Y. Yakubo et al. , “Crystalline Oxide Semiconductor-based 3D Bank Memory System for Endpoint Artificial Intelligence with Multiple Neural Networks Facilitating Context Switching and Power Gating,”ISSCC, 2023. Lee MS, Lee YK, Pae DS, Lim MT, Kim DW, Kang TK. “Fast Emotion Recognition Based on Single Pulse PPG Signal with Convolutional Neural Network.”Applied Sciences. 2019;9(16):3355. S. -W. Wang and S. -N. Yu, “Emotion Recognition Based on Photoplethysmography Using ResNet and BiLSTM Networks,” 2021 International Conference on e-Health and Bioengineering (EHB), Iasi, Romania, 2021, pp. 1-4, doi:10.1109 / EHB52898.2021.9657742.

[0010] Although biometric authentication using image data is a strong authentication method, it may require a high-resolution imaging device and a large-scale information processing device for processing a huge amount of image information.

[0011] Furthermore, while authentication devices are important as a security measure, they are also positioned as a supplementary step toward moving on to the next step. Furthermore, emotion recognition devices are also expected to be used in conjunction with other devices in practice. In other words, biometric information acquisition devices such as authentication devices and emotion recognition devices are often used not standalone, but connected to or incorporated into other devices. Therefore, while sufficient discrimination functionality is necessary for identifying individuals or recognizing emotions, simple, small-scale biometric information acquisition devices are also desired.

[0012] Furthermore, in biometric authentication using image data, glasses, masks, hats, gloves, or injuries can hinder the accuracy of the information. Furthermore, the influence of natural or artificial light can make it difficult to accurately obtain image data. Also, as mentioned above, people can hide their emotions, making it difficult to read their emotions from their appearance.

[0013] Therefore, an object of one embodiment of the present invention is to provide a small-scale, highly accurate biometric information acquisition device. Another object is to provide a biometric information acquisition device that can easily acquire accurate information. Another object is to provide a biometric information acquisition device that can acquire information without being significantly affected by the surrounding environment. Another object is to provide a novel biometric information acquisition device. Another object is to provide a method for operating the biometric information acquisition device. Another object is to provide an electronic device that includes the biometric information acquisition device. Another object is to provide a novel semiconductor device or the like.

[0014] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these can be extracted from the description of the specification, drawings, claims, etc.

[0015] One aspect of the present invention relates to a small-scale, highly accurate biometric information acquisition device.

[0016] One aspect of the present invention is a biometric information acquisition device having the functions of acquiring pulse waves as analog data, converting the analog data into digital data, performing a fast Fourier transform on the digital data, extracting multiple points of data from the low-frequency side of the data after the fast Fourier transform, quantizing each of the multiple points of data, and authenticating an individual or recognizing emotions by inference using each of the quantized data.

[0017] The pulse wave is preferably a photoplethysmogram, and the photoplethysmogram is preferably acquired at a sampling rate of 10 Hz or more and 100 Hz or less.

[0018] The number of data points extracted from the low frequency side is preferably between 150 and 200. The data is preferably up to a maximum of 12 Hz.

[0019] The quantized data is preferably 4-bit notated 5-value data that reflects the flag notation.

[0020] For inference using quantized data, it is preferable to use a fully connected neural network.

[0021] Emotion recognition can be performed for either positive or negative pleasantness or positive or negative arousal, or for four emotions: "joy," "anger," "sadness," and "pleasure."

[0022] The present invention also provides a wristwatch-type electronic device having a biometric information acquisition device, a case, an optical sensor module, and a light-receiving window, the light-receiving window being provided on the surface of the case that comes into contact with the wrist, and the optical sensor module being installed in the case so as to overlap with the light-receiving window. The biometric information acquisition device can also have a function of inferring physical condition from pulse waves using a neural network model that has learned the relationship between pulse waves and brain waves.

[0023] One embodiment of the present invention can provide a biometric information acquisition device that can easily acquire accurate information. Alternatively, it can provide a biometric information acquisition device that can acquire information without being significantly affected by the surrounding environment. Alternatively, it can provide a small-scale and highly accurate biometric information acquisition device. Alternatively, it can provide a novel biometric information acquisition device. Alternatively, it can provide a method for operating the biometric information acquisition device. Alternatively, it can provide a novel semiconductor device and a method for operating the same.

[0024] Note that the description of these effects does not preclude the existence of other effects. Note that one embodiment of the present invention does not necessarily have all of these effects. Note that effects other than these can be extracted from the description in the specification, drawings, claims, etc.

[0025] FIG. 1 is a block diagram illustrating a biometric information acquisition device. FIG. 2 is a flowchart illustrating the operation of the biometric information acquisition device. FIG. 3A is a diagram illustrating a photoplethysmogram (after digital conversion). FIG. 3B is a diagram illustrating data after a Fourier transform of the photoplethysmogram. FIGS. 4A and 4B are diagrams illustrating the accuracy rate and cross entropy error of a neural network. FIG. 5 is a diagram illustrating the accuracy rate for each quantization method. FIG. 6 is a diagram illustrating an example of a neural network. FIG. 7 is a diagram illustrating an emotion selection model. FIGS. 8A and 8B are diagrams illustrating emotion inference results. FIGS. 9A to 9D are diagrams illustrating a biometric information acquisition device. FIGS. 9E and 9F are diagrams illustrating an electronic device having a biometric information acquisition device. FIGS. 10A to 10C are diagrams illustrating an electronic device having a biometric information acquisition device. FIGS. 11A and 11B are diagrams illustrating an electronic device having a biometric information acquisition device.

[0026] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and those skilled in the art will readily understand that various modifications in form and detail may be made without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below. In the configuration of the invention described below, the same parts or parts having similar functions will be designated by the same reference numerals in different drawings, and repeated description thereof may be omitted. Hatching of the same elements constituting the drawings may be omitted or changed as appropriate in different drawings.

[0027] In this specification, "connection" includes, as an example, "electrical connection." Note that the term "electrical connection" is sometimes used to define the connection relationship between circuit elements as a physical entity. Furthermore, "electrical connection" includes "direct connection" and "indirect connection." "A and B are directly connected" means that A and B are connected without the intervention of a circuit element (e.g., a transistor, a switch, etc.; wiring is not considered a circuit element). On the other hand, "A and B are indirectly connected" means that A and B are connected via one or more circuit elements.

[0028] For example, assuming that a circuit including A and B is operating, if there is a time during the operation of the circuit when an electrical signal is exchanged or an interaction of electrical potential occurs between A and B, then it can be defined that "A and B are indirectly connected" as objects. Note that even if there is a time during the operation of the circuit when no electrical signal is exchanged or an interaction of electrical potential occurs between A and B, it can still be defined that "A and B are indirectly connected" as long as there is a time during the operation of the circuit when an electrical signal is exchanged or an interaction of electrical potential occurs between A and B.

[0029] An example of a case where "A and B are indirectly connected" is when A and B are connected via the source and drain of one or more transistors. On the other hand, an example of a case where it cannot be said that "A and B are indirectly connected" is when an insulator is present in the path from A to B. Specifically, there are cases where a capacitive element is connected between A and B, and cases where a gate insulating film of a transistor is present between A and B. Therefore, it cannot be said that "the gate (A) of a transistor and the source or drain (B) of the transistor are indirectly connected."

[0030] Another example of a case where it cannot be said that "A and B are indirectly connected" is when multiple transistors are connected via their sources and drains to the path from A to B, and a constant potential V is supplied to a node between one transistor and another transistor from a power supply, GND, etc.

[0031] Embodiment 1 In this embodiment, an authentication device using a pulse wave will be described as a biometric information acquisition device according to one embodiment of the present invention.

[0032] One aspect of the present invention is a small-scale, highly accurate authentication device. This authentication device uses pulse waves, which are biometric information. Pulse waves are waveforms that indicate changes in internal arterial pressure, and this information can be acquired without significantly affecting the surrounding environment. Pulse waves or similar information can be obtained from electrocardiograms, photoplethysmograms, blood pressure, heart sounds, and the like. Photoplethysmograms, in particular, can be obtained using a small optical sensor, allowing for the construction of a low-cost authentication device.

[0033] In one aspect of the present invention, the acquired photoplethysmogram is converted into multiple frequency components using a fast Fourier transform, and a portion of the data is quantized and input into a neural network (NN) to identify the individual. By quantizing the data to a low number of bits and inputting it, a small-scale NN with few input and intermediate layers can be used, and high-speed determination can be made. Furthermore, quantization that reflects flag notation can improve accuracy.

[0034] In one aspect of the present invention, a small-scale neural network is used, allowing for high-speed learning and inference. Furthermore, because the neural network is small, calculations can be performed using a microcomputer, a personal computer, a tablet computer, a smartphone, or the like, without using a high-performance workstation or the like that has been required for large-scale neural network calculations up to now. In other words, inference processing and the like are possible without using data communication via the Internet or the like, allowing for high-speed and secure authentication processing.

[0035] 1 is a block diagram illustrating an authentication device according to one embodiment of the present invention. The authentication device 130 can include a sensor 101, a microcomputer 120, and a display unit 110. Note that the block diagram illustrated in FIG. 1 can also be applied to an emotion recognition device 140, which will be described later. Therefore, in FIG. 1, the authentication device 130 and the emotion recognition device 140 are collectively denoted as 130 (140).

[0036] The authentication device 130 has a function of determining the biometric information acquired by the sensor 101 using a microcomputer 120 having an NN function, and displaying the authentication result on the display unit 110. The microcomputer 120 is not limited to a one-chip type, and may be a board computer type.

[0037] Furthermore, the authentication device 130 is not limited to being an independent device, but may be a part of device A having other functions, or may be incorporated into device A. In other words, whether or not the functions of device A can be used can be determined depending on the authentication result by the authentication device 130.

[0038] Furthermore, the elements of authentication device 130 may be shared within device A. For example, if device A is an information terminal such as a personal computer, a tablet computer, or a smartphone, these devices have elements equivalent to those of microcomputer 120, and therefore some or all of the elements of authentication device 130 can be shared with device A.

[0039] For example, the camera of the smartphone can be used as the sensor 101, the input / output unit, calculation unit, and memory unit of the smartphone can be used as elements of the microcomputer 120, and the display unit of the smartphone can be used as the display unit 110.

[0040] In one aspect of the present invention, a pulse wave is used as the biological information acquired by the sensor 101. The pulse wave can be acquired from devices that measure electrocardiograms, photoplethysmograms, blood pressure, or heart sounds, but the devices that measure electrocardiograms and blood pressure are large in scale, and heart sounds are easily affected by environmental noise. Therefore, photoplethysmograms are preferred as a method for acquiring pulse waves. Photoplethysmograms can be acquired as analog data using a small optical sensor.

[0041] Although photoplethysmography is easily affected by ambient light, accurate data can be obtained by placing the optical sensor and light source in a dark place. For example, when obtaining data from a fingertip, the optical sensor can be placed inside a box-shaped, bag-shaped, or band-shaped light-shielding object that covers the fingertip. Alternatively, data can be obtained in a bright place by using an optical sensor for ambient light and performing processing such as taking the difference from the ambient light.

[0042] Because blood contains oxyhemoglobin, which absorbs light G at wavelengths of 550 nm and nearby, changes in the volume of blood vessels change the amount of light G absorbed. Therefore, by capturing the reflected or transmitted light of light G irradiated onto a living body, it is possible to obtain a pulse wave that changes in accordance with the pulsation of the heart.

[0043] A photodiode or a phototransistor can be used as the optical sensor for detecting light G. A green-emitting LED element or the like is suitable as the light source. Therefore, it is preferable to use an optical sensor module with a light source as the sensor 101.

[0044] The microcomputer 120 has an input unit 102, a calculation unit 103, a calculation unit 104, a memory unit 105, a memory unit 106, an ADC (analog-to-digital conversion circuit) 107, a display driver 108, and an output unit 109. The above elements are connected to each other via a bus line 115. The microcomputer 120 may have elements other than the above elements. Alternatively, the microcomputer 120 may be configured without some of the above elements. Alternatively, the microcomputer 120 may be configured with some of the above elements integrated together.

[0045] The input unit 102 has an interface function for inputting information from the outside. The input information includes analog data of biometric information input from the sensor 101, a trained NN model, and the like. The input unit 102 can also input biometric data acquired by an external device 112. The data is not limited to photoplethysmograms, but may also be, for example, data of electrocardiograms, blood pressure, heart sounds, or electroencephalograms. These data can also be used instead of pulse waves for biometric authentication. The data input from the outside is not limited to analog data, but may also be digital data.

[0046] The calculation unit 103 may have, for example, a central processing unit (CPU) and has a function of controlling each element such as the input unit 102, the calculation unit 103, the calculation unit 104, the storage unit 105, the storage unit 106, the ADC 107, the display driver 108, and the output unit 109.

[0047] Signals are transmitted between the calculation unit 103 and each element via a bus line 115. The calculation unit 103 has functions such as processing signals input from each element connected via the bus line 115 and generating signals to be output to each element, and can comprehensively control each element connected to the bus line 115.

[0048] The calculation unit 103 performs various data processing and program control by interpreting and executing commands from various programs using a CPU. The programs that can be executed by the calculation unit 103 may be stored in a memory area provided within the calculation unit 103, or may be stored in the storage unit 105 or the storage unit 106 described below.

[0049] It is preferable to use a computing device specialized for parallel computation rather than a CPU as the computing unit 104. For example, it is preferable to use a computing device having a large number of cores (tens to hundreds) capable of parallel processing, such as a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), or an NPU (Neural Processing Unit). These can function as AI accelerators and can perform NN-related computations at high speed. Note that a fully connected NN or a convolutional NN can be used as the NN.

[0050] The calculation unit 104 may be configured to be implemented by a programmable logic device (PLD) such as a field programmable gate array (FPGA) or a field programmable analog array (FPAA). Alternatively, the calculation unit 104 may be configured as a dedicated AI accelerator implemented by an application specific integrated circuit (ASIC). Although the calculation unit 103 and the calculation unit 104 are shown as separate elements in FIG. 1, they may be integrated elements.

[0051] The storage unit 105 may be, for example, a storage device having a storage element such as a dynamic RAM (DRAM) or a static RAM (SRAM). These storage elements are volatile but can be accessed at high speed, and can be used as auxiliary storage areas for the calculation units 103 and 104. They can also be used as a storage area for data input from the sensor 101, a storage area for analog-to-digital converted data, a storage area for output data after calculation, or an area for temporarily storing programs read from the storage unit 106.

[0052] The storage unit 106 may be, for example, a storage device having a nonvolatile storage element such as a flash memory, a magnetoresistive random access memory (MRAM), a phase change RAM (PRAM), a resistive RAM (ReRAM), or a ferroelectric RAM (FeRAM). Alternatively, a mask ROM, a one-time programmable read-only memory (OTPROM), an erasable programmable read-only memory (EPROM), or the like may be used. Alternatively, a recording media drive such as a hard disk drive (HDD) or a solid state drive (SSD) may be used.

[0053] Although these are nonvolatile, their access speeds are slow, so it is preferable to store information that is not frequently accessed by the calculation units 103 and 104 and that does not need to be rewritten or is rewritten infrequently. For example, BIOS (Basic Input / Output System), firmware, an operating system, application programs, programs related to the operation of the authentication device 130, learning information for the NN, etc. can be stored.

[0054] At least some of the elements included in the above-described processing units 103 and 104 and the memory units 105 and 106 preferably include transistors using an oxide semiconductor in a semiconductor layer (hereinafter referred to as OS transistors). Because OS transistors have extremely low off-state current, they are preferably used as switches for retaining charge (data) flowing into a capacitor functioning as a memory element. By using an OS transistor as the switch, data can be retained for a long period of time. By using this characteristic in at least one of the register and cache memory included in the processing unit, the processing unit can be operated only when necessary and can be turned off at other times by saving information from the previous processing to the memory element. In other words, normally-off computing is possible, and the power consumption of the microcomputer 120 can be reduced.

[0055] The ADC 107 is an analog-to-digital conversion circuit and has a function of converting analog data input to the input unit 102 into digital data. For example, it can convert analog data of biological information acquired by the sensor 101 into 10-bit digital data. Note that the calculation unit 103 or 104 may have the function of the ADC 107. Alternatively, the sensor 101 may have a function of outputting digital data. In this case, the ADC 107 may be omitted.

[0056] The display driver 108 is a driver circuit that generates image data for displaying the authentication result and outputs the image data and a drive signal to the display unit 110. The form of the driver circuit differs depending on the form of the display unit 110. Also, while Fig. 1 shows an example in which the authentication result is displayed on the display unit 110, the authentication result may also be output by sound, vibration, or the like.

[0057] The output unit 109 has an interface function that outputs data handled by the authentication device 130 to the outside. For example, NN learning and inference can be performed within the microcomputer 120, but data acquired by the sensor 101 can also be output to the outside via the output unit 109 and processed by an AI workstation (AI-WS) 111 on the cloud. By performing some data processing within the microcomputer 120 and then transmitting the data to the AI ​​workstation 111, it is possible to reduce the power consumption for data transmission and the load on the AI ​​workstation 111.

[0058] The NN learning model or inference results generated by the AI ​​workstation 111 can be input into the microcomputer 120 via the input unit 102. Note that the external NN learning device and inference device are not limited to the AI ​​workstation 111, and may also be a personal computer, a tablet terminal, a smartphone, or the like.

[0059] Furthermore, the output unit 109 can output information relating to the authentication result to the external device 113. The external device 113 can grant access rights to the user depending on the authentication result.

[0060] 1 shows an example in which the input unit 102 and the output unit 109 are provided, but they may be integrated into a single input / output unit. Furthermore, the input unit 102 and the output unit 109, or the input / output unit, may have a wireless communication function. When performing wireless communication, a communication protocol or technology that can be used may be, for example, a communication standard such as LTE (Long Term Evolution), or a communication specification established by the IEEE, such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0061] Next, an example of the operation of the authentication device 130 will be described using the flowchart shown in Fig. 2. Note that on the right side of the flowchart, the elements in Fig. 1 that correspond to each step are shown.

[0062] First, in step S1, pulse wave information (analog data) is acquired using the sensor 101. Here, it is necessary to accurately acquire the shape of the pulse wave, but it is preferable to have a small amount of data in order to perform subsequent calculations efficiently.

[0063] In other words, it is appropriate to acquire several tens of points of numerical data for one pulse wave (a waveform corresponding to one heartbeat). Since a normal human pulse rate is 60 to 100 beats per minute, the sampling rate should be between 10 Hz and 100 Hz, preferably between 20 Hz and 80 Hz, and more preferably between 30 Hz and 60 Hz.

[0064] The sampling time is preferably long during neural network learning and short during authentication. For example, it is preferably about 15 seconds during learning so that multiple pulse waves can be acquired, and it is preferably 1 to 3 seconds during authentication so that at least one pulse wave can be acquired. If further improvement in authentication accuracy is desired, it is preferably 2 to 6 seconds so that two or three pulse waves can be acquired.

[0065] Next, in step S2, the pulse wave (analog data) is converted into digital data using the ADC 107. Fig. 3A shows an example of a pulse wave illustrated using the values ​​after digital conversion. Note that the data shown in Fig. 3A was acquired at a sampling rate of 64 Hz.

[0066] Next, in step S3, the digital data of the pulse wave is subjected to a fast Fourier transform using the calculation unit 103 to resolve it into frequency components. Fig. 3B shows the result of performing a fast Fourier transform on the pulse wave shown in Fig. 3A, with the amplitude spectrum normalized. Note that while a method using a fast Fourier transform is exemplified here, a discrete cosine transform can also be used.

[0067] Next, in step S4, a plurality of continuous data points are extracted from the results of the fast Fourier transform, starting from the low-frequency side. As shown in FIG. 3B, almost no clear frequency components are extracted on the high-frequency side above approximately 12 Hz. Therefore, it is assumed that the feature values ​​are on the low-frequency side below approximately 12 Hz, and the high-frequency side can be ignored. Furthermore, limiting the range of data is effective in reducing the size of the neural network.

[0068] Here, to determine the effective data range, we calculated the accuracy rate of the NN inference when the number of data items was changed, and the results are explained. Note that the input data was 32-bit floating point, and a fully connected NN model with 128x3 intermediate layers (activation function = Tanh) and 10 output layers (activation function = softmax) was used for the calculation.

[0069] In addition, in the process of step S1, pulse wave information was acquired from four individuals, and 15.6 seconds of data (64 Hz, 1000 points) x 100 was acquired for each of the four individuals. The data was processed from steps S2 to S4, and 80% of it was used for learning and 20% was used for inference. The accuracy rate was calculated by identifying individuals by comparing the numbers assigned to each output layer during learning and the output layer number with the maximum value during inference.

[0070] 4A shows the relationship between the number of data (the number of consecutive data extracted from the low-frequency side) and the accuracy rate during inference, as well as the cross-entropy error during learning and inference. The cross-entropy error during learning is small even when the number of data is small, but the entropy error during inference continues to decrease until the number of data exceeds 150. This region suggests overlearning due to a small number of data.

[0071] From the above, it can be said that the number of data points is preferably 150 or more, but from the viewpoint of reducing the size of the neural network, it is preferable to have a small number of data points. Therefore, it can be said that the number of data points is appropriate to be 150 or more and close to 150, and for example, the range of the effective number of data points can be approximately 150 to 200. It can also be said that the number of data points of 150 is the point at which the accuracy rate begins to show a tendency to saturate. Therefore, in order to obtain more accurate inference results, the optimal number of data points will be treated as 175 below.

[0072] Figure 4B is a diagram in which the horizontal axis of Figure 4A is replaced with the maximum frequency. Since the maximum frequency corresponding to 175 data points is between 11 Hz and 12 Hz, it can be said that data up to about 12 Hz is effective. From this result, it can be said that the clear data shown in the fast Fourier transform results in Figure 3B is used almost effectively.

[0073] Next, in step S5 of Fig. 2, the data extracted in step S4 (for example, data of 175) is quantized. Since the data extracted in step S4 is a 32-bit floating point, a large-scale neural network is required to perform calculations as is. Therefore, the data is quantized to a low bit number so that it can be used with a small-scale neural network.

[0074] There are various methods of quantization, and the inventors have tried the following as references: a method of performing calculations on the original data using 32-bit floating point; method A of dividing the original data into 4-bit numerical values; method B of dividing the original data into 4-bit numerical values ​​after logarithmic transformation; and method C of dividing the original data using several threshold values ​​and displaying it as 4-bit data that reflects flag notation.

[0075] Figure 5 shows the accuracy rate of inference in the above methods. Excluding the reference (ref.), the method with the highest accuracy rate was method C, which used data reflecting flag notation, and achieved an accuracy rate of over 90%, close to 95% of the reference.

[0076] The details of Method C are as follows: First, the original data, which is a 32-bit floating point, is normalized to 0 to 255, and threshold values ​​are set for the 0th bit, the 1st bit, the 2nd bit, and the 3rd bit, respectively. If the value of the normalized original data (hereinafter referred to as data S) is equal to or greater than each threshold value, a flag (1) is set.

[0077] Specifically, if the value of data S is less than the threshold value of the 0th bit, it is set to

[0000] . If the value of data S is equal to or greater than the threshold value of the 0th bit and less than the threshold value of the 1st bit, a flag is set to the 0th bit and it is set to

[0001] . If the value of data S is equal to or greater than the threshold value of the 1st bit and less than the threshold value of the 2nd bit, it is set to flag the 0th and 1st bits and it is set to

[0011] . If the value of data S is equal to or greater than the threshold value of the 2nd bit and less than the threshold value of the 3rd bit, it is set to flag the 0th to 2nd bits and it is set to

[0111] . If the value of data S is equal to or greater than the threshold value of the 3rd bit, it is set to flag the 0th to 3rd bits and it is set to

[1111] . In other words, data S is classified into five values ​​in 4-bit notation. This notation is called flag notation.

[0078] Table 1 shows the results of calculations using the fully connected neural network model described above, applying multiple different thresholds to the 175 pieces of data S extracted in step S4 according to the above rules. It can be seen that the cross-entropy error and accuracy rate differ depending on the threshold conditions in trials No. 1 to No. 13. Considering overlearning, it can be determined that the threshold settings of No. 7 or No. 8, which produce small cross-entropy error values ​​during inference, are appropriate.

[0079]

[0080] The threshold values ​​in Table 1 were experimentally set by the inventors, but it is possible to set appropriate threshold values ​​by learning the neural network using cross-entropy error and accuracy rate as the criteria for judgment. In the above example, data S is classified into five values ​​in four-bit notation, but this is not limiting. For example, data S can be classified into four values ​​in three-bit notation, six values ​​in five-bit notation, etc.

[0081] 2, the data quantized in step S5 is input to the neural network (NN) in the calculation unit 104, where calculation is performed. A fully connected NN or a convolutional NN can be used as the NN. Since each of the 175 pieces of data is quantized to 4 bits in steps S4 and S5, the NN only needs to be able to input a minimum of 700 pieces of 1-bit data.

[0082] 6 shows a fully connected neural network (NN) shown in Non-Patent Document 1, which has an input layer of 784, an intermediate layer of 128 × 3, and an output layer of 10. By quantizing pulse wave data so that such a small-scale NN can be used, inference can be performed at high speed.

[0083] In step S7, if the calculation is during learning, the process returns to step S1, and the processes of steps S2 to S6 are repeated using different data. If the calculation is during inference, the process proceeds to step S8, where the individual is identified and the authentication result is displayed on the display unit 110. Note that part or all of the learning and inference processes may be performed by the AI ​​workstation 111 or the like.

[0084] As described above, by using pulse waves as biometric information and appropriately quantizing the data, it is possible to identify an individual even with a small-scale neural network. In other words, by using one aspect of the present invention, it is possible to form a small-scale, highly accurate authentication device.

[0085] This embodiment mode can be implemented by appropriately combining at least a part thereof with other embodiment modes described in this specification.

[0086] Embodiment 2 In this embodiment, an emotion recognition device using a pulse wave will be described as a biometric information acquisition device according to one embodiment of the present invention.

[0087] The emotion recognition device described in this embodiment can be configured using elements common to the authentication device 130 described in embodiment 1, and the configuration of the device used as the NN and its operation flow can also be basically the same. Therefore, for the configuration of the emotion recognition device described in this embodiment and the device used as the NN, the description of the configuration of the authentication device 130 described above can be referenced, and detailed description thereof will be omitted.

[0088] One aspect of the present invention is a small-scale, highly accurate emotion recognition device. The emotion recognition device uses pulse waves, which are biological information. In one aspect of the present invention, a small-scale neural network is used, allowing for high-speed learning and inference. Furthermore, because the neural network is small, calculations can be performed without using high-performance workstations, which have been required for large-scale neural network calculations up to now. In other words, inference processing and the like can be performed without using data communication via the Internet, etc., allowing for high-speed and safe recognition of human emotions.

[0089] By recognizing human emotions, it will be possible to, for example, provide services according to emotions, use data on emotional trends in healthcare, or provide external stimuli to alleviate negative emotions, thereby preventing crime or accidents.

[0090] 1 can be applied to the emotion recognition device, similar to the authentication device 130. The emotion recognition device 140 can be configured to include a sensor 101, a microcomputer 120, and a display unit 110.

[0091] The authentication device 130 shown in the first embodiment is used with the intention of the user himself / herself performing authentication, so it is appropriate to use a contact-type optical sensor module or the like with a clear usage method as the sensor 101. On the other hand, an emotion recognition device is used not only for recognizing the emotion of a user at a timing intended by the user, but also for recognizing the emotion of a user at a timing unintended by the user, or for recognizing the emotion of a person other than the user. Therefore, it is appropriate to use not only a contact-type sensor but also a camera module or the like that can acquire information in a non-contact manner as the sensor 101.

[0092] Next, an example of the operation of the emotion recognition device 140 will be described with reference to the flowchart shown in FIG.

[0093] First, pulse wave information (analog data) is acquired in step S1, and the analog data is converted to digital data in step S2. The appropriate conditions for acquiring the pulse wave can be found in the explanation of the operation of the authentication device 130.

[0094] An emotion recognition device requires training data corresponding to each of the various emotions that humans experience. In this embodiment, two subjects watch videos (14 videos in total) of up to about five minutes each that are estimated to evoke one of the emotions of joy, anger, sadness, or happiness, and pulse waves are acquired while the subjects are watching the videos.

[0095] In Russell's circumplex model, human emotions can be expressed as a combination of pleasantness (unpleasant-pleasant) and arousal (calm-excited). Figure 7 shows an emotion selection model based on Russell's circumplex model. The X-axis is valence, which represents pleasantness (valence), with the negative (-) region representing unpleasant emotions and the positive (+) region representing pleasant emotions. The Y-axis is valence, which represents arousal (arousal), with the negative (-) region representing non-arousal emotions and the positive (+) region representing arousal emotions.

[0096] The model in FIG. 7 has four quadrants. The first quadrant Q1 represents an emotion with a positive comfort level and a positive arousal level, and this emotion is defined as "joy" among the three levels of happiness. The second quadrant Q2 represents an emotion with a negative comfort level and a positive arousal level, and this emotion is defined as "anger" among the three levels of happiness. The third quadrant Q3 represents an emotion with a negative comfort level and a negative arousal level, and this emotion is defined as "sad" among the three levels of happiness. The fourth quadrant Q4 represents an emotion with a positive comfort level and a negative arousal level, and this emotion is defined as "happy" among the three levels of happiness. Note that a value of 0 is not included in any quadrant, so here, positive values ​​are defined as ≧0 and negative values ​​are defined as <0.

[0097] After viewing each video, participants are given a score based on the intensity of their emotions toward it. Scoring is done on a nine-point scale from -4 to +4 for both comfort level (X) and arousal level (Y), and the pulse waves acquired while viewing each video are labeled as positive or negative for comfort level (left or right of the model), positive or negative for arousal level (top or bottom of the model), and any of the first to fourth quadrants Q1 to Q4 for emotions.

[0098] For example, in the case of targeting the first quadrant Q1 to the fourth quadrant Q4, if the comfort level is +2 and the arousal level is +3 while watching a video, the pulse wave data is labeled as "joy" in the first quadrant Q1. Also, if the comfort level is -1 and the arousal level is -4 while watching a video, the pulse wave data is labeled as "sad" in the third quadrant Q3. These labeled pulse wave data can be used as learning data.

[0099] Note that pulse waves are not only affected by watching videos, but are also affected by other factors. For example, pulse waves can change due to external stimuli such as sound or physical contact, or physical movement. Furthermore, noise or malfunctions may occur in the device that acquires the pulse waves. Since such pulse wave data becomes noise for neural network learning and inference, it is preferable to delete it.

[0100] For example, if the difference between the maximum and minimum values ​​in the acquired pulse wave data (see FIG. 3A ) is equal to or greater than value A or equal to or less than value B, the data may be deleted. Values ​​A and B can be freely set within a range that allows determination of an abnormal pulse wave. Note that noise removal may be performed after Fourier transform. Note that if a stable pulse wave is acquired, noise removal may not be necessary.

[0101] Next, in step S3, the digital pulse wave data is subjected to a fast Fourier transform and broken down into frequency components. The digital pulse wave data is acquired for several seconds, and it is preferable to use data acquired over a period of time that allows one or two pulse waves to be acquired. In this embodiment, 256 points of data acquired at 64 Hz for four seconds are used.

[0102] Next, in step S4, a plurality of consecutive data points are extracted from the result of the fast Fourier transform, starting from the low frequency side. Here, as with the authentication device 130 described in the first embodiment, 175 data points (data points greater than 0 Hz and up to about 12 Hz) are used.

[0103] Next, in step S5, the data extracted in step S4 is quantized. Here, the original 32-bit floating point data is normalized to 0 to 15, and similar to the quantization by method C of the authentication device 130 described in the first embodiment, several threshold values ​​are applied to the 175-point original data (data S), and the data is converted into five-value 4-bit data (

[0000] ,

[0001] ,

[0011] ,

[0111] ,

[1111] ) that reflects the flag notation.

[0104] Here, we will explain the results of searching for an appropriate threshold value using a fully connected NN similar to that of the authentication device 130. Of the data processed in steps S1 to S4 described above, 80% was used for learning and 20% was used for inference.

[0105] The accuracy rate during inference was calculated by comparing the numbers corresponding to the labels assigned to each output layer during learning with the output layer number that had the maximum value during inference. Here, the labels are labels assigned to the pulse waves acquired during video viewing that correspond to whether the comfort level is positive or negative, whether the arousal level is positive or negative, and which of the first to fourth quadrants Q1 to Q4 the pulse waves belong to.

[0106] The threshold search was first performed for each of the comfort level and the arousal level, and calculations were performed by assigning multiple thresholds to the 175 data S extracted in step S4 according to the above rules. The thresholds used in the search were 0.1 to 0.5 (five conditions of Δ0.1) for the 0th bit, 1.5 to 1.9 (five conditions of Δ0.1) for the 1st bit, 2.3 to 2.7 (five conditions of Δ0.1) for the 2nd bit, and 3 to 5 (three conditions of Δ1) for the 3rd bit, for a total of 375 different thresholds.

[0107] Table 2 shows the search results for thresholds suitable for inferring only comfort level. Table 3 shows the search results for thresholds suitable for inferring only arousal level. Tables 2 and 3 each show the top 10 conditions with the highest accuracy rates from a total of 375 search results. The accuracy rates represent the consistency between the labels attached to the pulse waves while watching a video and the labels inferred by the NN.

[0108]

[0109]

[0110] From Table 2, it can be seen that for inference of only pleasantness, the threshold value No. 1 (3rd bit: 4, 2nd bit: 2.7, 1st bit: 1.5, 0th bit: 0.1) with the highest accuracy rate is appropriate. From Table 3, it can be seen that for inference of only arousal, the accuracy rate of No. 1 and No. 2 is the same, but the threshold value No. 1 (3rd bit: 3, 2nd bit: 2.4, 1st bit: 1.6, 0th bit: 0.3) with the smaller cross-entropy error is appropriate.

[0111] Considering that emotions in the first quadrant Q1 to the fourth quadrant Q4 are ultimately inferred, it is most desirable that the threshold value No. 1 in Table 2 and the threshold value No. 1 in Table 3 match, and these threshold values ​​should be set as the threshold values ​​for inferring emotions in the first quadrant Q1 to the fourth quadrant Q4.

[0112] However, as described above, it is not always possible to find the appropriate threshold values ​​for inferring the comfort level and the arousal level. In such cases, it is preferable to find conditions with many common threshold values ​​from among conditions with a high rate of correct answers and set the threshold values ​​accordingly.

[0113] Specifically, No. 1 in Table 2 and No. 2 in Table 3 have the same values ​​for the first to third bits and have high accuracy rates, so it can be said that a threshold value of 1.5 for the first bit, 2.7 for the second bit, and 4 for the third bit are appropriate.

[0114] The threshold value for the remaining 0th bit is different between No. 1 in Table 2 and No. 2 in Table 3, so the threshold values ​​for the 1st to 3rd bits can be fixed as above, and the threshold value for the 0th bit can be changed for emotions in the first quadrant Q1 to the fourth quadrant Q4, and a threshold search can be performed again.

[0115] The threshold values ​​used in the search were 4 for the 3rd bit, 2.7 for the 2nd bit, 1.5 for the 1st bit, and 0.1 to 0.4 (four conditions of Δ0.1) for the 0th bit, for a total of four threshold values.

[0116] Table 4 shows the results of searching for thresholds suitable for inferring emotions from the first quadrant Q1 to the fourth quadrant Q4. From Table 4, it can be seen that the threshold condition No. 1 (3rd bit: 4, 2nd bit: 2.7, 1st bit: 1.5, 0th bit: 0.4) is the condition with the highest accuracy rate. The thresholds can be determined using this threshold search method.

[0117]

[0118] For step S6 and subsequent steps, the explanation of the authentication device 130 can be referred to.

[0119] Using the thresholds found above and the acquired labeled data, NN learning was performed to verify emotion inference. The inferred emotions were positive and negative comfort levels, positive and negative arousal levels, and emotions in the second quadrant Q2 (corresponding to "anger") and the fourth quadrant Q4 (corresponding to "ease"). Note that learning was also performed using only "anger" and "ease."

[0120] Figure 8A shows the accuracy rates of the inferred "comfort level," "arousal level," and "anger-happiness" emotions. It can be seen that a relatively high accuracy rate of around 70% was obtained for each. While Figure 8A shows the results for the method using fast Fourier transform (FFT) as described in Figure 3B and elsewhere, it has been found that similar results can be obtained when using discrete cosine transform (DCT) (see Figure 8B).

[0121] As explained above, by using pulse waves as biometric information and appropriately quantizing the data, emotions can be identified even using a small-scale neural network. Furthermore, the neural network model of one aspect of the present invention is smaller in scale and capable of faster processing than the techniques disclosed in Non-Patent Documents 2 and 3, and has an equal or higher accuracy rate for emotion inference. In other words, by using one aspect of the present invention, it is possible to form a small-scale, high-speed, and highly accurate emotion recognition device.

[0122] This embodiment mode can be implemented by appropriately combining at least a part thereof with other embodiment modes described in this specification.

[0123] In this embodiment, an example of a biometric information acquisition device according to one embodiment of the present invention and an example of an electronic device using the biometric information acquisition device will be described. Note that in the biometric information acquisition device and the electronic device described in this embodiment, elements having the same functions are denoted by the same reference numerals.

[0124] 9A to 9C are diagrams illustrating an authentication device 200 that acquires a pulse wave from a finger. Fig. 9A is a diagram illustrating an example of how to use the authentication device 200, Fig. 9B is a perspective view of the authentication device 200, and Fig. 9C is a diagram illustrating an optical sensor module 210 included in the authentication device 200.

[0125] The authentication device 200 has a display unit 202, operation buttons 203, an input / output unit 207, and an opening 204 for inserting a finger, which are provided in a housing 201. The housing 201 also has an optical sensor module 210 and a microcomputer 206 inside.

[0126] Here, the display unit 202, the optical sensor module 210, and the microcomputer 206 correspond to the display unit 110, the sensor 101, and the microcomputer 120, respectively, in the block diagram of Fig. 1. Furthermore, the input / output unit 207 shown in Fig. 9B corresponds to an interface that integrates the input unit 102 and the output unit 109 in Fig. 1.

[0127] The authentication operation is performed by acquiring a photoplethysmogram using an optical sensor module 210 provided below the opening 204. As shown in Fig. 9C, the optical sensor module 210 has a light source 211 and a light receiving element 212. The light source 211 may be, for example, a green-emitting LED element. The light receiving element 212 may be a photodiode or phototransistor that is sensitive to the light emitted by the light source 211.

[0128] The optical sensor module 210 is provided at a position overlapping with a part of the finger when the finger is inserted into the opening 204. Light emitted from the light source 211 penetrates the inside of the finger, and the light reflected by the tissue of the finger is received by the light receiving element 212. Because a part of the light that penetrates the finger is absorbed by the blood in the blood vessels 215, information on changes in the volume of the blood vessels can be obtained by detecting changes in the amount of reflected light. In other words, a pulse wave that changes in accordance with the pulsation of the heart can be obtained.

[0129] The authentication device 200 processes the acquired photoplethysmogram according to the flowchart in Fig. 2 and can output the result of personal authentication to an external device via the input / output unit 207. The authentication device 200 according to one embodiment of the present invention uses a small-scale neural network, and therefore can perform inference at high speed using the microcomputer 206 in the authentication device 200. Therefore, authentication processing can be performed quickly and safely without connecting to an AI workstation on the cloud or the like.

[0130] 9D is a diagram showing an example of an authentication device 220 that acquires a pulse wave from the palm of the hand. The authentication device 220 has a display unit 202, operation buttons 203, an input / output unit 207, and a light receiving window 213, which are provided in a housing 221. The housing 201 also has an optical sensor module 210 and a microcomputer 206.

[0131] The authentication operation is performed by placing a palm in close contact with the light receiving window 213 and acquiring a photoelectric pulse wave using the optical sensor module 210 provided in the housing 221 so as to overlap the light receiving window 213. The light receiving window 213 is preferably sized so that it can be covered by the palm to prevent ambient light from entering. However, the light receiving window 213 may be larger than the palm if a sensor capable of acquiring the amount of ambient light is provided. In this case, the photoelectric pulse wave can also be acquired without contacting the light receiving window 213.

[0132] The authentication device 220 processes the acquired photoplethysmogram according to the flowchart in Fig. 2 and can output the result of personal authentication to an external device via the input / output unit 207. The authentication device 220 of one embodiment of the present invention uses a small-scale neural network, and therefore can perform inference at high speed using the microcomputer 206 in the authentication device 220. Therefore, authentication processing can be performed quickly and safely without connecting to an AI workstation on the cloud or the like.

[0133] 9E is a diagram showing an example of a door 230 having an electronic lock 232 with an authentication device built in. The electronic lock 232 has an operation button 203, a handle 233, an input / output unit 207, and a microcomputer 206. The handle 233 is provided with a light receiving window 213, and a photoplethysmogram can be acquired by an optical sensor module 210 provided inside the handle 233.

[0134] The electronic lock 232 processes the acquired photoplethysmogram according to the flowchart in Figure 2 and can lock or unlock the door depending on the result of the personal authentication. Because the electronic lock 232 of one embodiment of the present invention uses a small-scale neural network, it can perform inference at high speed using the microcomputer 206 in the electronic lock 232. Therefore, authentication processing can be performed quickly and safely without connecting to an AI workstation on the cloud.

[0135] 9F is a diagram showing an example of a card 240, such as a credit card, incorporating an authentication device. Card 240 has a microcomputer 206, an input / output unit 207, a light receiving window 213, and an optical sensor module 210. Card 240 allows a photoplethysmogram to be acquired by placing a finger in close contact with light receiving window 213 provided on the surface thereof using optical sensor module 210, which is provided so as to overlap light receiving window 213.

[0136] The card 240 processes the acquired photoplethysmogram according to the flowchart in Fig. 2 and can output a determination result of personal authentication to an external device via the input / output unit 207. Note that the determination result of personal authentication may be output using a wireless function of the output unit of the microcomputer 206. The card 240 of one embodiment of the present invention uses a small-scale neural network, and therefore, the microcomputer 206 in the card 240 can perform inference at high speed. Therefore, authentication processing can be performed quickly and safely without connecting to an AI workstation on the cloud or the like.

[0137] 10A is a diagram showing an example of a smartphone 250 incorporating an authentication device. The smartphone 250 has a display unit 202, a power button 253, operation buttons 203, a speaker 254, a microphone 256, a camera 257, a light source 258, an input / output unit 207, and a microcomputer 206, which are provided in a housing 251. Note that the microcomputer 206 can be replaced with an element such as a CPU included in the smartphone 250.

[0138] In the smartphone 250, as shown in an enlarged view of the display unit 202, for example, the pixel configuration can be made up of a red-emitting subpixel R, a green-emitting subpixel G, a blue-emitting subpixel B, and a subpixel S having a light-receiving sensor. Each of the subpixels R, G, and B has a light-emitting element.

[0139] Therefore, subpixel G and subpixel S can be substituted for the optical sensor module 210 shown in FIG. 9C, and a photoplethysmogram can be acquired by touching a part of the body, such as a finger or a palm, to the display unit 202.

[0140] Note that, although the enlarged view of the display unit 202 shown in FIG. 10A illustrates pixels in a delta arrangement, the pixel arrangement is not limited to this and may be, for example, a stripe arrangement, an S-stripe arrangement, a Bayer arrangement, a zigzag arrangement, a pentile arrangement, or a diamond arrangement.

[0141] Alternatively, the adjacent camera 257 and light source 258 may be substituted for the optical sensor module 210. In this case, a photoplethysmogram can be acquired by placing a part of the body, such as a finger or a palm, on the camera 257 and light source 258.

[0142] Here, the color of the light emitted by the light source 258 is preferably green, but white light may be used as long as it contains a green light component. Note that the optical sensor module 210 may be provided in the housing 251.

[0143] The smartphone 250 processes the acquired photoplethysmogram according to the flowchart in Fig. 2 and can use various functions of the smartphone 250 depending on the determination result of the personal authentication. The smartphone 250 of one embodiment of the present invention uses a small-scale neural network, and therefore can perform inference at high speed using the microcomputer 206 in the smartphone 250. Therefore, authentication processing can be performed quickly and safely without connecting to an AI workstation on the cloud or the like.

[0144] 10B and 10C are diagrams showing an example of a smartwatch 260, which is a wristwatch-type electronic device incorporating an authentication device. Fig. 10B shows the top side, and Fig. 10C shows the back side. The smartwatch 260 has a display unit 202, operation buttons 203, an input / output unit 207, a microcomputer 206, and the like, all of which are provided in a case 261. Note that the microcomputer 206 can be replaced with an element such as a CPU that the smartwatch 260 has.

[0145] The smart watch 260 has a light receiving window 213 on the surface of the case 261 that comes into contact with the arm, and can acquire a photoplethysmogram using an optical sensor module 210 provided inside the case 261.

[0146] The smartwatch 260 processes the acquired photoplethysmogram according to the flowchart in Fig. 2, and various functions of the smartwatch 260 can be used depending on the result of the personal authentication. The smartwatch 260 of one embodiment of the present invention uses a small-scale neural network, and therefore, can perform inference at high speed using the microcomputer 206 in the smartwatch 260. Therefore, authentication processing can be performed quickly and safely without connecting to an AI workstation on the cloud.

[0147] A wearable device such as the smart watch 260 is worn on the body for a long period of time, and therefore can acquire various information from the body. Therefore, in addition to the authentication function, the wearable device may also have functions such as health management.

[0148] In the first embodiment, a method for identifying an individual based on a pulse wave was described, but a disease or fatigue level may also be inferred based on a pulse wave. Furthermore, by using an NN model that has learned the relationship between pulse waves and brain waves, brain waves can be inferred by acquiring pulse waves, allowing the physical condition to be confirmed. It is said that analyzing brain waves can reveal, for example, a state of relaxation, tension, excitement, drowsiness, and fatigue level. Therefore, by inferring brain waves, an alert can be issued before the user becomes aware of their physical condition, thereby preventing accidents.

[0149] Although a specific device is required to detect brain waves, an authentication device according to one embodiment of the present invention can input data from an external device 112, as shown in the block diagram of Fig. 1. Voltage fluctuation data acquired from the head is input as the data, and by performing a Fourier transform on the data in the same manner as for pulse waves, brain waves separated into frequency components (α waves, β waves, θ waves, δ waves, and γ waves) can be acquired. Note that a sensor for acquiring brain waves may be used as the sensor 101 shown in the block diagram of Fig. 1.

[0150] By learning the relationship between these brain waves and pulse waves, it is possible to indirectly infer the state of the body from the pulse waves.

[0151] Although the authentication device has been described in the above-mentioned FIGS. 9A to 9F, 10B, and 10C, it may also be an emotion recognition device.

[0152] For example, in the configurations shown in FIGS. 9A, 9B, 10B, and 10C, emotion data can be acquired by performing the same operations as the authentication device, and the accumulated emotion data can be used for healthcare.

[0153] 10A , the smartphone 250 can recognize the user's emotions by capturing an image of a part of the body, such as the face, using the camera 257 to acquire a photoplethysmogram. The recognized emotions can be used as information source for an AI-based dialogue engine, and can provide information such as advice, images, or music that is appropriate for the user's emotions.

[0154] In addition to smartphones, information processing devices with cameras, such as tablet computers and computers with connected cameras, can be used as emotion recognition devices in stores to analyze customer emotions and use them for product research.

[0155] Furthermore, if an emotion recognition device is incorporated into a customer service robot, it will be able to respond appropriately to the customer's emotions, thereby increasing customer satisfaction.

[0156] FIG. 11A is a diagram showing an example of a customer service robot 270 incorporating an emotion recognition device. The customer service robot 270 has a camera 271, a speaker 272, a display unit 273, a microphone 274, a running mechanism 275, and a microcomputer 206. The display unit 273 preferably has a touch panel function. The customer service robot moves to a position where it faces a customer using the running mechanism 275, and responds to the customer by emitting audio from the speaker 272 or displaying information on the display unit 273. The camera 271 is used as a sensor for acquiring the customer's photoplethysmogram. The display unit 273 and the microphone 274 function as input / output units.

[0157] The customer service robot 270 processes the acquired photoplethysmogram according to the flowchart in Figure 2, recognizes the customer's emotion, and then provides an appropriate response. The customer service robot 270 of one embodiment of the present invention uses a small-scale neural network, so that the microcomputer 206 within the customer service robot 270 can perform inference at high speed. Therefore, emotion recognition can be performed quickly and safely without connecting to an AI workstation on the cloud.

[0158] Furthermore, if emotion recognition devices are incorporated into vehicles that require driving, such as automobiles, it will be possible to prevent accidents by encouraging the driver to take a break or by providing external stimuli to alleviate negative emotions.

[0159] 11B is a diagram showing an example of an in-vehicle device 280 incorporating an emotion recognition device. The in-vehicle device 280 has a camera 281, a speaker 282, a display unit 283, and a microcomputer 206. The camera 281 functions as a sensor for acquiring photoplethysmograms, and is effectively incorporated into or placed in the vicinity of a rearview mirror 285, which is close to the driver. The display unit 283 can also have the function of a navigation system.

[0160] The driver responds by following the voices emitted from the speaker 282 or the guidance displayed on the display unit 283, and can continue driving if it is determined to be safe. Also, if it is determined that the driver is experiencing negative emotions that make safe driving difficult, accidents can be prevented by providing external stimuli such as encouraging the driver to take a break, playing music that alleviates the negative emotions, or adjusting the air conditioning. Also, if it is recognized that the driver is experiencing emotions influenced by alcohol or drugs, a response can be taken that makes it impossible for the driver to start driving.

[0161] The in-vehicle device 280 processes the acquired photoplethysmogram according to the flowchart in Fig. 2, recognizes the driver's emotion, and then takes appropriate action. The in-vehicle device 280 according to one embodiment of the present invention uses a small-scale neural network, and therefore can perform inference at high speed using the microcomputer 206 within the in-vehicle device 280. Therefore, emotion recognition can be performed quickly and safely without connecting to an AI workstation on the cloud or the like.

[0162] This embodiment mode can be implemented by appropriately combining at least a part thereof with other embodiment modes described in this specification.

[0163] 101: sensor, 102: input unit, 103: calculation unit, 104: calculation unit, 105: memory unit, 106: memory unit, 107: ADC, 108: display driver, 109: output unit, 110: display unit, 111: AI workstation, 112: external device, 113: external device, 115: bus line, 120: microcomputer, 130: authentication device, 140: emotion recognition device, 150: number of data, 175: number of data, 200: authentication device, 201: housing, 202: display unit, 203: operation button, 204: opening, 206: microcomputer, 207: input / output unit, 210: optical sensor module, 211: light source, 212: light receiving element, 213: light receiving window, 215: blood vessel, 220: authentication device, 221: housing, 230: door, 232: electronic lock, 233: handle, 240: card, 250: smartphone, 251: housing, 253: power button, 254: speaker, 256: microphone, 257: camera, 258: light source, 260: smart watch, 261: case, 270: customer service robot, 271: camera, 272: speaker, 273: display unit, 274: microphone, 275: driving device, 280: in-vehicle device, 281: camera, 282: speaker, 283: display unit, 285: rearview mirror

Claims

The function to obtain pulse wave data as analog data, converting said analog data into digital data; A function of performing a fast Fourier transform on the digital data; A function of extracting data of a plurality of points from the low frequency side from the data after the fast Fourier transform; A function of quantizing each of the plurality of points of data; A function of performing individual authentication or emotion recognition by inference using each of the quantized data; A biometric information acquisition device having the above structure. 　 In claim 1, The pulse wave is a photoelectric pulse wave. 　 In claim 2, The photoelectric pulse wave is acquired at a sampling rate of 10 Hz or more and 100 Hz or less. 　 In claim 1, A biometric information acquisition device, wherein the number of data points extracted from the low frequency side is 150 or more and 200 or less. 　 In claim 1, A biological information acquisition device, wherein the multiple points of data extracted from the low frequency side are data up to 12 Hz. 　 In claim 1, The quantized data is 4-bit 5-value data reflecting flag notation. 　 In claim 1, The biometric information acquisition device uses a fully connected neural network for inference using the quantized data. 　 In claim 1, The biological information acquisition device recognizes the emotion as either a positive or negative degree of comfort or a positive or negative degree of arousal. 　 In claim 1, The biometric information acquisition device recognizes four emotions: "joy," "anger," "sorrow," and "pleasure." 　 A wristwatch-type electronic device comprising the biometric information acquisition device according to any one of claims 1 to 7, a case, an optical sensor module, and a light receiving window, The light receiving window is provided on a surface of the case that contacts the arm, The electronic device comprises: an optical sensor module disposed in the case so as to overlap with the light receiving window; 　 In claim 10, The biometric information acquisition device is an electronic device having a function of inferring physical condition from the pulse wave using a neural network model that has learned the relationship between the pulse wave and brain waves.

Images